---------------------------------------------------------------------- Message-ID: <nkuh7b$nb$1@grapevine.csail.mit.edu> Date: Tue, 28 Jun 2016 18:55:39 +0000 (UTC) From: wollman@bimajority.org (Garrett Wollman) Subject: Re: Are telephone surveys statistically valid? In article <barmar-958F99.10375127062016@88-209-239-213.giganet.hu>, Barry Margolin <barmar@alum.mit.edu> wrote: >In article <nkmg1f$pof$1@grapevine.csail.mit.edu>, > wollman@bimajority.org (Garrett Wollman) wrote: > >> Of course you could argue it another way: they are "statistically >> valid" by construction, the only question is whether the population >> being sampled is sufficiently similar to the population of interest to >> allow for generalization. > >If I were them, I'd compare the surveyed population with the US census, >which is about as accurate a description of the US population as likely >exists. That is exactly what they do do, as a part of the reweighting process. This does however mean that the effect of sampling error is magnified for some demographic groups, depending on how poorly those groups are represented in the sample. -GAWollman -- Garrett A. Wollman | What intellectual phenomenon can be older, or more oft wollman@bimajority.org| repeated, than the story of a large research program Opinions not shared by| that impaled itself upon a false central assumption my employers. | accepted by all practitioners? - S.J. Gould, 1993 ------------------------------ Message-ID: <Faudnd7CkfGB1u7KnZ2dnUU7-YfNnZ2d@posted.internetamerica> Date: Tue, 28 Jun 2016 23:14:20 -0500 From: gordonb.b8q5b@burditt.org (Gordon Burditt) Subject: Re: Are telephone surveys statistically valid? >> Of course you could argue it another way: they are "statistically >> valid" by construction, the only question is whether the population >> being sampled is sufficiently similar to the population of interest to >> allow for generalization. I don't think that is possible unless survey participation is mandated at gunpoint. And in such a survey, you won't get accurate opinions about gun control. I don't think "statistically valid by construction" exists. There are way more dimensions to people than age, sex, and race, and maybe religion, and it's easy to match someone in these but be on opposite sides of an issue (consider labor union strikes - in any issue related to a strike, it DOES MATTER whether the person in question is labor (striking union), labor (another union), management, disgruntled customer, or uninvolved). (Did you make sure that the sample matches the population in income? weight? height? education level? number and ages of children? marital status? number of divorces? employer? personality? job title? food allergies? gun ownership? number of abortions? drug use? number of minutes in cell phone plan? favorite sports team? Internet Service Provider? ) Self-selection is a less blatant problem on satisfaction surveys, but it is still an issue. Those with extreme views (pro or con) are more likely to expend more effort making sure their survey is counted. If the call drops, they'll try again. They'll spend more effort getting the survey done on an overloaded internet server. They'll bug tech support for a way around the problem. Some guy who thinks he has been cheated out of $10 may be willing to spend $100 to take down the offending merchant (say, by ballot-box stuffing rating surveys, giving 1-star reviews) even if he's got no chance of getting his money back. Any survey can be blatantly ruined by stupidity, and that includes limiting your view of the population to age, sex, and race. If self-selection or self-deselection (e.g. hanging up if they hear the word "survey", or putting that word in their SPAM filter for email) is correlated to the questions on the survey, you're in trouble. One of the worst problems is only being able to access the survey if you have a phone, or internet access, and you are being asked questions related to phone service or internet service. > If I were them, I'd compare the surveyed population with the US census, > which is about as accurate a description of the US population as likely > exists. That still doesn't protect you against a HUGE bias caused by the method you use to conduct the survey. If a self-selection bias is correlated with the questions on the survey, you've got a problem, especially if it has nothing to do with age, sex, and race. Example: if Trump wants to shut down the Internet (and anyone takes this seriously), and Hillary wants to leave it open, you may well have a strong bias if you take a poll on the Presidential race over the Internet. Network neutrality is a similar issue. That's a problem for Presidential race polls that hasn't existed in the past. I recall a discussion of some kind of campus-wide student opinion poll at RPI for some purpose, and it was proposed to do it by phone. (The internet didn't exist yet) Several problems were listed, including: 1. Freshmen do not have phones. (True when I was there, and cell phones were pretty much nonexistent in the early 1970's. A dorm holding maybe 100 freshmen had 3 pay phones near the common room. The numbers to call *IN* were known to a few, and that excluded most freshman dorm residents.) Even today, a phone poll may not be feasable: presumably most every student has a cell phone, but is there a phone book of campus residents? another problem, if the poll were supposed to represent something beyond the limits of the campus: 2. The female RPI engineering student is so rare less than half of the sophomore class claims to have seen one, ever. (Someone actually did a poll on this). Remember, early 1970's. Well, it was easy to stand up in an Electrodynamics class, look around, and identify a female or two, but if "engineering student" required "an Engineering Major", you'd have to approach them with "Hi, what's your major?" and if male engineering students could do that, they'd be talking about something else. You also couldn't be sure those females weren't girlfriends or parents of class members they were sitting next to and were not enrolled at RPI at all. Well, if this was supposed to be a campus-wide poll, well, women are rare, and the survey reflects that. If it was supposed to predict a national political race, women as voters are NOT rare, and it's a problem for the survey. The basic 2010 census didn't ask whether you had a phone or not, or what kind of phone. Nor did it ask whether you had an indoor toilet. Nor did it ask whether you are a Democrat or a Republican or Tea Party or Weed Party. They also didn't ask about sexual preference, if any, which restroom they used, or what it said on their birth certificate, if any. It didn't ask if they were a citizen or were in the USA legally. They might have had longer forms that asked whether you have a phone, but what they asked on the basic 2010 census was pretty short, and seemed mostly related to a total count and distribution of age/sex/race. The Census may be a great thing for checking that your survey ends up representing the population by age, sex, and race (whatever sex and race are - census takers were supposed to encourage people to give answers on the list, but if they insisted on saying they were Romulan, take them at their word, and do NOT look at them and try to guess what they are. If they said they are a woman, write it down, regardless of how much they look like a man dressed as a woman.) It doesn't help with political party affiliation. It doesn't help much with being sure that you are weighting the left and right "wings" of each party properly. And those "wings" are multi-dimensional: a person may be a fiscal conservative and liberal on social issues. Suppose that about 48% of the population is in favor of the death penalty for making an unsolicited call for a survey, and 48% is against it. 4% undecided, (40% for those who don't have a phone) and it's a dead heat. Suppose you call a whole bunch of people asking whether they are in favor or against such a law, and whether they have a phone. What kind of results would you expect to get? I think you'd end up with, out of the people who actually responded to the survey, about 20% in favor and 80% against the law, (survey writers tend to not allow "undecided" as a choice, which I think is a big mistake, although most political polls seem to allow it) and about 100.000000% of them have a phone. Of those who hung up, there might be 80% in favor and 20% against, and 100.000000% of these people also have a phone, but you don't get survey results from these people. That survey got the opinion on the law badly wrong, and missed entirely the don't-have-a-phone people (yes, it's a very small number, but not zero, and you might conclude that Lifeline service was helping everyone that needed it so it doesn't need a bigger budget). It failed because of sampling bias, but not sampling bias that checking against the Census could warn you about. ------------------------------ ********************************************* End of telecom Digest Thu, 30 Jun 2016