One of the most important factors that determines if your [fill in research mode here … survey, focus group, etc.] produces accurate results is your sample. A sample, by definition, is a subset of the population you are studying that is selected for the actual research study. Perform your research with the wrong sample, or just one that is inaccurately designed, and you will almost certainly get misleading results (in the industry, this is called external validity–the extent to which your sample’s results can generalize to the population you care about).

I bring this up because I am seeing more and more surveys where anyone can respond, and can often respond multiple times. Recently, I’ve seen several online surveys with no participation restrictions (and not just the fun opinion polls) and email snowball samples (a snowball sample is one in which additional respondents are recruited from referrals from initial respondents). I was going to post the survey links here until I realized I would only be contributing to their poor results.

These ranged from usage and opinions of outdoor hiking areas; an economic impact study on the impact that access to a rock climbing area produces; and a public opinion survey on criminals. In each example, the survey would have most likely gone to only those with strong opinions, and in some cases, only those with an opinion that they wanted to hear anyway. Not to mention, anyone who wanted to could take the survey multiple times with little effort. The best sampling is one in which all respondents in your study population have an equal chance of being chosen to participate and limits respondents to those that you chose.

This brings up a little relevant, election year history lesson too (I know you love history). Back in 1936, Literary Digest (whose readers were largely wealthy Americans) predicted Alfred Landon (who?) would win over Franklin Roosevelt. George Gallup, with a smaller, but more scientifically sampled poll, accurately predicted Roosevelt would win (and he did in a landslide). Then, 12 years later in 1948, Gallup got it wrong when he shut off the polling two weeks before the election (that missed some of the action) and inaccurately predicted Thomas Dewey would win over Harry Truman.