Evaluating the quality of a community survey can be confusing and counterintuitive. Over the years, I have developed a few different analogies to help describe to community leaders what a good survey includes. In Montana, I used an elk hunting analogy. In Arizona, I compared it to pitching baseballs. Those were fun place-based equivalencies, but I’ve found one that is more universal.  A survey is like the three parts of an ice cream cone: the ice cream, the cone, and the sprinkles. Let’s take a closer look at what I mean…

The Ice Cream

The essence of an ice cream cone is the ice cream itself (a crispy waffle cone with sprinkles dumped inside would be pretty weird, right?). Similarly, the essence of a good survey is good measurement. Just like a chemist measures water temperature with a thermometer, survey researchers measure topics like satisfaction using a survey instrument (aka, a questionnaire). The accuracy of the measurement depends on the quality of the survey instrument—a good survey instrument is understandable, relevant, and answerable. As survey researchers, our first job is to make good measurements by designing accurate and reliable survey instruments, just like an ice cream shop starts with tasty ice cream.

The Cone

After choosing your flavor, the clerk doesn’t scoop it directly into your hands. What a mess that would make. Instead, we use a cone or cup to hold and contain the treat. The cone keeps things organized. In survey research, representative sampling (i.e., the deliberate process of deciding who gets invited to participate in the study) is like the cone. Good representative sampling makes the measurement useful. Haphazard sampling, such as posting a survey link on a website or social media, just leads to more questions than answers. Representation is about which people are included; not how many people are included. Getting more survey participants online does not mean they are the right participants.

Let’s face it, the cone is not the star of the ice cream store, but it’s crucial if you want to avoid cold and sticky hands. Likewise, representative sampling is not the most exciting part of the survey process, but it is the foundation of a good survey, and it is one of the best ways that a research consultant can improve a survey’s quality.

The Sprinkles

Finally, we get to the toppings – the most eye-catching part of the entire ice cream cone. In this analogy, the sprinkles are akin to the total number of surveys collected. Many clients intuitively focus on this aspect when evaluating the quality of the survey, expecting that a survey with a high number of responses is good and one with fewer response is not good. But that would be like evaluating an ice cream store by only the number of sprinkles they give you, regardless of the quality of their ice cream or if they just scoop it into your hands. Once good measurement and representative sampling is achieved, then more participants is better (to an extent), just like sprinkles make your treat better once you have ice cream and a cone. But if there are serious measurement or representation issues, then collecting more surveys won’t fix your problem, just like pouring on a pound of sprinkles won’t help if you are stuck holding melted dairy product.

The bottom line is the number of survey responses collected is only a useful indicator of the quality of a survey if there is first good measurement and representative sampling. A community survey with 1,000 responses but poor measurement and sampling is just a pile of sprinkles.