We’ve talked plenty about good survey methodology, sampling, and making sure you’re getting the right information, and here is another excellent example:

I was up in the mountains this past week skiing and decided to demo (i.e. try out) some new skis.  So, I went to the shop at the resort and tried several different pairs.  Of course, they only carry certain brands of skis and one of the pairs I wanted to try wasn’t stocked (for those of you that care, I wanted to demo a pair of Atomic Nomads).  I talked to the ski tech while he was adjusting a pair for me about how they pick the skis they carry.

Apparently, among other things, they do slope-side surveys of what brands of skis people are using.  Despite wanting to have that job, I immediately thought of the problem this created.  They pick the skis they’re going to carry based on what people are already skiing, AND people ski (largely) what they already carry.  I hope they account for that (and do the people spotting the skis know how to screen out rentals?), otherwise all of Colorado will soon be skiing the same pair of skis!

So, if you’re trying to measure something through a survey (or even another mode of research), make sure you’re not directly influencing what you’re trying to measure, and if you cannot limit the influence, at least understand its implications.