A recent survey from Deloitte found that one in five U.S. residents say they have either cut the cord (i.e., cancelled cable or satellite service) or are thinking about doing it. Nine percent of survey respondents indicated they have recently “cut the cord” while another eleven percent are thinking about it. Cord cutters are turning to services like Netflix, Hulu, and Amazon Prime, to name a few.

Interesting and a considerable percentage, but probably not true.

The problem?  The TV industry reported flat subscriptions last year.  As this article notes, even with a large margin of error, the numbers don’t work.

What happened?  Assuming the TV industry is not misleading their subscribers (a 10% drop would be hard to hide in earnings reports) or there isn’t a flood of new subscribers hiding the cutters (though the article seems to discount that, too), the one culprit is likely poor sampling.

The survey was administered online, and while more and more Americans are online, biases are still likely.  While they didn’t reveal how they sampled respondents online, my initial guess is that they skewed towards early adopters and people with faster, more consistent access to the internet.  The exact population you would expect to be shunning cable in favor of online options.

If you’re not really surveying a representative sample of the population you’re interested in, then you’re not really surveying them.

• Set expectations. Inform them of the length, topic, nature of questions, etc.  Don’t lead them to believe one thing then surprise them on the survey with another.
• Allow them to opt out. Make opting out of the research easy.
• Inform them how you got their information. Respondents are increasingly wary of threats to their privacy.  Be sure to convey how you got their information.  Better yet, have your client contact them first alerting them to the research.
• And tell them how you will protect their information and privacy. Once you tell them how you got their information, tell them how it will be used AND not used.  Reselling their information?  Will they be linked to their responses?
• Let them know how the research is being used. While not always possible due to blinded studies, when possible include them in what the research is being used for – make them feel like a partner in the process.
• Inform them how to get in touch with a human if they have concerns. No matter how respectful you are, some people may still be upset that you contacted them (privacy being a top concern).  Make the process of doing so easy, unless you want to make them even more angry.
• Providing a personal reply. Respondents are taking the time to provide us a personalized response of their opinions – we should return the courtesy when they contact us with a problem.

Respondents are a nonrenewable resource – let’s take care of them like any other precious resource.

Take for example a tracking study we did measuring recreation activities.  We saw a large drop in certain outdoor recreation between two studies – but the first one was conducted in spring and the other in winter.  If you measured the concern over the flu among the general public, the research during the time of the flu season would likely result in higher concern.

A recent example?  Toyota took a hard fall in J.D. Power and Associates’ Initial Quality Study this year.  While Toyota obviously has had a rough year, their ratings were likely hurt further by the timing of the survey which took place from February through May (The below graph shows Google Trends data on “Toyota Recall“, point C,  which peaked immediately before the survey).

While the goal is often to measure the impact of these events, other times it is not ideal to measure the impact of a short-lived event and you at least should be aware of the possible impact.  What to keep an eye out for?  Three big ones are:

• Seasonal and weather (e.g blizzards, heat waves, etc.)
• Political events
• News stories (yes, a catch-all, but even a brief news story on a topic can change the opinions of respondents)*

* You can use news monitoring services like Google Alerts to keep an eye on developing stories.

The Economist summed it up nicely in an article from last week’s edition.

When asked whether they supported a variety of issues, most people showed strong support.  However, when the downsides of those propositions were pointed out, support decreased significantly.  The strongest example was support for outfitting soldiers with the best equipment – 83% strongly supported.  When reminded that it would mean less spent on public services that they’d receive, support dropped to 46%.

Interestingly, alcohol bucked the trend.  When asked if the government should try to limit people’s boozing by taxing alcohol more 32% strongly agreed.  When reminded that it would mean they’d personally pay more for alcohol, there was no statistical change.  Presumably, people understood this issue well enough from the start and knew it meant they’d pay more, and therefore that was already calculated into their first response.

Many, if not all, issues can be framed in terms of gains or losses for those concerned.  Asking a question that only addresses one side of the coin is considered a research bias.  Pundits and politicians may rely on biased questioning to sway support for their beliefs, but researchers shouldn’t.

In the end, be careful what you’re asking.  People don’t make decisions in  a vacuum and your research shouldn’t be conducted in one either.

So, we were very happy to see this article on ABCnews.com discussing the results of a recent study by Stanford University scientists (full report here), showing that, in fact, opt-in panels produce results that are significantly less accurate than results from randomly- (i.e., probabilistically-) selected panels.  And perhaps even more important, weighting the results from the opt-in panel to match population demographics did not increase accuracy in the opt-in panel results.

In a sense, these results vindicate our commitment to using only proven, high-quality research methodologies, to ensure that our clients receive results they can trust – results they can rely on to make decisions.

Why?  Well, you hope to learn something new, hence the 20 percent.  But you also hope that a good proportion of the survey matches your view of reality.  If not, it means that either you’re really out of touch with your topic, or it means that something may be amiss with the survey.  Neither of those is a happy thought.

When we first run the numbers on one of our surveys, one of the first things we do is a common-sense check.  If we were guessing the answers, what would we guess?  And do the numbers seem reasonable in light of those guesses?  Do the numbers make sense relative to each other?  If we do a survey where all of the results seem counterintuitive, we start to worry and we start going through the numbers with a microscope.  Even with a strong staff, we’ll sometimes mistype a weighting factor or mislabel a data field.  The key is that we have the quality control processes in place to fix those errors before they ever see the light of day.

When I saw this study released recently by the Oklahoma Council on Public Affairs , alarm bells immediately went off.  The survey, conducted more or less as a spot civics quiz among Oklahoma high school students, indicated a dismal awareness of basic American knowledge.  When given a sample of questions from the U.S. Citizenship Test given to immigrants, the survey reported that only 2.8 percent of Oklahoma high school students would pass the test based on that sample.  They also cite another survey with similar results in Arizona as backup to their conclusion.

However, the individual responses to questions on this survey were brow-furrowing.  Only 23 percent of students could name the first president of the United States?  Only 43 percent could name the two major political parties in the U.S.?  Those figures seem absurdly low.  They don’t pass my common-sense test, and if we were finding those results on a Corona survey I’d be hip-deep in the numbers trying to figure out if they’re really accurate or if we misplaced a decimal somewhere along the line.

And I wasn’t the only one who found the results suspicious.  Oklahoma State Representative Ed Cannaday, a former social studies teacher, questioned it based on his experience in the field , and it has been the subject of various blogs.   The furor is rising.

Now, we haven’t seen the data and we haven’t reviewed the methodology.  It’s possible that the results are accurate, because if we all knew the right answer ahead of time we wouldn’t be doing surveys.  It’s also possible that there was an innocent error in the methodology or the calculations, and of course there are other less innocent scenarios as well.  It would be imprudent to immediately make a judgment of any type until the data is carefully reviewed.

So is this a rare survey where the results completely blow up our normal perception of reality?  Or is it an example of “market research gone wrong”?  I suspect we’ll find out soon enough.

While one might think this means LinkedIn understands research, a recent survey I received from them makes me think otherwise.

I took their survey, which was done on survey monkey (with all of their partners, they couldn’t find a better service?); skip patterns didn’t work (or they were asking me irrelevant questions for fun), and overall it gave me little confidence in their research knowledge.  (I wish I would have taken screen shots to show here.)

Can I really trust them to provide sample and execute a survey if they can’t do it for themselves?

Has anyone else had experiences with this service yet – good or bad?

While the purpose of these polls is often for entertainment the way they are executed is often too similar to that of real surveys – anyone can take it and can take it as many times as they want.  Take this example of a Time Magazine ‘Most Influential’ poll.  While it was obvious in this case that the poll was rigged, would be as obvious in your own survey if someone with  a strong opinion – or just a weird sense of humor – decided to skew the results?

Be careful how you execute your surveys and control who can take them (and how many times).  Unless, of course, you’re only conducting research for sheer entertainment.

When asking respondents to rank items, it is important to keep in mind how many items they can meaningfully keep track of when ranking.

To keep things manageable, there are several methods you can employ:

• Ask the respondent to only rank their top choices (this is probably all you really are concerned with anyway); likewise, if you really want to know, you can ask them to rank their bottom choices (if you do this, remove the choices they already listed as their top choices when later asking their bottom choices).
• Ask respondents to first check all that apply from a list.  For example, “Which attributes do you consider when purchasing X?”  Then have them only rank those attributes that they care about.
• Break the list up into logical segments to help respondents focus their thinking.  Don’t ask everything at once.  For example, for an automobile, you can sort the list by entertainment system, safety features and so on.
• Randomly select list items to create a few smaller lists for ranking.  Lists can even be divided among participants so not every person ranks every item.

Creating ranking questions that are easy to decipher and answer from the respondents’ point of view creates a better survey that leads to better, more meaningful data.  For an analysis of the implications of using various ranking methods, check out this article in this month’s issue of Quirk’s Marketing Research Review (free registration required).

I happily started the survey and realized it was supposed to be about my most recent visit to their service department.  Hmm.  Couldn’t remember when that was.  Oh well, I’ve been there a few times so I thought I could still answer.  Wrong.

The questions were very specific to that visit, which isn’t necessarily bad except that they were asking respondents to recall details of their visit that most people would probably have forgotten right after they left the dealer, let alone remembered weeks or months later.  For example…

• I was promptly greeted upon arrival [Scale: Completely Disagree to Completely Agree]
• My paperwork was ready when I walked in to pick up my car [Scale: Completely Disagree to Completely Agree]
• Availability of business cards [Scale: Unacceptable to Extraordinary]

After struggling to find questions that I could answer, I went through my car files to see when I was actually there.  While I did buy an oil filter in the service department back in January (2 months ago), my car was last serviced in October (5 months ago!).

While I applaud businesses for wanting to measure customer satisfaction (then hopefully act on it), it must remembered that customers cannot always recall specific parts of the experience (or care), and the more specific the information you want, the closer to the actual experience you must be when you measure it (exit survey? secret shopper? one-on-one interview during the experience?).

Finally, the actionability of the information gathered should be considered.  What will change because of the result?