The Economist summed it up nicely in an article from last week’s edition.

When asked whether they supported a variety of issues, most people showed strong support.  However, when the downsides of those propositions were pointed out, support decreased significantly.  The strongest example was support for outfitting soldiers with the best equipment – 83% strongly supported.  When reminded that it would mean less spent on public services that they’d receive, support dropped to 46%.

Interestingly, alcohol bucked the trend.  When asked if the government should try to limit people’s boozing by taxing alcohol more 32% strongly agreed.  When reminded that it would mean they’d personally pay more for alcohol, there was no statistical change.  Presumably, people understood this issue well enough from the start and knew it meant they’d pay more, and therefore that was already calculated into their first response.

Many, if not all, issues can be framed in terms of gains or losses for those concerned.  Asking a question that only addresses one side of the coin is considered a research bias.  Pundits and politicians may rely on biased questioning to sway support for their beliefs, but researchers shouldn’t.

In the end, be careful what you’re asking.  People don’t make decisions in  a vacuum and your research shouldn’t be conducted in one either.

So, we were very happy to see this article on ABCnews.com discussing the results of a recent study by Stanford University scientists (full report here), showing that, in fact, opt-in panels produce results that are significantly less accurate than results from randomly- (i.e., probabilistically-) selected panels.  And perhaps even more important, weighting the results from the opt-in panel to match population demographics did not increase accuracy in the opt-in panel results.

In a sense, these results vindicate our commitment to using only proven, high-quality research methodologies, to ensure that our clients receive results they can trust – results they can rely on to make decisions.

It’s like an Olympics for marketers only it happens once every 10 years.  And its effects are far reaching — from government needs (distribution of congressional seats, federal funding, etc.) to consumer goods (where to open new stores, product mixes, etc.).

The Census will provide the data to confirm or refute large societal shifts and greater diversity among neighborhoods and communities.  Better business decisions, better marketing, all with greater confidence, should follow.

So it should be no surprise given what rests on the outcomes that the Census is spending significant time and money getting the word out.  In fact, more than twice the money (more than $300 million on getting the word out) and in 11 more languages (28 total) than the 2000 Census.  New media (i.e., Facebook, Twitter, YouTube), traditional media (i.e., TV, radio, print), and even a traveling road show will play a part.

Given the importance of the outcomes for our clients, we’ll certainly be keeping tabs on how things progress.

Above ad is part of the campaign to get the word out about the Census.  What do you think of it?  More at AdFreak.

In a recent post, he comments on a misleading NY Times post on the Kindle’s declining satisfaction. In summary, he noted…

• People can provide reviews even without owning the device
• Only people who are passionate (positive or negative) will take time to review (see a previous post from us on a similar topic here)
• Opinions will vary depending on the type of customer and the product life cycle stage

While user feedback and comments are key in a user-based, participation web, it is hard, if not impossible, to draw meaningful statistics that can be projected onto all users from these comments (at best you can only say, “X% of PEOPLE WHO COMMENTED, said Y”). And looking at the bigger picture through more insightful analysis (e.g. what are possible reasons behind the numbers?) should always be done to prevent the numbers from leading you astray.

For another interesting story on the Kindle, see this recent Mashable post.

Why?  Well, you hope to learn something new, hence the 20 percent.  But you also hope that a good proportion of the survey matches your view of reality.  If not, it means that either you’re really out of touch with your topic, or it means that something may be amiss with the survey.  Neither of those is a happy thought.

When we first run the numbers on one of our surveys, one of the first things we do is a common-sense check.  If we were guessing the answers, what would we guess?  And do the numbers seem reasonable in light of those guesses?  Do the numbers make sense relative to each other?  If we do a survey where all of the results seem counterintuitive, we start to worry and we start going through the numbers with a microscope.  Even with a strong staff, we’ll sometimes mistype a weighting factor or mislabel a data field.  The key is that we have the quality control processes in place to fix those errors before they ever see the light of day.

When I saw this study released recently by the Oklahoma Council on Public Affairs , alarm bells immediately went off.  The survey, conducted more or less as a spot civics quiz among Oklahoma high school students, indicated a dismal awareness of basic American knowledge.  When given a sample of questions from the U.S. Citizenship Test given to immigrants, the survey reported that only 2.8 percent of Oklahoma high school students would pass the test based on that sample.  They also cite another survey with similar results in Arizona as backup to their conclusion.

However, the individual responses to questions on this survey were brow-furrowing.  Only 23 percent of students could name the first president of the United States?  Only 43 percent could name the two major political parties in the U.S.?  Those figures seem absurdly low.  They don’t pass my common-sense test, and if we were finding those results on a Corona survey I’d be hip-deep in the numbers trying to figure out if they’re really accurate or if we misplaced a decimal somewhere along the line.

And I wasn’t the only one who found the results suspicious.  Oklahoma State Representative Ed Cannaday, a former social studies teacher, questioned it based on his experience in the field , and it has been the subject of various blogs.   The furor is rising.

Now, we haven’t seen the data and we haven’t reviewed the methodology.  It’s possible that the results are accurate, because if we all knew the right answer ahead of time we wouldn’t be doing surveys.  It’s also possible that there was an innocent error in the methodology or the calculations, and of course there are other less innocent scenarios as well.  It would be imprudent to immediately make a judgment of any type until the data is carefully reviewed.

So is this a rare survey where the results completely blow up our normal perception of reality?  Or is it an example of “market research gone wrong”?  I suspect we’ll find out soon enough.

Looking around, it would seem like Mac has a very healthy market share (I’m one of the sole PC users here), but of course they are a niche maker of computers.

In contrast, the article I was reading made it sound like doom and gloom for Mac, until at the end, where they stated the proportion of students who want a Mac is still double the current market share of Macs in the US.

Obviously, my coffee shop “sample” and the online sample the article relied on are not representative of the same larger population, though each could be representative of some larger population.

The bottom line is, don’t forget to consider how your sample (who you’re asking) no matter how large or small, may be different from the larger population (who you’re looking at).

We’ve had some “debates” here at the office about the best types of graphs, charts, or other visual means of portraying data (Word clouds anyone?).  We even had a few posts (here and here) last year when we disagreed with Seth Godin’s three laws of graphs.

I was amused when I saw this post from the Future Place Blog about pie charts.  We agree that, except for the simplest of data, pie charts are usually a bad idea.

Perhaps the movement is growing!

When asking respondents to rank items, it is important to keep in mind how many items they can meaningfully keep track of when ranking.

To keep things manageable, there are several methods you can employ:

• Ask the respondent to only rank their top choices (this is probably all you really are concerned with anyway); likewise, if you really want to know, you can ask them to rank their bottom choices (if you do this, remove the choices they already listed as their top choices when later asking their bottom choices).
• Ask respondents to first check all that apply from a list.  For example, “Which attributes do you consider when purchasing X?”  Then have them only rank those attributes that they care about.
• Break the list up into logical segments to help respondents focus their thinking.  Don’t ask everything at once.  For example, for an automobile, you can sort the list by entertainment system, safety features and so on.
• Randomly select list items to create a few smaller lists for ranking.  Lists can even be divided among participants so not every person ranks every item.

Creating ranking questions that are easy to decipher and answer from the respondents’ point of view creates a better survey that leads to better, more meaningful data.  For an analysis of the implications of using various ranking methods, check out this article in this month’s issue of Quirk’s Marketing Research Review (free registration required).

I happily started the survey and realized it was supposed to be about my most recent visit to their service department.  Hmm.  Couldn’t remember when that was.  Oh well, I’ve been there a few times so I thought I could still answer.  Wrong.

The questions were very specific to that visit, which isn’t necessarily bad except that they were asking respondents to recall details of their visit that most people would probably have forgotten right after they left the dealer, let alone remembered weeks or months later.  For example…

• I was promptly greeted upon arrival [Scale: Completely Disagree to Completely Agree]
• My paperwork was ready when I walked in to pick up my car [Scale: Completely Disagree to Completely Agree]
• Availability of business cards [Scale: Unacceptable to Extraordinary]

After struggling to find questions that I could answer, I went through my car files to see when I was actually there.  While I did buy an oil filter in the service department back in January (2 months ago), my car was last serviced in October (5 months ago!).

While I applaud businesses for wanting to measure customer satisfaction (then hopefully act on it), it must remembered that customers cannot always recall specific parts of the experience (or care), and the more specific the information you want, the closer to the actual experience you must be when you measure it (exit survey? secret shopper? one-on-one interview during the experience?).

Finally, the actionability of the information gathered should be considered.  What will change because of the result?

The other day, someone showed me this graph out of an advertisement for, what else, but using a specific direct mail service for advertising.

At first, we made fun of the graph.  It looked utterly useless, had no axis labels or scale, and you don’t even really know what it’s measuring.

Then, as I was about to dismiss it, it occurred to me that maybe it isn’t that bad.  Okay, from a mathematical standpoint, it’s still horrible, but one thing it does right is that it tells a story and does so distinctly.  Someone can look at this graph and grasp what it is trying to say pretty quickly.

What do you think?  Great graph or horrible graph?

(For fun, I’ll just leave you with this:  One of our analysts wanted to label the Y-axis, “Number of stamps used.”)