Working with big databases is hard work.  It’s not just the mechanics of data manipulation and calculations, but also knowing where the data come from, what they mean, and how to interpret them.  You have to have a feel for data, and the ability to recognize the little clues that point to problems.  Mistakes happen, and kudos to the National Association of Realtors for finding these problems, fixing them, and being forthright about the changes that we’ll be seeing in their numbers.  And if you sold your home twice in their database, we hope you can get back some of those Realtor fees.

You might be able to guess the two largest populations of foreign students, particularly if you went to an engineering college – Chinese, then Indian.  But what countries occupy Spots 3 through 10 on the list?  Some of them may surprise you.

1. Chinese
2. Asian Indian
3. Korean
4. Japanese
5. African (no detail)
6. Sudanese
7. Russian
8. Colombian
9. Taiwanese
10. Serbian

Serbian?  Sudanese?  We’re guessing you can win some friendly wagers with this information.  And we’re wondering if the Colombian students disproportionately attend the University of Missouri-Columbia in a lucky marketing coincidence.

• Everyone knows about the whole “aging boomer” thing, so we won’t belabor that point. However, it’s interesting to see that another fast-growing segment is the 18-24 age category. While the senior population grew 15 percent over the past decade (and the 45-64 age group at a whopping 32 percent), our traditional college-age segment of 18 to 24 year olds grew by 13 percent.
• The median age of the nation has risen from 29.5 in 1960 to 37.2 today. The oldest state in the nation? Maine, at a median of 42.7. The oldest states are clustered in the northeast other than the notable exception of Florida.
• Throughout the United States, the county with the highest ratio of men to women is our own Crowley County in southeast Colorado. But single women probably shouldn’t flock there. The ratio is high because of the presence of a state prison for men.

What trends are interesting to you and your work?

During the last two years as indicated by the orange and red lines (and previous years too – just not displayed) we see a small spike in February (people rushing to get their refunds early) and then a much larger spike in April (right before the deadline).  This year (blue line) the overall trend is no different, though we see people searching for tax terms earlier and perhaps not procrastinating as much in April with the quicker rise in the blue line.  An oddity or is something causing the difference?

Where are you on the curve?

Source: Google Insights

We talk a lot about the virtues of data, but there is one important caveat – you have to have good data.  Sounds easy, right?

While I’m not going to write a dissertation here on how to set up databases or program data collection forms, I do hope to provide some high level thoughts on the process of data collection (not just the tools):

• Determine what data needs to be collected. Start with the end in mind – what do you hope to gain from the data?  Also consider what is possible – what is feasible to collect?
• Create a system. This one of the biggest areas I have personally seen gone astray.  You can have all the databases, software, and forms, but if the data doesn’t get in there right, or consistently, the whole database breaks down.  Develop a simple process that everyone can follow.  Then enforce it.  Where possible, make it automated or at least integrate with other processes.
• Share success. It can be hard for people collecting, entering, and sorting the data to understand how they are helping the big picture.  Keep them informed with how their efforts have helped and the benefits gained.
• Review and update regularly. Something not working?  Need additional/different information?  Check your data and processes regularly to ensure you get the most out of your data.

What else has been helpful in getting the right data?

Enter Pivot.

As the Economist recently described in a special report, the amount of data we are creating is immense.  So much so that we can’t even store it all, let alone make sense of it.  Tools like Pivot, and many others, are designed to help us create true connections and insight.

I walked over to the post office across the street to mail it, and was surprised to see a table there that was manned by a temporary U.S. Census worker.  Above the table was a sign that said, “Census Questions Answered Here”.  I can never resist a census worker, so I wandered over and asked her what type of questions she was fielding.

None yet, I was told, because the census forms had not yet been delivered.  They were being proactive, I guess, so I won’t second-guess that.  So I asked what types of questions they were expecting.  Atop the basic questions about content and how to answer certain questions, the friendly census worker showed me her various language forms, 60 or more, with instructions in languages ranging from Somali to Polish to Creole, and said that language was a likely issue.  So she was prepared for Problem #1, which was making sure that people could understand the forms.

Problem #2 was access.  This was a small town with significant “informal housing” by an immigrant workforce, so she mentioned that many residents live in informal housing such as garages, spare rooms, and other homes without formal addresses.  Making sure that all of these people receive census forms was a potential problem, so much so that they were actually hand-delivering the forms rather than mailing them.  So Problem #2 was making sure that everyone had access to a form, which is another reason to have a physical presence at the post office.

And finally, she mentioned the big picture.  Why are people receiving these forms?  Why should they fill them out?  The census worker was ready and prepared to talk to people about all the reasons to fill out a census form.

Of course, they were doing the same thing that we at Corona do, but on a larger scale.  Anyone can throw questions up on survey monkey or zoomerang and generate numbers.  The key, though, and the hard part, is to generate accurate numbers and relevant numbers that truly answer your questions.   On surveys, you do this by making sure that you’re reaching a representative portion of the population, by making sure they understand what they’re being asked, and by maximizing their likelihood of responding.  On a census, you do this by making sure that you reach everybody, by making sure they understand what they’re being asked, and by maximizing their likelihood of responding.  And you do this via good distribution and by explaining the value of what you’re doing.

Kudoes to the people at the Census Bureau for their hard work in making sure that the government – and Corona Insights – has good census data to analyze for our clients.

What do we do with Census data once it is released?  Check out our data analytics practice to find out some of the ways we utilize census data.

Baseball and statistics are inseparable.  There seems to be a stat for every aspect of the game from the classic batting average, ERA, and strikeouts, to the more complex “sabermetrics” BABIP, BsR, and EqA.  Compiling the voluminous data to create these statistics is a monumental task, but making use of the numbers is a greater challenge.

Bloomberg, known for their financial analysis, is hoping to use their expertise to help major league baseball teams.  Bloomberg is focusing on determining trends in order to predict future performance.  Major league teams already compile much of the same information, but the advantage of the Bloomberg system is the speed at which teams can access the data and having one repository for all stats.  The data is not limited for distribution to major league teams; Bloomberg also has a similar product available to fantasy baseball players.

One more way analytics can provide a competitive advantage.  In this case, literally.  The three of us just hope our team, the St. Louis  Cardinals, are making full use of it.

Some of the trends we noted last year – economic pressures, instantaneous research – will continue and become even more pronounced. Here we offer some additional thought specifically on market research and analytics.

• As companies seek answers to their important questions, they will first turn to the data they already have. Using data analytics to squeeze every last drop out of their ever growing data will be the most cost effective way for many to get their answers. (see a related Business Week article here)
• Related to the economy and overall cost cutting/justifying, the need to have demonstrable results will be even more important. In research, this will mean producing insights that drive profitable action. For broader strategy, this will mean setting (or keeping) a course that drives the organization forward.
• While we noted it last year, the need for instant feedback is growing even stronger (one sign of this is the major search engines indexing social media sites like Twitter). While the need has always been there, newer technologies continue to make getting it even easier. Mining online conversations and any “buzz” will become commonplace for all companies.
• As companies and organizations seek ever better information to inform their decisions, surveys, especially using online panels, are becoming more and more popular for their speed and cost effectiveness. However, as we continue to ask people to take an ever greater number of surveys, the cost of straining survey takers is becoming greater – greater burnout, lower data quality, and professional respondents, to name a few. (see our other posts on the topic here and here)
• One reason for the popularity of monitoring social media and online surveys is their speed. Real-time feedback through quick measurements is crucial for altering strategy and achieving demonstrable results.
• All the data in the world is useless unless it’s properly conveyed. From the quick overview of key metrics using data “dashboards” to telling the deeper story, clients and end users will demand to get the most out of their research. Deciding how to convey findings, tell the story, and do so efficiently will be crucial. (See a related article on Quirks about how to get those stories from respondents…Article ID 20091206 free registration required)
• Similarly, online communities will continue to gain momentum. These communities build off the common experience of online networks (i.e. Facebook) and allow for richer information to be gathered over a longer period of time than a one-time focus group or survey can provide.
• Finally, will location based services such as Foursquare and Gowalla create new opportunities for research? Combined with mobile research, the ability for short surveys, instant feedback, all in a relevant environment may provide one more tool in the researcher’s toolbox.

These are just a few key ways that new tools and the changing nature of respondents are impacting how you find your answers.  Do you see other changes taking shape?

Tomorrow, we’ll have part 2 of our trends post on strategy and the social sector.

But for all the fun we have with numbers (not a typo), we always enjoy hearing the many other ways numbers impact our lives. What’s Benford’s Law? Do you have an Erdos number?

Check out the recent Radiolab episode, Numbers. Audio excerpt from the episode below.