25 Years of Change – Survey Methodologies

As Corona Insights arrives at our 25-year anniversary, we’re going to take a look back at the world 25 years ago and examine what has changed in the world of research.  This blog post will focus on the evolution of one of our most common research tools – public surveys.

Survey methodology has experienced huge seismic shifts over the past 25 years.  We’ll concentrate here on the most broad (and often most methodologically challenging) types of surveys – surveys of the general public.

So why have survey methodologies changed?  To a great extent, it’s the result of changes in society, driven in part by technology and in part by changes in how people interact with the world.  In large part, these changes have given us different opportunities and challenges in reaching the people we want to hear from.

Reaching the People

Given that the goal of a valid public survey is to hear the opinions and experiences of the population, it’s important that the survey responses represent the population.  And that generally means ensuring that our surveys are distributed to a representative sample of the population, or at least as close as one can get.  And we have to do so in a way that also welcomes them to respond.

We recently examined a sample of large-audience surveys that we conducted in the early years of the company, versus a sample from our most recent year of business.  Here are our results. Notice a difference?

Twenty-five years ago, the public survey world was dominated by telephone surveys.  The reason for this was simple:  we could identify where people lived based on their phone number (which helped us do proper sampling of the population), and people would readily answer the telephone and talk to us.

The world has changed, though, and we’ve had to change with it.  First, cell phone use exploded, eventually supplanting land lines for telephone communication.  At the same time, phone number portability was established, meaning that we could no longer definitively identify a person’s general location via their phone number.  A person with a 303 area code may no longer live in Colorado, and a person with an out-of-state area code may be a longtime Colorado resident.  So it has become more and more difficult to draw representative samples of the population for phone surveys, except under certain circumstances that are not common in our public surveying work. 

Atop those technical challenges, trust challenges have risen dramatically.  The public has become very wary about telephone calls from unknown parties, and with good reason due to the rampant rise of robocalls and phone scammers.  People are much less likely to even answer their phones these days, much less share their thoughts and opinions over the telephone.

Over the past 25 years, these forces have conspired against telephone surveys.  Today, we seldom conduct surveys via telephone, and only if certain conditions are met.  When used, they’re usually done in conjunction with other methods as a follow-up to targeted audiences.

On the other hand, new technologies have come aboard as a replacement to phone surveys.  In particular, the shift to an online world has created opportunities to conduct surveys via online methods.  Over the years, these have evolved into two main approaches:  panel surveys where pre-recruited people volunteer to take surveys for payment, and email surveys where a pre-identified audience with a prior relationship with the client are recruited to an online survey platform via an email invitation.  These different approaches are used in different situations based on our judgments.  (Another means of gathering data is open-link online data collection where a survey link is shared to the general public, but that approach is ’public input’ rather than a public survey, and will not be addressed here.)  These methods have evolved to be efficient and generally accurate if conducted properly.

There are certain tradeoffs to conducting surveys online, though, primarily in the arena of pure statistical rigor.  Sometimes when a survey needs to be rigorously defensible from a statistical standpoint, online methods with nonprobability samples shouldn’t be used.  Phone surveys are still possible, but challenging for the reasons discussed above.  So what do we do in that situation?

Enter the re-emergence of a historically strong tool – the classic mail-based survey where paper survey invites are physically mailed to households.

Mail surveys were the primary form of public surveying prior to the 1990s deregulation of phone companies.  They then fell out of favor for a couple of decades as deregulation allowed phone surveys to become a cheap and fast alternative.  But now with the decline of phone surveys in the era of call blockers and spammers, look who’s back.  If you need a strong and defensible general public survey, mail surveys have again risen as a great option, and they’ve once again become a trusted sampling method for its accuracy and defensibility.  As a result, they’ve been revived to become a very common survey mode.  Additionally, for some research needs, a variant of mail surveys, the ‘push to web’ approach can be used.  In this variant, invitations to an online link are physically mailed, either in conjunction with a paper survey or not.  Where appropriate, this approach can combine the rigor of mail surveys with the ease of online surveying.

New Approaches Require New Design

So we know that the way we conduct surveys has changed dramatically over the past 25 years.  This also has other impacts, and perhaps the most surprising is how we design survey questions.  One might think that the questions themselves should be independent of how we gather data; after all, the survey questions are the crux of the process.  However, the method of collection changes how we can ask questions.

Twenty-five years ago, when most of our surveys were conducted via telephone, we had to be very efficient with time, because phone surveys needed to be quick and crisp.  Additionally, our questions had to be simpler because we relied on people to quickly come up with their responses during a conversation.  With the mail and online survey modes that are prevalent today, we have the luxury of time.  People don’t have a social pressure to respond quickly to a live person, so we can ask more complex questions if we wish.  This is a nice, unexpected advantage of the newer modes of collection.  Additionally, our surveys can be slightly longer since the written format allows people to complete the survey at their leisure. The other big change is question format, particularly with online surveys.  Respondents often complete online surveys on their phones now, rather than on a computer or tablet, so our question formats must be designed with small screens in mind.

And The Technical Stuff

At the end of the data collection process, we have to examine how representative the survey responses are and develop statistical weightings to correct for any skewing of respondent demographics.  Then we do our analysis and reporting.  Survey weighting methods have evolved to address challenges of sampling cell phones, and also the inherent skewing of survey panels, and are more technical than we need to cover in this blog.  We have also evolved more complex sampling and weighting methods for mail surveys as new data resources allow us to optimize our approach.  Suffice to say that corrective statistical weightings have evolved alongside changes in sampling and data collection.

So a lot of things have changed in the surveying world over the past 25 years.  Is anything the same?  Well, analysis and reporting often use different tools now as more sophisticated analysis software has been developed and released, but the general process is the same.  We look for key findings and we report them.  Some things never change.

So What Does This Mean To Our Clients?

All in all, we can summarize the survey world’s changes in the past 25 years as an evolution in how we sample, collect, and analyze data, but the core goal is still the same.  We are gathering the opinions of large groups of people in order to draw conclusions about an issue.  Our clients still come to us with their questions, and we still answer them, but we use different techniques to accomplish the same process.  The world continues to change, and we’ll be interested in evolving our work to accommodate those changes over the next 25 years.