Survey design can be a complex and nuanced process.  We have made a multitude of posts on the subject, including asking the right people to participate, and how to ask the right questions, but one area we don’t talk about a lot is how the answers you provide in a survey can influence your results.  This is less of an issue in a verbal survey (such as a telephone survey) since interviewers are able to Answeraccept answers that aren’t directly given, but this can be a critical issue in online surveys and mail surveys since respondents can read all of the answers available to them before determining how they want to respond.   Here are a few of the decisions you may need to consider when designing questions for these types of surveys:

None of the above and Other

In cases where you provide respondents with a list of options to choose from, it is important that every respondent is able to answer the question. For example, consider the following hypothetical question for a survey of college students:

  • Which of the following best describes your year in school?
    • Freshman
    • Sophomore
    • Junior
    • Senior

In many cases, that question may be perfectly reasonable.  However, what if a student is in their fifth year?  Or what if the school has non-traditional students?  In these cases, respondents will likely either select a random answer or simply abandon your survey.  Neither of these possibilities is ideal, so it may be useful to include a “none of the above” option or an “other” option so that everyone can feel comfortable selecting an answer.

Our recommendation: Be comprehensive in your answer options, and give “other” and “none of the above” choices any time you aren’t sure you have all of the possibilities covered.

Don’t know

Another way in which this issue manifests itself is with regard to providing a “don’t know” option.  For example, consider the following hypothetical question:

  • How would you rate your opinion of Organization X?
    • Very positive
    • Somewhat positive
    • Neutral
    • Somewhat negative
    • Very negative

By not including a “don’t know” option, you are in effect forcing people to make a decision about the organization.  However, if someone in fact doesn’t know enough about the organization, they may potentially choose an answer that they don’t truly believe or abandon their response altogether.

On the other hand, including a “don’t know” option in your survey may not always be ideal either.  While a portion of your respondents may legitimately need to choose that answer, a number of others may choose that answer simply because it’s an easy option that doesn’t require a lot of thought.  If you are confident that all respondents know the organization well, including a “don’t know” option may actually reduce the quality of your data due to respondents taking the easy way out.  (As an aside, this is a similar issue as including a “neutral” option, but we’ll save that for another blog post!)

Our recommendation: Leave “don’t know” off if you believe that respondents should be able to form an opinion about the question, but include the option if there’s a good chance that many respondents may truly not have an answer.

Prefer not to answer

Similar to the above, when asking questions about sensitive topics, it can be beneficial to allow respondents to choose not to answer the question.  Household income, for example, is often valuable to use as a segmentation criteria about consumers, but many survey respondents are reluctant to share such information.  Including a “prefer not to answer” option may make respondents more comfortable with participating in your survey.  However, even more so than the “don’t know” option, suggesting that it’s OK to not answer the question will undoubtedly increase the number of respondents who do so, dramatically increasing the amount of missing data for the question.

Our recommendation: Put sensitive questions toward the end of the survey so that respondents are already comfortable with answering, but don’t include a “prefer not to answer” option unless the question is particularly objectionable for respondents.  Instead, consider allowing respondents to simply skip the question as described below.

Not forcing responses

That brings us to our final topic which is, in a sense, an answer option that you can’t even see: not requiring responses to survey questions in an online survey.   Even if you’ve tried to be comprehensive in your answers and provide alternate answers such as those described above, there is always a possibility that a respondent may simply be unwilling to answer a question.   If you require them to do so, they will likely either choose a random response or just abandon your survey entirely.  If having missing data will cause problems with your survey (for example, if meeting some criteria is required to direct respondents to appropriate questions) or in your analysis (for example, in a segmentation group you will use in your analysis), it may be necessary to require a response, but be sure to consider the implications of doing so.

Our recommendation: Use forced responses sparingly.  It’s usually better to allow someone to skip a question than it is to have data for a question that isn’t accurate.