spot-the-differences

Try this fun exercise to pick out 5 visible differences. Spotting statistically significant differences in large data sets isn’t as simple, but our staff people are experts at rigorously applying these tests.

Making data-driven strategic decisions frequently involves understanding differences.  For example, are there differences in public opinion, demographics, or the way people behave? Are there differences among groups of people, between two points in time, or differences from one program to another? Many of our clients ask for help measuring differences and sparking insights from the results. Using data to understand key differences can help leaders make smart strategic decisions, such as identifying their target audience or determining which programs are most effective.

Although it might seem easy to measure and understand a difference, the reality is that most questions are complicated and datasets are often incomplete.  For practical and budget reasons, we frequently need to answer questions about differences with data collected from a small sub-sample of the whole group. There is a suite of mathematical procedures that can help determine if the differences between two sub-samples is statistically significant, and our staff are experts at rigorously applying these tests. However, we are always mindful that the most important question in research is not whether a result is statistically significant, but rather if it is meaningful and useful.  Statistical tests can help inform, but should not replace, strategic decision-making.

Several research project elements contribute towards the likelihood of finding a statistically significant result. First, vary large sample sizes are usually statistically different from one another, even if the difference is trivial. For example, a $50 difference in annual income between two groups of people might be statistically significant, but it is unwise to make a strategic decision based on the outcome of this one test, considering the difference of $50 per year is practically insignificant. Second, the variety of answers that survey respondents provide will influence the results of statistical tests.  A lot of variation or a few outliers (data that is very different from the average, such as one player scoring 100 points in one basketball game) can produce results that are not statistically significant, even if there is a clear and practical difference. Last but not least, statistical tests are defined by a level of confidence that is arbitrarily set by the researcher.  While industry standards exist, in the real world there is no practical difference between two finding that are just above and below this arbitrary significance level.

The Corona staff is capable of applying advanced statistical tests to help reveal differences in opinions and behavior, but we also use critical thinking, real-world experience, and common sense to help our clients illuminate the insights they need to make truly strategic decisions.