Photo credit: Stephen Dawson

This blog post is dedicated to a trendy word in the social sector, “metrics.” Don’t get me wrong – metrics are important, and I’m often asked to help organizations define what metrics matter most to them.  But in this post, I want to outline some of the key challenges or “traps” that we can fall into as nonprofit professionals around metrics.  This is the first in a three-part blog series dedicated to metrics.  In the next two posts, I will share a framework that I highly recommend using when aligning metrics to your organization-wide goals, and clarify the distinction between two other buzzwords, we know and love, “monitoring” and “evaluation.” 

Understanding the Metric Trap

Every organization wants to talk about metrics.  What exactly are metrics?   Metrics tell you how well you’re performing compared to expectations.  Some common phrases you might here in the office that let you know metrics are the topic du jour are:

  • “To understand what’s happening around here, I need to see a dashboard.”
  • “Let’s highlight these numbers red, yellow or green.”
  • “We’re going to review these indicators every month.”

While understanding metrics is important, I have a few cautions to look out for before diving into defining them.  There are some traps that we often fall into when focusing on metrics, and if we’re aware of them we can avoid them. 

  • The time-frame trap:  For many organizations, your results will take many years to manifest. In other words, the results we see today are a result of the actions we took as an organization many months or years ago. Monitoring these kinds of long-term results on a month-to-month basis might be frustrating, to say the least. Let’s imagine that we’re running a rolling admission counseling program that individuals typically need 12-months to complete. Last month our metric “number of new graduates in month” was dismal – is that because we lost students? Or was that because we were at capacity a year ago and could only admit a few new participants?
    • One way to deal with this trap is to create a Theory of Change, which maps the relationship between shorter-term and longer-term outcomes. It might only be appropriate to monitor the shorter-term outcomes monthly. Those longer-term outcomes could be reviewed once a quarter or once a year. 
  • The linearity & uncertainty trap:  All possible consequences of your organization’s work can’t be known in advance. Particularly, if you’re a new organization or are exploring a new service, avoid the temptation to say “These are the primary results of our program,” if you don’t have the experience, beneficiary feedback, or research to back this up.
    • One way to avoid this trap is to focus on collecting less structured data during the piloting stage of a new program or service. At this stage, ask open-ended questions about participants’ experiences like, “How have you benefited from this?” and “Was your experience with us what you expected? Please explain.”
  • The indicator blindness trap: Ultimately tracking metrics often reiterates unchecked assumptions. Let’s imagine that in that 12-month counseling program we mentioned above, a metric is “x% of active participants completing at least 3 sessions per month.” An assumption inherent in this metric is that weekly sessions are the right frequency to help participants make meaningful progress. But what if it was actually more effective to have a deeper, 3-hour session once a month instead?  If you’re a new nonprofit or doing new work, the first question to ask is “How sound is our Theory of Change?”  Performance metrics are only relevant if you have reason to believe your theory is solid and works well.
    • One way to avoid this trap is to consider your Theory of Change as a working document. Keep an eye out for opportunities to refine it, and revise it every 6-12 months as you learn more.

Stay tuned for the next blog, where we will dive into A Framework for Investigating Metrics.

This post by Paul originally appeared on the Coeffect.co website.