Choosing the Right Metric
Misaligned goals, distorted behaviors, and a misguided sense of success... no, I’m not referring to college graduates. I’m talking about the problems caused by using the wrong metrics in your organization. You’ve probably seen examples like tracking average customer profitability and losing perspective on the variance in profitability or evaluating customer service reps on calls handled without regard for the quality of the experience. I’d like to offer up a quick-bake recipe for choosing the right metric.
Step 1: Set the context
Metrics generally serve one of two purposes. Start by understanding what you are trying to achieve.
1. Identifying problems. Defining the right metrics in this case requires you to do a little detective work: What is the data residue of a problem? What evidence can be found and how exactly does it show up?
2. Measuring performance. The right success metrics need to focus on measures that can be controlled and where improvement in the number is unabiguously a good thing.
Step 2: Balance the four dimensions of a good metric
Lots of metrics fail in at least one of these dimensions. A few examples:
Common interpretation: We had a client who made a distinction between "leads" and "prospects" in their marketing organization. Prospects had theoretically expressed more interest in the service through their actions. Unfortunately the line between leads and prospects was always hard to decipher and the definitions were hard to communicate. On a related note, we got a kick out of Tom Davenport’s (author of "Competing on Analytics") assertion that a company competing on analytics needs to "invent proprietary metrics for use in key business processes." There is nothing inherently wrong with "invented proprietary metrics" but it sounds like something that is designed to confuse anyone outside of the inner sanctum.
Actionable: Metrics are frequently too broad for the impact that a particular group can have. Customer satisfaction is a popular dashboard staple, but it is hard for most managers to see how they can have a significant impact on the number.
Accessible, credible data: Sometimes the most valuable and obvious metrics are frustratingly hard to track. In the web analytics world, unique visitors is important to know, but user deletion of cookies has thrown a wrench into the works.
Transparent, simple calculation: Top NFL agent Leigh Steinberg says of the famous quarterback ratings metric:"Other than one attorney in our office, I am unaware of a single human being who has the capacity to figure a quarterback rating." I don’t know what kind of art majors he hires, but all they need to do is use the simplified formula: (83.33 * Comp %) + (4.16667 * Yds per att) + (333.333 * TD pct) - (416.667 * INT pct) + 25/12.
(Want a little validation of this framework? Avinash, respected web analytics guru, just published a post with "Four Attributes of Great Metrics" and he landed on a strikingly similar set of four: 1) instantly useful (i.e. actionable); 2) relevant (i.e. common interpretation); 3) timely (i.e. accessible); 4) uncomplex (i.e. transparent and simple).)
Step 3: Avoid the metrics bugaboos
Finally, here are a few traps that I’ve seen in deciding on appropriate metrics:
Trending and distributions: Don’t always try to compress a metric into a single number. Often it is more revealing to show the metric across time or as a distribution to uncover variance.
Edge cases: There will always edge cases where a metric may not mean what you think it means. These situations are worth understanding, but you shouldn’t allow the perfect to be the enemy of the good.
Setting goals: Could you hold someone accountable for this metric without them throwing out a half-dozen reasons why it doesn’t make sense? It’s a decent test of the value of the metric.
Self-serving: Be careful that you don’t select metrics simply because you know they’ll make you look good.