Metrics are a powerful thing, but as Spiderman knows, “with great power comes great responsibility.” Bad metrics and even good metrics that fall into the wrong hands, can be used in ways that can harm the organization and its people.
Any metric that is used to reward or punish people, is a bad use of metrics. Wait, metrics to reward people are bad?! YES! Because if you are rewarding some you are punishing others. Use metrics to improve company performance, not as a behavior-shaping tool. When companies use metrics to drive behavior, unforeseen side effects crop up. (gaming the system, blame, frustration)
What are bad metrics? Bad metrics are those that don’t connect to an outcome. They simply measure what is happening. Some call these vanity metrics because they might make you look good, but there’s no depth to them.
Clicks. An example of bad, vanity metrics is “clicks” when measuring web site performance. Clicks are a start, but if you are not converting clicks to sales, the clicks won’t do you much good.
% done. Does anyone really know what % done they are? When you have been 80% done for the past 4 months, we all know the % done is not a useful metric.
Insignificant digits. Remember in algebra class when you learned that your answer cannot have more significant digits than your input? The average of 3 and 4 can’t be 3.5? Lookout for this in metrics that show you in excruciating precision that was calculated from numbers that were guesses.
Ratios. Blending numbers into an index or giving me a ratio between numbers that have nothing to do with each other is just bad practice. I had a Six Sigma black belt tell me that we needed to measure the ratio between bugs in test and bugs in production. This is what they call the “escaped bug %.” If I had 50 bugs in test and 4 bugs in production, then he calls that an 8% escaped bug %. So what happens when I have 50 bugs in prod and we only find 4 in test? Are my escaped bugs at -75%? And do we want this number to go up or down? Bottom line is, it makes no sense to compare test bugs to prod bugs so just don’t do it.
Indexes. Indexes are just as bad. I imagine people sitting in a room and trying to find a metric, but they can’t agree so they create an index. Indexes are like encryption algorithms, no one knows how they got the number or what it means. Metrics do not need to cover every possible measure you can think of. If you have any type of quality index, I guarantee no one knows what it means or how to affect it.
Analysis metrics. Metrics should not do the analysis for you. They should give you hints about where to look further. I see metrics dashboards that have all the information on them ‘in case someone asks”. Well, no one is ever going to ask because no one can read your dashboard! Make it simple, and then dig in where you need to.
What are good metrics? Good metrics drive good dialogue. Metrics should be a way of keeping a pulse on what’s happening and aligning people to what’s important. Metrics do not need to cover every possible measurement. Metrics should be an indicator, that’s all.
Impact-based. If your ultimate goal is to make money, there's got to be some metric that shows those dollars. I often hear teams concerned that they can't tie the dollars back to their work, and that the impact won't happen right away. That's ok, that's why we have leading indicators but don't stop looking at the actual impact. It's the business equivalent to taking your eye off the ball.
Simple. Keep it to a few simple metrics. 3-5 should be about right. You don't need to measure everything. Choose a few metrics that will let you know you need to dig deeper.
Connected to the work. People should be able to connect their work to the metrics. You may have several levels of metrics in order to accomplish this. OKRs are a great example of layered metrics. Employees have a right to know how their work affects the goals of the company and knowing this enables them to contribute in a meaningful way.
How have you used metrics in your work? What worked? What didn't? Let us know!
