Jerry Z. Muller - “The Tyranny of Metrics”
Starts off very strong against metric fixation but at some devolves into an argument against transparency.
Metric fixation
We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through ‘transparency.’ But the identification of accountability with metrics and with transparency is deceptive. Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement,… The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement.
Goodhart’s Law
Any measure used for control is unreliable. When a measure becomes a metric, it ceases to be a good measure.
To put it another way, anything that can be measured and rewarded will be gamed.
What and how to measure?
1. What kind of information are you thinking of measuring?
If the measured entity is activity that can be influenced by the act of measuring, it might be counter-productive to use metrics.
Rewards and punishments skew measurement validity.
2. How useful is the information?
The fact that something is measurable doesn’t make it worth measuring. If the metric is not what we want to know and it isn’t a proxy for what we want to know, it’s not worth measuring.
3. How useful are more metrics?
Measured performance identifies outliers like fraud but it isn’t useful in distinguishing between average and good performance. Make sure that the marginal the cost of measuring doesn’t exceed the benefits.
4. What’s the cost of not relying on standardized measurement?
An alternative might be surveying the clients.
5. To what purposes will the measurement be put? / To whom will the information be made transparent?
Internal monitoring of performance by practicioners is much better than external use of metrics for reward and punishment.
6. What are the costs of acquiring the metrics?
Often it’s expensive to collect and process data. Analysis also takes time and effort. Those costs are themselves hard to measure.
7. Ask why the people at the top of the organization are demanding performance metrics?
Demand sometimes floats from the ignorance about the institution that is managed.
8. How and by whom the meausures of performance developed?
Metrics are not effective when they are imposed from above. Instead, they should be developed from the bottom up. A representative group of practicioners needs to be part of both developing the metrics and analyzing them.
9. Remember that even the best measures are subject to corruption or goal diversion.
Incentives sometimes are in conflict with goals behind gathering metrics.
10. Recognizing the limits of the possible is the beginning of wisdom.
Not everything that can be measured, can be improved. Not everything that can be improved, can be measured. Transparency may make a situation more salient without making it more soluble.