Principles of Metric Design - how to design metrics that matter

[September 11, 2022]

Folks, today, we're diving into a topic close to my heart: how to design metrics.

I have often seen people what they can measure influences what they measure! This is such a common occurrence that I think it is worth calling out the obvious stuff at the risk of being redundant.

What is a metric?

I feel this question is necessary to get everybody on the same page. Let's distill down the concept of metric to the very core of it, and remove anything that does not qualify as the core(I often call this exercise trimming down to essentials, more on this may be next time).

To me,

• Metrics can be anything that provides meaningful information used to make better decisions.

• Metrics reduce uncertainty and are not necessarily a point measure. i.e. They take you from a higher state of uncertainty to a lower state of uncertainty.

• Metrics are tools for learning and improving, not just for monitoring or justifying.

The art and science of measuring involve determining what information would make a significant difference in decision-making and finding practical ways to obtain that information. (so meta! I know, but bear with me, this is necessary for decision-making folks)

Source

The Essence of a Good Metric: Relevance, Precision, Simplicity, Flexibility

Relevance of a metric (Why do you want to measure what you want to measure)

Simply put, this asks(very simple but very important question),

What would change if this metric goes up, or does not go up, or goes down, or does not go down? What happens then? What decision would you take or won't take, for that matter?

Ask these questions and you'll be surprised at the variations in answer! This is the beginning of metric you creating impact before even arriving at metric. Onboarding folks and getting alignment on decisions and outcomes is critical when building for impact.

Alternatively,

• a metric is relevant if it is directly connected to specific business objectives, goals, or performance indicators.

• It answers the question, "Does this metric provide meaningful information that helps understand, evaluate, or predict aspects of the business that matter?"

• A relevant metric has a clear relationship to the outcomes or processes it intends to measure. It's about ensuring that the metric aligns with what is significant to the business or project at hand.

Precision in metrics is key (Are you measuring the consequence w.r.t. to the change that you introduce in the system? How good/accurate are you?)

• A good metric is precise and sensitive to the levers being evaluated.

• A precise metric zeroes in on the levers you can pull, giving clear signals amidst the noise.

It refers to how accurately and consistently the metric can measure the performance or status of what it is intended to track.

Precision answers questions like, "How close can repeated measurements get to each other, and how finely tuned can our observations be?"

Precision v/s Relevance of a metric.

A metric can be precise but not relevant if it accurately measures something that doesn't align with business objectives. Conversely, a metric might be considered relevant because it aims to measure something important, but if it's not precise, it can lead to misguided conclusions and decisions. Therefore, the best metrics are both relevant (they measure what matters) and precise (they measure it well).

e.g. “Profit” is usually an ineffective metric to evaluate product changes. Note the context of evaluating product changes. It is of critical importance here. While ”Profit” might be an ultimate goal, in no way, it is effective to evaluate product changes as it depends on far too many factors other than the levers under consideration; it reduces the metric’s signal-to-noise ratio even if experimentation can estimate the incremental impact in an unbiased way.

It is better to choose a metric close to the levers we control, and that we know contributes to the ultimate business goal.

Often a good strategy is to normalize a system’s main output by its inputs so that the metric is responsive to the levers under control but invariant to other factors.

In my experience, precision is given far too much importance than relevance. Know that relevance drives the direction of business decisions, and precision augments the magnitude.

Simplicity of a metric.

Let me call this one out.

Simplicity isn't about dumbing down; it's about clarity and elegance.

The most powerful metrics are those that speak a clear language.

A good metric is simple. It should be transparent and easy to understand by a non-technical audience.

A metric to evaluate a complex system should not add to the complexity, eg. by using black-box models to estimate abstract value. (I have often seen this to be the case, more than I would have liked to see).  While model metrics have their value on iterative feedback of the implementation, they should not act as a barrier to effective communication to business.

Complexity, often seen as thoroughness, can become a barrier to understanding and action.

It should ideally also be inexpensive to calculate, eg. relying on readily available transactional data rather than external data sources or user feedback.

By adopting a minimalist approach, you craft metrics that are transparent and easily digested by anyone.

Note that, a simple metric does not mean simplistic; it means efficiently conveying the heart of the matter.

Advocate metrics that maintain a balance between clarity and depth.

A well-designed metric should convey the necessary information as simply as possible without undermining its relevance, accuracy, or completeness.

Actionability of a metric

We often measure things because we can, not because we should.

Your metrics should be actionable, lighting up the path to decision-making, not just decorating your dashboard.

Metrics should not exist in a vacuum; they should be directly linked to the strategic objectives of the business, providing a compass that guides decision-making and strategy.

Also, don't shy away from the uncertainty of a metric. A good metric informs action not despite uncertainty but because of the clarity it provides about this uncertainty.

Finally, a good metric is Flexible.

Flexibility means your metrics aren't set in stone but are living, breathing entities that grow with your business. Your business morphs; your metrics should too.

Metric design is never complete but an iterative process that must respond to changing business needs. Embrace this iterative process.

The journey of metric design is perpetual.

Start with a metric, test its waters, gather feedback, and refine. It’s this cycle that keeps your metrics in tune with the rhythm of your business.

Build your metrics with the future in mind, allowing them to evolve without losing their core essence. We can acknowledge this by building in scope to configure and extend our metrics.

This alignment is not just about numbers; it's about creating a shared language and understanding across your team, sparking dialogue, and fostering a unified direction.

Did you answer how to design a metric?

Yes, we now have a set of pointers that allow us to evaluate a good metric. Let's get a quick picture of the entire post.

• Know why are you building what you are building, and that includes metrics. Have clarity and alignment of the decision/action/inaction before the metric design.

• Prioritize relevance over precision for a metric.

• Metrics reduce uncertainty and are not necessarily a point measure. i.e. They take you from a higher state of uncertainty to a lower state of uncertainty.

• Choose a metric close to the levers we control.

• A metric to evaluate a complex system should not add to the complexity.

• Metric informs action not despite uncertainty but because of the clarity it provides about this uncertainty.

• Metrics evolve! They are not set in stone. This means the definitions of might metrics might not necessarily change, but the decisions they support could/should evolve with business.

Summary

Remember that the ultimate goal is not to measure everything but to measure what truly matters.

This isn’t just about tracking numbers; it's about sculpting a narrative that drives your business forward.

So, go forth and craft metrics that matter – metrics that illuminate the path to insight, action, and success.