After watching this video, you will be able to identify the limitations of vanity metrics,
describe the value of actionable metrics, and
list examples of actionable metrics.
Beware vanity metrics.
Vanity metrics are good for feeling awesome, but bad for taking action.
For example, consider the metric of “hits” to a website.
You might proclaim, “This is awesome, we got 10,000 hits to our website!”
But… what does a hit mean?
What does a hit represent?
Did one person nervously click 10,000 times?
Did 10,000 people click once and go away?
You don’t know.
You don’t know what action you should take next because you don’t know what action
drove the visitors to your website in the first place and so, you don’t know what
action to take next.
The number of clicks alone is not an actionable metric.
It indicates activity, but you don’t know if it’s good activity or bad activity.
In most cases, it’s not a very helpful metric.
Instead, you want to use actionable metrics.
Consider this: imagine you add a new feature to your website, and you initially introduce
it using A/B split testing in which 50% of the customers in group B see the new feature
and the other 50% in group A don’t.
A few days later, you compare the revenue you’ve earned from each customer group,
noticing that group B has 20% higher revenue per customer.
Think of all the decisions you can make.
Obviously, you’ll want to roll out that feature to 100% of your customers to start
increasing revenue from all of them.
You might continue to experiment with more features like this one.
Now you have a cause and effect.
You know what you did and you can take action to get more or less the same desired outcome.
Realize that you also probably discovered something that’s particularly valuable to
your customers.
This is the power of actionable metrics; so, make sure your metrics are actionable.
Here are some examples of actionable metrics.
These come from Eric Ries, author of The Lean Startup.
Reduce time to market for new features.
It’s very important to be able to get new features out to delight your customers as
quickly as possible.
Increase overall availability of the product.
It doesn’t matter what new features you introduce if the product isn’t available
for your customers to use them.
Reduce the time it takes to deploy a software release.
This is part of the first metric but focuses on deployment once the release is ready to
ship.
Increase the percentage of defects detected in testing before production.
It’s very important to detect defects as early as possible.
Another way of looking at this is reduced production-level defects.
Make more efficient use of hardware infrastructure.
This directly affects your cost of goods sold.
If you reduce costs, it means more profit, so efficiently using the resources you already
have is something you can measure and act on.
And providing performance and user feedback to the product manager quickly.
Having information when it matters is important for making informed decisions.
At her 2017 talk entitled “Tools Won’t Fix Your Broken DevOps,” Nicole Forsgren
identified what she considers the top four actionable metrics.
The first is mean lead time.
How long does it take for an idea to get to production?
From the time your stakeholder asks for the new feature, how long does it take that feature
to get into their hands?
The second is release frequency.
How quickly can you release things?
Hopefully, as quickly as you need and no sooner.
You don’t want to be disrupted by a competitor and take a long time to respond.
The third is change failure rate.
When you push things out, how often do they fail?
You want to make sure that, while you can release quicker, changes aren’t failing
when they are deployed.
Speed is meaningless if it destabilizes the system.
The fourth, mean time to recovery.
How long does it take when something does fail to recover from the failure?
Instead of being concerned with mean time to failure, you want to be resilient to fail
and recover quickly when it happens.
In this video, you learned that vanity metrics may be appealing at first glance but offer
limited actionable insights.
Actionable metrics provide meaningful ways to measure your process and work toward goals.
DevOps actionable metrics include mean lead time, release frequency, change failure rate,
and mean time to recovery.