After watching this video, you will be able
to identify the limitations of vanity metrics, describe the value of actionable metrics,
and list examples of actionable metrics. Beware vanity metrics. Vanity metrics are good for feeling awesome,
but bad for taking action. For example, consider the metric of “hits”
to a website. You might proclaim, “This is awesome, we
got 10,000 hits to our website!” But… what does a hit mean? What does a hit represent? Did one person nervously click 10,000 times? Did 10,000 people click once and go away? You don’t know. You don’t know what action you should take
next because you don’t know what action drove the visitors to your website in the
first place and so, you don’t know what action to take next. The number of clicks alone is not an actionable
metric. It indicates activity, but you don’t know
if it’s good activity or bad activity. In most cases, it’s not a very helpful metric. Instead, you want to use actionable metrics. Consider this: imagine you add a new feature
to your website, and you initially introduce it using A/B split testing in which 50% of
the customers in group B see the new feature and the other 50% in group A don’t. A few days later, you compare the revenue
you’ve earned from each customer group, noticing that group B has 20% higher revenue
per customer. Think of all the decisions you can make. Obviously, you’ll want to roll out that
feature to 100% of your customers to start increasing revenue from all of them. You might continue to experiment with more
features like this one. Now you have a cause and effect. You know what you did and you can take action
to get more or less the same desired outcome. Realize that you also probably discovered
something that’s particularly valuable to your customers. This is the power of actionable metrics; so,
make sure your metrics are actionable. Here are some examples of actionable metrics. These come from Eric Ries, author of The Lean
Startup. Reduce time to market for new features. It’s very important to be able to get new
features out to delight your customers as quickly as possible. Increase overall availability of the product. It doesn’t matter what new features you
introduce if the product isn’t available for your customers to use them. Reduce the time it takes to deploy a software
release. This is part of the first metric but focuses
on deployment once the release is ready to ship. Increase the percentage of defects detected
in testing before production. It’s very important to detect defects as
early as possible. Another way of looking at this is reduced
production-level defects. Make more efficient use of hardware infrastructure. This directly affects your cost of goods sold. If you reduce costs, it means more profit,
so efficiently using the resources you already have is something you can measure and act
on. And providing performance and user feedback
to the product manager quickly. Having information when it matters is important
for making informed decisions. At her 2017 talk entitled “Tools Won’t
Fix Your Broken DevOps,” Nicole Forsgren identified what she considers the top four
actionable metrics. The first is mean lead time. How long does it take for an idea to get to
production? From the time your stakeholder asks for the
new feature, how long does it take that feature to get into their hands? The second is release frequency. How quickly can you release things? Hopefully, as quickly as you need and no sooner. You don’t want to be disrupted by a competitor
and take a long time to respond. The third is change failure rate. When you push things out, how often do they
fail? You want to make sure that, while you can
release quicker, changes aren’t failing when they are deployed. Speed is meaningless if it destabilizes the
system. The fourth, mean time to recovery. How long does it take when something does
fail to recover from the failure? Instead of being concerned with mean time
to failure, you want to be resilient to fail and recover quickly when it happens. In this video, you learned that vanity metrics
may be appealing at first glance but offer limited actionable insights. Actionable metrics provide meaningful ways
to measure your process and work toward goals. DevOps actionable metrics include mean lead
time, release frequency, change failure rate, and mean time to recovery.