1 00:00:07,960 --> 00:00:12,520 After watching this video, you will be able to identify the limitations of vanity metrics, 2 00:00:12,520 --> 00:00:14,721 describe the value of actionable metrics, and 3 00:00:14,721 --> 00:00:18,240 list examples of actionable metrics. 4 00:00:18,240 --> 00:00:20,910 Beware vanity metrics. 5 00:00:20,910 --> 00:00:24,660 Vanity metrics are good for feeling awesome, but bad for taking action. 6 00:00:24,660 --> 00:00:28,000 For example, consider the metric of “hits” to a website. 7 00:00:28,000 --> 00:00:31,730 You might proclaim, “This is awesome, we got 10,000 hits to our website!” 8 00:00:31,730 --> 00:00:34,120 But… what does a hit mean? 9 00:00:34,120 --> 00:00:36,060 What does a hit represent? 10 00:00:36,060 --> 00:00:38,890 Did one person nervously click 10,000 times? 11 00:00:38,890 --> 00:00:42,230 Did 10,000 people click once and go away? 12 00:00:42,230 --> 00:00:43,640 You don’t know. 13 00:00:43,640 --> 00:00:47,080 You don’t know what action you should take next because you don’t know what action 14 00:00:47,080 --> 00:00:51,929 drove the visitors to your website in the first place and so, you don’t know what 15 00:00:51,929 --> 00:00:54,010 action to take next. 16 00:00:54,010 --> 00:00:57,300 The number of clicks alone is not an actionable metric. 17 00:00:57,300 --> 00:01:01,420 It indicates activity, but you don’t know if it’s good activity or bad activity. 18 00:01:01,420 --> 00:01:05,830 In most cases, it’s not a very helpful metric. 19 00:01:05,830 --> 00:01:10,230 Instead, you want to use actionable metrics. 20 00:01:10,230 --> 00:01:14,530 Consider this: imagine you add a new feature to your website, and you initially introduce 21 00:01:14,530 --> 00:01:20,180 it using A/B split testing in which 50% of the customers in group B see the new feature 22 00:01:20,180 --> 00:01:22,920 and the other 50% in group A don’t. 23 00:01:22,920 --> 00:01:28,170 A few days later, you compare the revenue you’ve earned from each customer group, 24 00:01:28,170 --> 00:01:33,380 noticing that group B has 20% higher revenue per customer. 25 00:01:33,380 --> 00:01:35,500 Think of all the decisions you can make. 26 00:01:35,500 --> 00:01:39,939 Obviously, you’ll want to roll out that feature to 100% of your customers to start 27 00:01:39,939 --> 00:01:42,189 increasing revenue from all of them. 28 00:01:42,189 --> 00:01:45,549 You might continue to experiment with more features like this one. 29 00:01:45,549 --> 00:01:48,000 Now you have a cause and effect. 30 00:01:48,000 --> 00:01:53,730 You know what you did and you can take action to get more or less the same desired outcome. 31 00:01:53,730 --> 00:01:57,799 Realize that you also probably discovered something that’s particularly valuable to 32 00:01:57,799 --> 00:01:59,009 your customers. 33 00:01:59,009 --> 00:02:04,340 This is the power of actionable metrics; so, make sure your metrics are actionable. 34 00:02:04,340 --> 00:02:07,189 Here are some examples of actionable metrics. 35 00:02:07,189 --> 00:02:11,170 These come from Eric Ries, author of The Lean Startup. 36 00:02:11,170 --> 00:02:13,010 Reduce time to market for new features. 37 00:02:13,010 --> 00:02:16,980 It’s very important to be able to get new features out to delight your customers as 38 00:02:16,980 --> 00:02:19,560 quickly as possible. 39 00:02:19,560 --> 00:02:21,750 Increase overall availability of the product. 40 00:02:21,750 --> 00:02:25,300 It doesn’t matter what new features you introduce if the product isn’t available 41 00:02:25,300 --> 00:02:27,840 for your customers to use them. 42 00:02:27,840 --> 00:02:30,440 Reduce the time it takes to deploy a software release. 43 00:02:30,440 --> 00:02:34,866 This is part of the first metric but focuses on deployment once the release is ready to 44 00:02:34,866 --> 00:02:35,866 ship. 45 00:02:36,173 --> 00:02:40,640 Increase the percentage of defects detected in testing before production. 46 00:02:40,640 --> 00:02:44,630 It’s very important to detect defects as early as possible. 47 00:02:44,630 --> 00:02:49,340 Another way of looking at this is reduced production-level defects. 48 00:02:49,340 --> 00:02:52,120 Make more efficient use of hardware infrastructure. 49 00:02:52,120 --> 00:02:54,390 This directly affects your cost of goods sold. 50 00:02:54,390 --> 00:02:59,200 If you reduce costs, it means more profit, so efficiently using the resources you already 51 00:02:59,200 --> 00:03:02,690 have is something you can measure and act on. 52 00:03:02,690 --> 00:03:06,690 And providing performance and user feedback to the product manager quickly. 53 00:03:06,690 --> 00:03:11,416 Having information when it matters is important for making informed decisions. 54 00:03:12,546 --> 00:03:18,530 At her 2017 talk entitled “Tools Won’t Fix Your Broken DevOps,” Nicole Forsgren 55 00:03:18,530 --> 00:03:22,470 identified what she considers the top four actionable metrics. 56 00:03:22,470 --> 00:03:24,660 The first is mean lead time. 57 00:03:24,660 --> 00:03:28,300 How long does it take for an idea to get to production? 58 00:03:28,300 --> 00:03:32,290 From the time your stakeholder asks for the new feature, how long does it take that feature 59 00:03:32,290 --> 00:03:34,710 to get into their hands? 60 00:03:34,710 --> 00:03:36,500 The second is release frequency. 61 00:03:36,500 --> 00:03:38,420 How quickly can you release things? 62 00:03:38,420 --> 00:03:41,500 Hopefully, as quickly as you need and no sooner. 63 00:03:41,500 --> 00:03:46,640 You don’t want to be disrupted by a competitor and take a long time to respond. 64 00:03:46,640 --> 00:03:49,070 The third is change failure rate. 65 00:03:49,070 --> 00:03:52,120 When you push things out, how often do they fail? 66 00:03:52,120 --> 00:03:56,900 You want to make sure that, while you can release quicker, changes aren’t failing 67 00:03:56,900 --> 00:03:58,240 when they are deployed. 68 00:03:58,240 --> 00:04:02,570 Speed is meaningless if it destabilizes the system. 69 00:04:02,570 --> 00:04:04,870 The fourth, mean time to recovery. 70 00:04:04,870 --> 00:04:08,460 How long does it take when something does fail to recover from the failure? 71 00:04:08,460 --> 00:04:13,330 Instead of being concerned with mean time to failure, you want to be resilient to fail 72 00:04:13,330 --> 00:04:16,250 and recover quickly when it happens. 73 00:04:16,250 --> 00:04:20,910 In this video, you learned that vanity metrics may be appealing at first glance but offer 74 00:04:20,910 --> 00:04:23,323 limited actionable insights. 75 00:04:23,323 --> 00:04:28,800 Actionable metrics provide meaningful ways to measure your process and work toward goals. 76 00:04:28,800 --> 00:04:34,889 DevOps actionable metrics include mean lead time, release frequency, change failure rate, 77 00:04:34,889 --> 00:04:36,868 and mean time to recovery.