Measure DevOps Metrics That Matter
May 19, 2020

Elysia Lock
Flux7

As the old adage goes, what gets measured gets done. Measurement is the key enabler to any DevOps transformation and yet it's an oft-neglected aspect of projects. Organizations struggle to get beyond the starting blocks when learning how to measure DevOps. As a result, in today's blog, I will share important DevOps metrics your team can use to get started on your journey to measuring positive change.

A common challenge I frequently hear from DevOps teams is that there is no clear starting place, no benchmark to begin measuring from. They ask, "If you don't know where you are starting, how do you measure improvement from that place?" My advice is to simply start. Start measuring and your yardstick will appear. You will see and be able to show improvement over time. Instead of measures like, "the release made it out to production," you'll start being able to report on metrics for DevOps that meaningfully impact the business.

Why DevOps Metrics Matter

DevOps metrics are important as they help inform data-driven decisions that can guide continuous improvement efforts. And, with the right measures, you can link DevOps improvements with measurable impact to greater goals like digital transformation efforts. DevOps Research and Assessment (DORA) group helpfully provides us with clear metrics to track and even more insights with its latest report, Accelerate State of DevOps 2019.

DORA's Research-Driven Guidelines

Over the past six years, DORA has worked to develop four DevOps measurements indicative of an organization's software delivery performance, and ability to meet its DevOps goals. This year the group has enriched its research by identifying the capabilities that drive improvement in each of these four key areas. Using DORA's four key metrics as a foundation, let's explore the options and tools available for gathering metrics in DevOps.

Deployment Frequency

This metric gauges the throughput of your software delivery process, telling you how often and how quickly new services or features are deployed to production. This measure tells you quite a bit about your process effectiveness. For example, if there are bottlenecks in the process, measuring deployment frequency will help you unearth them by asking key questions such as:

■ Are there unnecessary steps in the process or are these steps in the wrong order?

■ What can we automate?

■ Are we the right team to manage this part of the process?

■ Do upstream issues exist that affect our responsiveness?

■ Do we have access to the tools we need to ensure timely deployments?

Over time deployment frequency should remain even — or increase. In the spirit of continuous improvement, decreases or dips should be reviewed closely to identify (and remediate when possible) the root cause. DORA identifies elite performers as those able to deploy on-demand, multiple times a day. Conversely, low performers deploy once every one to six months.

Lead Time for Code Changes

Along with deployment frequency, this metric measures the throughput of the software delivery process. DORA recommends measuring the lead time for code changes from the point in time when code is checked-in to the point it is released. This measure can also help you gauge the efficiency of your processes, supporting system effectiveness, and the general capabilities of your development team. For example, lengthy lead times can unearth inefficiencies in the development process or deployment bottlenecks.

As your team becomes more familiar and efficient with its DevOps processes, you should expect to see your lead time for changes metric decrease over time. Elite performers lead time is less than one day whereas low performers need between one and six months.

DevOps Change Failure Rate

DORA flags the change fail rate as a measure of the quality of the release process. It gets to the heart of how many application or service changes, builds or deployments that create a service issue large enough that it requires remediation. The change fail rate would ideally be managed down to as close to zero as possible. And, indeed, all but low performers have a change fail rate between zero and 15%.

The IT ticket system is an effective tool for measuring fail rates, tracking for each change its success rate, the impact of the failure and any required remediation. For example, your ticket systems can report if an approved change led to a service outage that required a rollback of the change.

Time to Restore Service

Once a service-impacting incident is detected, how long does it take to remediate and restore the service? This question measures system stability. Naturally, you'll want to restore services as quickly as possible as the cost of service outages can be extreme to the business. A Fortune 1000 survey by IDC found that the average cost of an infrastructure failure is $100,000 per hour.

When it comes to this measure, DORA research finds a significant gap between elite and low performers. Elite organizations are able to restore services on average in less than one hour whereas low performers report taking between one week and one month. High and medium performers are able to restore service within a day.

If you issue tickets for system repairs, your ticket system should be able to report on time to repair service. Tracking this metric will give you a distinct trend line illustrating progress over time. This is just one way to measure this metric. Often, a look at the monitoring tools that come with your cloud resources will give you this information. In the best scenarios, failures are self-healing and take milliseconds to fail over.

Business-Impacting

While these four metrics are a very helpful starting place to measure DevOps improvement and success, it is absolutely critical that teams take the initiative to link these metrics to the business. For example, increased deployment frequency allows the DevOps team to address new customer requests faster, growing customer satisfaction. Tracking key metrics is important to the business and even more so if you can show the business how DevOps processes are driving improvement over time that directly impacts key corporate goals.

Some tools allow for value stream mapping which directly ties code changes to features released. In some cases, e.g. retail applications, you can directly tie new feature introductions with impact to revenue.

DevOps Dashboard

With these four key metrics in hand, you are now in a position to build a dashboard for ongoing tracking and reporting. Theirs is a range of commonly used DevOps metrics dashboard tools available — both commercial and open-source, suitable for most needs and budget.

DORA's four key metrics will not only allow you to show progress and highlight areas for improvement for the DevOps team but by using these four common key metrics, you will be able to benchmark your team against their peers for external validation of their progress. And, you'll have a genuine numbers-based response when your boss drops by to ask how the team is progressing. Most importantly, with this data in hand, you will be prepared to change course quickly when you don't see a benefit in something you have built, leverage the insights you gain from your experiments, and capitalize on your successes helping the business reach its ultimate goals.

Elysia Lock is a Solutions Architect at Flux7, an NTT DATA Company
Share this

Industry News

June 04, 2020

Exadel announced Appery.io, its low code development platform, now offers new subscription tiers.

June 04, 2020

NetApp has entered into a definitive agreement to acquire Spot, a provider of compute management and cost optimization on the public clouds, to establish leadership in Application Driven Infrastructure.

June 04, 2020

Fluree announced the release of the Fluree JavaScript Library, a feature included in Fluree V0.13.0 that enables developers to launch a version of Fluree as an in-memory, code-resident data source for front-end applications, enabling sub-millisecond data delivery.

June 03, 2020

Netcracker Technology announced the launch of its Netcracker 2020 portfolio to help service providers focus on their customer’s digital lifestyle.

June 03, 2020

Navisite announced its acquisition of Privo, a Premier Consulting Partner in the Amazon Web Services (AWS) Partner Network (APN).

June 03, 2020

Grafana Labs released Grafana 7.0 with significant enhancements to simplify the development of custom plugins and drastically increase the power, speed and flexibility of visualization.

June 02, 2020

Chef announced a number of new products designed to enable coded enterprises to work across silos to build competitive advantage through automation.

June 02, 2020

Rancher Labs announced the general availability of Longhorn, an enterprise-grade, cloud-native container storage solution.

June 02, 2020

Checkmarx announced the launch of Checkmarx SCA (CxSCA), the company’s new, SaaS-based software composition analysis solution.

June 01, 2020

IT Revolution announced a full conference agenda for DevOps Enterprise Summit London, June 23-25, 2020.

June 01, 2020

Caltech CTME announced that Simplilearn, a global provider of digital skills training, will collaborate with CTME (Caltech's Center for Technology and Management Education) to offer a specialized Post Graduate Program in DevOps software engineering.

June 01, 2020

DevOps Institute, a global member-based association for advancing the human elements of DevOps, announced the introduction of its SKILup Playbook Library, a dynamic collective body of knowledge (cBok) that aligns thought leadership from industry experts with a set of dynamic, orchestrated artifacts, research and assets.

May 28, 2020

Docker has extended its strategic collaboration with Microsoft to simplify code to cloud application development for developers and development teams by more closely integrating with Azure Container Instances (ACI).

May 28, 2020

Eggplant announced updates to its Digital Automation Intelligence (DAI) platform.

May 28, 2020

Aptum launched its Managed DevOps Service in partnership with CloudOps, a cloud consulting and professional services company specializing in DevOps.