DEVOPSdigest

What does a high-performing engineering team really look like? It can be hard to know, but diving into the effectiveness of your delivery capabilities can tell you quite a bit.

Do deploys require a lot of cross-team coordination?

When production breaks, is it a long time before you can get it back up and running?

Are you getting feedback and results from your changes quickly?

The global challenges faced in 2020 have highlighted the competitive differentiator that being a well-oiled software delivery team provides. Your team's ability to deliver is a competitive advantage, and industry benchmarks are the only way to get a clear understanding of how your DevOps practices measure up.

When it comes to optimizing continuous integration and continuous delivery (CI/CD), Throughput, Duration, Mean Time to Recovery, and Success Rate are the most important metrics to consider. Measuring these benchmarks will tell you whether you're delivering product to your customers in the most efficient way possible.

How to Measure DevOps Success

After analyzing more than 55 million data points from 44,000 organizations on CircleCI, our 2020 State of Software Delivery report found baseline numbers for each of these benchmarks to guide engineering teams in making smarter decisions around CI/CD.

■ Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.

■ Duration: teams want to aim for workflow durations in the range of five to ten minutes.

■ Mean Time to Recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.

■ Success Rate: success rates above 90% should be your standard for the default branch of an application.

While some teams may have business-specific reasons for choosing different metrics as goals, any effort to improve engineering productivity or process will hinge on your ability to measure your team's baseline metrics and make incremental improvements.

4 Key Benchmarks

Let's take a closer look at each of these metrics so you can understand what they mean and why they're valuable.

■ Throughput is defined as the average number of workflow runs per day. I recommend monitoring Throughput rates rather than setting explicit goals. It's important to see how often things are happening, and Throughput is a direct measurement of commit frequency. A particular number of deploys per day is not the goal, but continuous validation of your codebase via your pipeline is.

■ Duration is defined as the length of time it takes for a workflow to run. It's the most important metric in this list because creating a fast feedback cycle hinges on Duration.

It's important to emphasize here that speed alone is not the goal. A workflow without tests can run quickly and return green, a signal that is not helpful to anyone. Without a quality testing suite, workflows with short durations aren't contributing valuable information to the feedback cycle. The goal, then, is rich information combined with short Duration.

■ Mean Time to Recovery is defined as the average time between failures and their next success. This is the second most important metric in this list. Because Mean Time to Recovery improves with more comprehensive test coverage, this metric can be a proxy for how well-tested your application is.

Failed build, valuable signal, rapid fix, passing build: continuous integration makes these rapid feedback loops possible. The fast signals enable teams to try new things and respond to any impact immediately.

■ Success Rate is defined as the number of passing runs divided by the total number of runs over a period of time. Git-flow models that rely on topic branch development, rather than default branch development, enable teams to keep their default branches green.

By scoping feature development to topic branches, we can differentiate between intentional experiments (where failing builds are valuable and expected) and stability issues (where failing builds are undesirable). Success Rate on the default branch is a more meaningful metric than Success Rate on a topic branch.

How Does Your Team Measure Up?

While there is no universal standard that every team should aspire to, the data collected on software delivery patterns globally show that there are reasonable benchmarks for teams to set as goals.

How does your team compare to the most successful teams developing software today?