Data-Driven Benchmarks for High Performing Engineering Teams
February 02, 2021

Michael Stahnke
CircleCI

What does a high-performing engineering team really look like? It can be hard to know, but diving into the effectiveness of your delivery capabilities can tell you quite a bit.

Do deploys require a lot of cross-team coordination?

When production breaks, is it a long time before you can get it back up and running?

Are you getting feedback and results from your changes quickly?

The global challenges faced in 2020 have highlighted the competitive differentiator that being a well-oiled software delivery team provides. Your team's ability to deliver is a competitive advantage, and industry benchmarks are the only way to get a clear understanding of how your DevOps practices measure up.

When it comes to optimizing continuous integration and continuous delivery (CI/CD), Throughput, Duration, Mean Time to Recovery, and Success Rate are the most important metrics to consider. Measuring these benchmarks will tell you whether you're delivering product to your customers in the most efficient way possible.


How to Measure DevOps Success

After analyzing more than 55 million data points from 44,000 organizations on CircleCI, our 2020 State of Software Delivery report found baseline numbers for each of these benchmarks to guide engineering teams in making smarter decisions around CI/CD.

Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.

Duration: teams want to aim for workflow durations in the range of five to ten minutes.

Mean Time to Recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.

Success Rate: success rates above 90% should be your standard for the default branch of an application.

While some teams may have business-specific reasons for choosing different metrics as goals, any effort to improve engineering productivity or process will hinge on your ability to measure your team's baseline metrics and make incremental improvements.

4 Key Benchmarks

Let's take a closer look at each of these metrics so you can understand what they mean and why they're valuable.

Throughput is defined as the average number of workflow runs per day. I recommend monitoring Throughput rates rather than setting explicit goals. It's important to see how often things are happening, and Throughput is a direct measurement of commit frequency. A particular number of deploys per day is not the goal, but continuous validation of your codebase via your pipeline is.

Duration is defined as the length of time it takes for a workflow to run. It's the most important metric in this list because creating a fast feedback cycle hinges on Duration.

It's important to emphasize here that speed alone is not the goal. A workflow without tests can run quickly and return green, a signal that is not helpful to anyone. Without a quality testing suite, workflows with short durations aren't contributing valuable information to the feedback cycle. The goal, then, is rich information combined with short Duration.

Mean Time to Recovery is defined as the average time between failures and their next success. This is the second most important metric in this list. Because Mean Time to Recovery improves with more comprehensive test coverage, this metric can be a proxy for how well-tested your application is.

Failed build, valuable signal, rapid fix, passing build: continuous integration makes these rapid feedback loops possible. The fast signals enable teams to try new things and respond to any impact immediately.

Success Rate is defined as the number of passing runs divided by the total number of runs over a period of time. Git-flow models that rely on topic branch development, rather than default branch development, enable teams to keep their default branches green.

By scoping feature development to topic branches, we can differentiate between intentional experiments (where failing builds are valuable and expected) and stability issues (where failing builds are undesirable). Success Rate on the default branch is a more meaningful metric than Success Rate on a topic branch.

How Does Your Team Measure Up?

While there is no universal standard that every team should aspire to, the data collected on software delivery patterns globally show that there are reasonable benchmarks for teams to set as goals.

How does your team compare to the most successful teams developing software today?

Michael Stahnke is VP of Platform at CircleCI
Share this

Industry News

September 27, 2022

DevOps Institute will host SKILup Festival in Singapore on November 15, 2022.

September 27, 2022

Delinea announced the latest release of DevOps Secrets Vault, its high-speed vault for DevOps and DevSecOps teams.

September 27, 2022

The Apptainer community announced version 1.1.0 of the popular container system for secure, high-performance computing (HPC). Improvements in the new version provide a smaller attack surface for production deployments while offering features that improve and simplify the user experience.

September 26, 2022

Secure Code Warrior unveiled Coding Labs, a new mechanism that allows developers to more easily move from learning to applying secure coding knowledge, leading to fewer vulnerabilities in code.

September 26, 2022

ActiveState announced the availability of the ActiveState Artifact Repository.

September 26, 2022

Split Software announced the availability of its Feature Data Platform in the Microsoft Azure Marketplace.

September 22, 2022

Katalon announced the launch of the Katalon Platform, a modern and comprehensive software quality management platform that enables teams of any size to easily and efficiently test, launch, and optimize apps, products, and software.

September 22, 2022

StackHawk announced its Deeper API Security Test Coverage release.

September 21, 2022

Platform9 announced the launch of its latest open source project, Arlon.

September 21, 2022

Redpanda Data announced Redpanda Console.

September 21, 2022

mabl announced its availability as a private listing on Google Cloud Marketplace.

September 21, 2022

Zesty announced a $75 million Series B funding round led by B Capital and Series A investor Sapphire Ventures.

September 20, 2022

Opsera, the Continuous Orchestration platform for DevOps, announced a free trial of its no-code Salesforce Release Management platform for fast and secure Salesforce releases.

September 20, 2022

Sysdig announced ToDo and Remediation Guru.

September 20, 2022

AutoRABIT announced CodeScan Shield.