Data-Driven Benchmarks for High Performing Engineering Teams
February 02, 2021

Michael Stahnke
CircleCI

What does a high-performing engineering team really look like? It can be hard to know, but diving into the effectiveness of your delivery capabilities can tell you quite a bit.

Do deploys require a lot of cross-team coordination?

When production breaks, is it a long time before you can get it back up and running?

Are you getting feedback and results from your changes quickly?

The global challenges faced in 2020 have highlighted the competitive differentiator that being a well-oiled software delivery team provides. Your team's ability to deliver is a competitive advantage, and industry benchmarks are the only way to get a clear understanding of how your DevOps practices measure up.

When it comes to optimizing continuous integration and continuous delivery (CI/CD), Throughput, Duration, Mean Time to Recovery, and Success Rate are the most important metrics to consider. Measuring these benchmarks will tell you whether you're delivering product to your customers in the most efficient way possible.


How to Measure DevOps Success

After analyzing more than 55 million data points from 44,000 organizations on CircleCI, our 2020 State of Software Delivery report found baseline numbers for each of these benchmarks to guide engineering teams in making smarter decisions around CI/CD.

Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.

Duration: teams want to aim for workflow durations in the range of five to ten minutes.

Mean Time to Recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.

Success Rate: success rates above 90% should be your standard for the default branch of an application.

While some teams may have business-specific reasons for choosing different metrics as goals, any effort to improve engineering productivity or process will hinge on your ability to measure your team's baseline metrics and make incremental improvements.

4 Key Benchmarks

Let's take a closer look at each of these metrics so you can understand what they mean and why they're valuable.

Throughput is defined as the average number of workflow runs per day. I recommend monitoring Throughput rates rather than setting explicit goals. It's important to see how often things are happening, and Throughput is a direct measurement of commit frequency. A particular number of deploys per day is not the goal, but continuous validation of your codebase via your pipeline is.

Duration is defined as the length of time it takes for a workflow to run. It's the most important metric in this list because creating a fast feedback cycle hinges on Duration.

It's important to emphasize here that speed alone is not the goal. A workflow without tests can run quickly and return green, a signal that is not helpful to anyone. Without a quality testing suite, workflows with short durations aren't contributing valuable information to the feedback cycle. The goal, then, is rich information combined with short Duration.

Mean Time to Recovery is defined as the average time between failures and their next success. This is the second most important metric in this list. Because Mean Time to Recovery improves with more comprehensive test coverage, this metric can be a proxy for how well-tested your application is.

Failed build, valuable signal, rapid fix, passing build: continuous integration makes these rapid feedback loops possible. The fast signals enable teams to try new things and respond to any impact immediately.

Success Rate is defined as the number of passing runs divided by the total number of runs over a period of time. Git-flow models that rely on topic branch development, rather than default branch development, enable teams to keep their default branches green.

By scoping feature development to topic branches, we can differentiate between intentional experiments (where failing builds are valuable and expected) and stability issues (where failing builds are undesirable). Success Rate on the default branch is a more meaningful metric than Success Rate on a topic branch.

How Does Your Team Measure Up?

While there is no universal standard that every team should aspire to, the data collected on software delivery patterns globally show that there are reasonable benchmarks for teams to set as goals.

How does your team compare to the most successful teams developing software today?

Michael Stahnke is VP of Platform at CircleCI
Share this

Industry News

February 02, 2023

Red Hat announced a multi-stage alliance to offer customers a greater choice of operating systems to run on Oracle Cloud Infrastructure (OCI).

February 02, 2023

Snow Software announced a new global partner program designed to enable partners to support customers as they face complex market challenges around managing cost and mitigating risk, while delivering value more efficiently and effectively with Snow.

February 02, 2023

Contrast Security announced the launch of its new partner program, the Security Innovation Alliance (SIA), which is a global ecosystem of system integrators (SIs), cloud, channel and technology alliances.

February 01, 2023

Red Hat introduced new security and compliance capabilities for the Red Hat OpenShift enterprise Kubernetes platform.

February 01, 2023

Jetpack.io formally launched with Devbox Cloud, a managed service offering for Devbox.

February 01, 2023

Jellyfish launched Life Cycle Explorer, a new solution that identifies bottlenecks in the life cycle of engineering work to help teams adapt workflow processes and more effectively deliver value to customers.

January 31, 2023

Ably announced the Ably Terraform provider.

January 31, 2023

Checkmarx announced the immediate availability of Supply Chain Threat Intelligence, which delivers detailed threat intelligence on hundreds of thousands of malicious packages, contributor reputation, malicious behavior and more.

January 31, 2023

Qualys announced its new GovCloud platform along with the achievement of FedRAMP Ready status at the High impact level, from the Federal Risk and Authorization Management Program (FedRAMP).

January 30, 2023

F5 announced the general availability of F5 NGINXaaS for Azure, an integrated solution co-developed by F5 and Microsoft that empowers enterprises to deliver secure, high-performance applications in the cloud.

January 30, 2023

Tenable announced Tenable Ventures, a corporate investment program.

January 26, 2023

Ubuntu Pro, Canonical’s comprehensive subscription for secure open source and compliance, is now generally available.

January 26, 2023

Mirantis, freeing developers to create their most valuable code, today announced that it has acquired the Santa Clara, California-based Shipa to add automated application discovery, operations, security, and observability to the Lens Kubernetes Platform.

January 25, 2023

SmartBear has integrated the powerful contract testing capabilities of PactFlow with SwaggerHub.

January 25, 2023

Venafi introduced TLS Protect for Kubernetes.