Data-Driven Benchmarks for High Performing Engineering Teams
February 02, 2021

Michael Stahnke
CircleCI

What does a high-performing engineering team really look like? It can be hard to know, but diving into the effectiveness of your delivery capabilities can tell you quite a bit.

Do deploys require a lot of cross-team coordination?

When production breaks, is it a long time before you can get it back up and running?

Are you getting feedback and results from your changes quickly?

The global challenges faced in 2020 have highlighted the competitive differentiator that being a well-oiled software delivery team provides. Your team's ability to deliver is a competitive advantage, and industry benchmarks are the only way to get a clear understanding of how your DevOps practices measure up.

When it comes to optimizing continuous integration and continuous delivery (CI/CD), Throughput, Duration, Mean Time to Recovery, and Success Rate are the most important metrics to consider. Measuring these benchmarks will tell you whether you're delivering product to your customers in the most efficient way possible.


How to Measure DevOps Success

After analyzing more than 55 million data points from 44,000 organizations on CircleCI, our 2020 State of Software Delivery report found baseline numbers for each of these benchmarks to guide engineering teams in making smarter decisions around CI/CD.

Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.

Duration: teams want to aim for workflow durations in the range of five to ten minutes.

Mean Time to Recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.

Success Rate: success rates above 90% should be your standard for the default branch of an application.

While some teams may have business-specific reasons for choosing different metrics as goals, any effort to improve engineering productivity or process will hinge on your ability to measure your team's baseline metrics and make incremental improvements.

4 Key Benchmarks

Let's take a closer look at each of these metrics so you can understand what they mean and why they're valuable.

Throughput is defined as the average number of workflow runs per day. I recommend monitoring Throughput rates rather than setting explicit goals. It's important to see how often things are happening, and Throughput is a direct measurement of commit frequency. A particular number of deploys per day is not the goal, but continuous validation of your codebase via your pipeline is.

Duration is defined as the length of time it takes for a workflow to run. It's the most important metric in this list because creating a fast feedback cycle hinges on Duration.

It's important to emphasize here that speed alone is not the goal. A workflow without tests can run quickly and return green, a signal that is not helpful to anyone. Without a quality testing suite, workflows with short durations aren't contributing valuable information to the feedback cycle. The goal, then, is rich information combined with short Duration.

Mean Time to Recovery is defined as the average time between failures and their next success. This is the second most important metric in this list. Because Mean Time to Recovery improves with more comprehensive test coverage, this metric can be a proxy for how well-tested your application is.

Failed build, valuable signal, rapid fix, passing build: continuous integration makes these rapid feedback loops possible. The fast signals enable teams to try new things and respond to any impact immediately.

Success Rate is defined as the number of passing runs divided by the total number of runs over a period of time. Git-flow models that rely on topic branch development, rather than default branch development, enable teams to keep their default branches green.

By scoping feature development to topic branches, we can differentiate between intentional experiments (where failing builds are valuable and expected) and stability issues (where failing builds are undesirable). Success Rate on the default branch is a more meaningful metric than Success Rate on a topic branch.

How Does Your Team Measure Up?

While there is no universal standard that every team should aspire to, the data collected on software delivery patterns globally show that there are reasonable benchmarks for teams to set as goals.

How does your team compare to the most successful teams developing software today?

Michael Stahnke is VP of Platform at CircleCI
Share this

Industry News

October 02, 2023

Spectro Cloud announced Palette EdgeAI to simplify how organizations deploy and manage AI workloads at scale across simple to complex edge locations, such as retail, healthcare, industrial automation, oil and gas, automotive/connected cars, and more.

September 28, 2023

Kong announced Kong Konnect Dedicated Cloud Gateways, the simplest and most cost-effective way to run Kong Gateways in the cloud fully managed as a service and on enterprise dedicated infrastructure.

September 28, 2023

Sisense unveiled the public preview of Compose SDK for Fusion.

September 28, 2023

Cloudflare announced Hyperdrive to make every local database global. Now developers can easily build globally distributed applications on Cloudflare Workers, the serverless developer platform used by over one million developers, without being constrained by their existing infrastructure.

September 27, 2023

Kong announced full support for Kong Mesh in Konnect, making Kong Konnect an API lifecycle management platform with built-in support for Kong Gateway Enterprise, Kong Ingress Controller and Kong Mesh via a SaaS control plane.

September 27, 2023

Vultr announced the launch of the Vultr GPU Stack and Container Registry to enable global enterprises and digital startups alike to build, test and operationalize artificial intelligence (AI) models at scale — across any region on the globe. \

September 27, 2023

Salt Security expanded its partnership with CrowdStrike by integrating the Salt Security API Protection Platform with the CrowdStrike Falcon® Platform.

September 26, 2023

Progress announced a partnership with Software Improvement Group (SIG), an independent technology and advisory firm for software quality, security and improvement, to help ensure the long-term maintainability and modernization of business-critical applications built on the Progress® OpenEdge® platform.

September 26, 2023

Solace announced a new version of its Solace Event Portal solution that gives organizations with Apache Kafka deployments better visibility into, and control over, their Kafka event streams, brokers and associated assets.

September 26, 2023

Reply launched a proprietary framework for generative AI-based software development, KICODE Reply.

September 26, 2023

Harness announced the industry-wide Engineering Excellence Collective™, an engineering leadership community.

September 25, 2023

Harness announced four new product modules on the Harness platform.

September 25, 2023

Sylabs announced the release of SingularityCE 4.0.

September 25, 2023

Timescale announced the launch of Timescale Vector, enabling developers to build production AI applications at scale with PostgreSQL.