Data-Driven Benchmarks for High Performing Engineering Teams
February 02, 2021

Michael Stahnke
CircleCI

What does a high-performing engineering team really look like? It can be hard to know, but diving into the effectiveness of your delivery capabilities can tell you quite a bit.

Do deploys require a lot of cross-team coordination?

When production breaks, is it a long time before you can get it back up and running?

Are you getting feedback and results from your changes quickly?

The global challenges faced in 2020 have highlighted the competitive differentiator that being a well-oiled software delivery team provides. Your team's ability to deliver is a competitive advantage, and industry benchmarks are the only way to get a clear understanding of how your DevOps practices measure up.

When it comes to optimizing continuous integration and continuous delivery (CI/CD), Throughput, Duration, Mean Time to Recovery, and Success Rate are the most important metrics to consider. Measuring these benchmarks will tell you whether you're delivering product to your customers in the most efficient way possible.


How to Measure DevOps Success

After analyzing more than 55 million data points from 44,000 organizations on CircleCI, our 2020 State of Software Delivery report found baseline numbers for each of these benchmarks to guide engineering teams in making smarter decisions around CI/CD.

Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.

Duration: teams want to aim for workflow durations in the range of five to ten minutes.

Mean Time to Recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.

Success Rate: success rates above 90% should be your standard for the default branch of an application.

While some teams may have business-specific reasons for choosing different metrics as goals, any effort to improve engineering productivity or process will hinge on your ability to measure your team's baseline metrics and make incremental improvements.

4 Key Benchmarks

Let's take a closer look at each of these metrics so you can understand what they mean and why they're valuable.

Throughput is defined as the average number of workflow runs per day. I recommend monitoring Throughput rates rather than setting explicit goals. It's important to see how often things are happening, and Throughput is a direct measurement of commit frequency. A particular number of deploys per day is not the goal, but continuous validation of your codebase via your pipeline is.

Duration is defined as the length of time it takes for a workflow to run. It's the most important metric in this list because creating a fast feedback cycle hinges on Duration.

It's important to emphasize here that speed alone is not the goal. A workflow without tests can run quickly and return green, a signal that is not helpful to anyone. Without a quality testing suite, workflows with short durations aren't contributing valuable information to the feedback cycle. The goal, then, is rich information combined with short Duration.

Mean Time to Recovery is defined as the average time between failures and their next success. This is the second most important metric in this list. Because Mean Time to Recovery improves with more comprehensive test coverage, this metric can be a proxy for how well-tested your application is.

Failed build, valuable signal, rapid fix, passing build: continuous integration makes these rapid feedback loops possible. The fast signals enable teams to try new things and respond to any impact immediately.

Success Rate is defined as the number of passing runs divided by the total number of runs over a period of time. Git-flow models that rely on topic branch development, rather than default branch development, enable teams to keep their default branches green.

By scoping feature development to topic branches, we can differentiate between intentional experiments (where failing builds are valuable and expected) and stability issues (where failing builds are undesirable). Success Rate on the default branch is a more meaningful metric than Success Rate on a topic branch.

How Does Your Team Measure Up?

While there is no universal standard that every team should aspire to, the data collected on software delivery patterns globally show that there are reasonable benchmarks for teams to set as goals.

How does your team compare to the most successful teams developing software today?

Michael Stahnke is VP of Platform at CircleCI
Share this

Industry News

March 02, 2021

JFrog announced that its DevOps Platform tools – JFrog Artifactory and JFrog Xray – are available with native deployment templates for customers using AWS GovCloud (US) and Azure Government clouds.

March 02, 2021

Spectro Cloud announced support for existing Kubernetes environments, including clusters on public cloud services such as Amazon EKS, Azure AKS and Google GKE, has been added to the Spectro Cloud Kubernetes management platform.

March 02, 2021

Idera announced the acquisition of PreEmptive Solutions, LLC, a provider of application protection and security.

March 01, 2021

CloudBolt Software announced the launch of OneFuse Community Edition, a free version of its codeless integration platform for automating, integrating, and extending private and hybrid cloud infrastructures.

March 01, 2021

DBmaestro launched support for Snowflake, the Data Cloud company.

March 01, 2021

Platform9 closed Series-D funding with an additional $12.5 million for a total of $37.5 million.

February 25, 2021

Red Hat announced Red Hat OpenShift 4.7, the latest version of the company’s enterprise Kubernetes platform.

February 25, 2021

Granulate announced the release of its open-source platform, the G-Profiler, a production profiling solution that measures the performance of code in production applications to facilitate compute optimization.

February 25, 2021

Checkmarx announced the launch of KICS (Keeping Infrastructure as Code Secure), an open source static analysis solution that enables developers to write more secure infrastructure as code (IaC).

February 24, 2021

Applause launched its Product Excellence Platform (PEP).

February 24, 2021

Mabl announced the beta release of their new native desktop application that empowers users to easily automate testing for browsers, mobile browsers, and APIs.

February 24, 2021

D2iQ announced the general availability of D2iQ Kaptain, the cloud native end-to-end platform for running ML workloads on Kubernetes.

February 23, 2021

Edge Delta announced new capabilities for dynamically processing and routing logs, metrics, and traces -- allowing DevOps, Security, and SRE teams to analyze large volumes of streaming data without the cost, delay, or complexity of requiring data to be indexed, helping customers decouple where machine data is analyzed from where it is stored.

February 23, 2021

env0 announced a remote run SaaS solution for the automation of Terragrunt workloads across multiple Terraform modules.

February 23, 2021

Platform9 closed its Series-D funding with an additional $12.5 million for a total of $37.5 million.