Differentiating DevOps Leaders from Laggards
August 02, 2018

Anand Akela
Tricentis

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.

The research examined 94 quality metrics that were classified into four categories. As a high-level summary, the research uncovered the following:

■ Understanding of business risk is the most important determining factor of DevOps and Agile maturity.

■ Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).

■ Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Agile/DevOps Leaders vs Laggards

The study also found that companies with the most successful Agile and DevOps initiatives do a number of things differently:

■ They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.

■ They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator” (53% vs. 27%).

■ They are significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

“Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” explained Clark Golestani, Managing Director of C Sensei Group and former President, Emerging Businesses and Global CIO of Merck. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors. Automated, continuous testing is the gate that allows you to protect your organization from significant risk as you innovate at the speed modern business demands.”

The Risk “Blind Spot”

Risk is a prevalent thread throughout the various parts of the report. Most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Moreover, just 15% of respondents say that their test suites reliably provide a good indication of business risk.

Interestingly, although risk-related metrics do not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts do. In fact, Agile and DevOps experts consistently rank risk-related metrics among their top 3 most valuable metrics in different phases and categories.

Other notable findings include:

■ Only 26% of firms with active Agile/DevOps adoptions are currently practicing the five core Continuous Testing practices shown to separate Agile/DevOps leaders from laggards.

■ Many CXOs and high-level decision makers are vastly overestimating the maturity of their firms’ Agile and DevOps practices.

■ Firms that rely on manual testing are more than twice as likely to report that testing is a bottleneck (59%), compared to those with more automated quality processes (26%).

Anand Akela is VP of Product Marketing at Tricentis
Share this

Industry News

February 13, 2020

CyberArk announced the CyberArk Blueprint for Privileged Access Management Success, designed to help customers take a future-proof, phased and measurable approach to reducing privilege-related risk.

February 13, 2020

Cloudentity announced a partnership with Signal Sciences to provide a holistic approach for enterprise companies looking to secure their web applications and APIs.

February 13, 2020

OutSystems announced that the Portuguese government has agreed to co-finance €5.1 million through the Portugal 2020 economic development program for a pair of innovative research and development programs led by OutSystems.

February 12, 2020

Tata Consultancy Services (TCS) launched Jile 4.0, a major release of its on-the-cloud enterprise Agile DevOps platform that enables software teams to manage, automate and measure the end-to-end software delivery value stream from ideation to deployment.

February 12, 2020

Synopsys announced that on Feb. 18 it will release a major update to the Polaris Software Integrity Platform to extend its static application security testing (SAST) and software composition analysis (SCA) capabilities to the developer's desktop through the native integration of the Code Sight IDE plugin.

February 12, 2020

Tufin announced the availability of Tufin SecureCloud, a security policy automation service for enterprises needing to gain visibility and control of the security posture of their cloud-native and hybrid cloud environments.

February 11, 2020

Applause released its new Applause Accessibility Tool that automatically finds and fixes key accessibility issues earlier in the software development lifecycle.

February 11, 2020

Chef announced a new channel program specifically designed to ensure that partners and customers are able to take maximum advantage of Chef’s 100 percent open source business model.

February 11, 2020

IT Revolution announced the first round of speakers for DevOps Enterprise Summit London 2020.

February 10, 2020

Dynatrace announced new enhancements to its support for Kubernetes.

February 06, 2020

DevOps Institute announced its newly revamped Global Education Partner Program.

February 06, 2020

Automox raised $30 million in Series B funding.

February 05, 2020

Couchbase introduced Couchbase Cloud, a fully-managed Database-as-a-Service (DBaaS).

February 05, 2020

Univa announced the general availability of Navops Launch 2.0, its flagship cloud-automation platform, designed to help enterprises simplify the migration of HPC and AI workloads to their choice of cloud.

February 05, 2020

Fugue announced a Team plan to help cloud engineering teams collaborate and innovate faster and more securely.