Differentiating DevOps Leaders from Laggards
August 02, 2018

Anand Akela
Tricentis

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.

The research examined 94 quality metrics that were classified into four categories. As a high-level summary, the research uncovered the following:

■ Understanding of business risk is the most important determining factor of DevOps and Agile maturity.

■ Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).

■ Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Agile/DevOps Leaders vs Laggards

The study also found that companies with the most successful Agile and DevOps initiatives do a number of things differently:

■ They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.

■ They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator” (53% vs. 27%).

■ They are significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

“Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” explained Clark Golestani, Managing Director of C Sensei Group and former President, Emerging Businesses and Global CIO of Merck. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors. Automated, continuous testing is the gate that allows you to protect your organization from significant risk as you innovate at the speed modern business demands.”

The Risk “Blind Spot”

Risk is a prevalent thread throughout the various parts of the report. Most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Moreover, just 15% of respondents say that their test suites reliably provide a good indication of business risk.

Interestingly, although risk-related metrics do not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts do. In fact, Agile and DevOps experts consistently rank risk-related metrics among their top 3 most valuable metrics in different phases and categories.

Other notable findings include:

■ Only 26% of firms with active Agile/DevOps adoptions are currently practicing the five core Continuous Testing practices shown to separate Agile/DevOps leaders from laggards.

■ Many CXOs and high-level decision makers are vastly overestimating the maturity of their firms’ Agile and DevOps practices.

■ Firms that rely on manual testing are more than twice as likely to report that testing is a bottleneck (59%), compared to those with more automated quality processes (26%).

Anand Akela is VP of Product Marketing at Tricentis
Share this

Industry News

July 01, 2020

JFrog announced the launch of ChartCenter, a free, security-focused central repository of Helm charts for the community.

July 01, 2020

Kong announced a significant upgrade to open source Kuma, Kuma 0.6, available today.

July 01, 2020

Compuware Corporation, a BMC company, announced new capabilities that further automate and integrate test data and test case execution, empowering IT teams to achieve high-performance application development quality, velocity and efficiency.

June 30, 2020

Couchbase announced the general availability of Couchbase Cloud, a fully-managed Database-as-a-Service (DBaaS).

June 30, 2020

Split Software announced new capabilities designed to accelerate the adoption of feature flags in large-scale organizations.

June 30, 2020

WhiteHat Security announced a discounted Web + Mobile Application Security bundle to help organizations secure the digital future.

June 29, 2020

Puppet introduced the public beta availability of Relay, an event-driven automation platform.

June 29, 2020

D2iQ introduced KUDO for Kubeflow to simplify and accelerate machine learning (ML) deployments on Kubernetes.

June 29, 2020

Codefresh announced $27M in new funding led by Red Dot Capital Partners.

June 25, 2020

Micro Focus announced the general availability of Visual COBOL 6.0 and Enterprise Suite 6.0, providing versatile application, process and infrastructure modernization solutions for today’s enterprise developer.

June 25, 2020

SaltStack announced new features available in SaltStack Enterprise 6.3 that integrate best-of-breed IT monitoring and vulnerability management solutions, including Splunk, Tenable, Qualys, Rapid7, and Kenna Security.

June 25, 2020

Keysight Technologies has completed the acquisition of Eggplant from The Carlyle Group.

June 24, 2020

JFrog unveiled new capabilities to address the growing problem of software distribution bottlenecks. The newly introduced CDN-based and Peer-to-Peer software package distribution mechanisms empower companies to overcome the challenge of frequently delivering large volumes of artifacts to internal teams and external clients.

June 24, 2020

Copado announced its Summer 20 release to accelerate, optimize and measure innovation delivery on the Salesforce platform.

June 24, 2020

Bugsnag launched Stability Center, a centralized location that offers a holistic view into stability stats and trends across releases for multiple client and server-side applications.