Differentiating DevOps Leaders from Laggards
August 02, 2018

Anand Akela
Tricentis

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.

The research examined 94 quality metrics that were classified into four categories. As a high-level summary, the research uncovered the following:

■ Understanding of business risk is the most important determining factor of DevOps and Agile maturity.

■ Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).

■ Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Agile/DevOps Leaders vs Laggards

The study also found that companies with the most successful Agile and DevOps initiatives do a number of things differently:

■ They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.

■ They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator” (53% vs. 27%).

■ They are significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

“Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” explained Clark Golestani, Managing Director of C Sensei Group and former President, Emerging Businesses and Global CIO of Merck. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors. Automated, continuous testing is the gate that allows you to protect your organization from significant risk as you innovate at the speed modern business demands.”

The Risk “Blind Spot”

Risk is a prevalent thread throughout the various parts of the report. Most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Moreover, just 15% of respondents say that their test suites reliably provide a good indication of business risk.

Interestingly, although risk-related metrics do not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts do. In fact, Agile and DevOps experts consistently rank risk-related metrics among their top 3 most valuable metrics in different phases and categories.

Other notable findings include:

■ Only 26% of firms with active Agile/DevOps adoptions are currently practicing the five core Continuous Testing practices shown to separate Agile/DevOps leaders from laggards.

■ Many CXOs and high-level decision makers are vastly overestimating the maturity of their firms’ Agile and DevOps practices.

■ Firms that rely on manual testing are more than twice as likely to report that testing is a bottleneck (59%), compared to those with more automated quality processes (26%).

Anand Akela is VP of Product Marketing at Tricentis

The Latest

January 22, 2019

In 2019, competitive disruption will drive remaining laggards to a DevOps boiling point. As the industry moves to the plateau of productivity with DevOps automation and standard tooling, laggard executives will reach a management crisis point that will force actions ...

January 17, 2019

To better align business and IT objectives, enterprise organizations should focus on the core "problems" that individual business units face today in driving out real consumer value. Until the roadblocks and inhibitors — and, ultimately, the resultant technical debt — are removed from the equation, large enterprise organizations will continue struggling to succeed ...

January 16, 2019

Technical debt is what results when legacy platforms or highly integrated and dependent systems and processes inhibit large enterprise organizations from meeting the needs of internal business stakeholders. In many cases, the core objectives that drive real, monetizable business value are not aligned to the esoteric IT goals of "automation" and "Agile development." This creates a fundamental disconnect between business and IT ...

January 14, 2019

Budget season is an important time of the year for businesses because it gives senior IT and security leaders time to reflect on what went right this year and what initiatives need to be given priority in the new year. Recent research from Threat Stack shows security budgets are expected to increase by 19 percent over the next two years, but business leaders are still facing challenges determining where to allocate this budget in the face of rapidly evolving infrastructure ...

January 10, 2019

As organizations of all sizes are embracing hybrid and multi-cloud infrastructures, they are experiencing the many benefits of a more agile, distributed and high-speed environment where new applications and services can be built and delivered in days and weeks, rather than months and years. But as the adoption of these next generation architectures continues to grow, so do the complexities of securing the cloud workloads running on them ...

January 09, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 9, the final installment, covers microservices, containers and APIs ...

January 08, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 8, covers microservices and containers ...

January 07, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 7, covers the Cloud ...

December 20, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 6, covers DevOps Analytics, including AI and Machine Learning ...

December 19, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 5 is all about testing ...

Share this