Test Smarter Not Harder: Shift Testing Left and Right with Test Impact Analysis - Part 1
August 15, 2019

Mark Lambert
Parasoft

Confirmed frequently by industry surveys and reports, software testing is still a bottleneck, even after the implementation of modern development processes like Agile, DevOps, and Continuous Integration/Deployment. In some cases, software teams aren't testing nearly enough and have to deal with bugs and security vulnerabilities at the later stages of the development cycle, which creates a false assumption that these new processes can't deliver on their promise. One solution to certain classes of issues is shift right testing, which relies on monitoring the application in a production environment, but it requires a rock solid infrastructure to roll back new changes if a critical defect arises.

As a result, organizations are still missing deadlines, and quality and security is suffering. But there's a better way! To test smarter, organizations are using technology called test impact analysis to understand exactly what to test. This data-driven approach supports both shift left and right testing.

Agile and DevOps and the Testing Bottleneck

Testing in any iterative process is a compromise of how much testing can be done in a limited cycle time. In most projects, it's impossible to do a full regression on each iteration. Instead, a limited set of testing is performed, and exactly what to test is based on best guesses. Testing is also back-loaded in the cycle since there isn't usually enough completed new features to test. The resulting effort vs. time graph ends up like a saw tooth, as shown below. In each cycle only a limited set of tests are executed until the final cycle where a full regression test is performed.


Figure 1: Agile processes result in a "saw tooth" of testing activity. Only the full regression cycle is able to do a "complete" test.

Unfortunately, no project reaches the final cycle with zero bugs and zero security vulnerabilities. Finding defects at this stage adds delays as bugs are fixed and retested. And even with those delays and all, many bugs still make their way into the deployed product, as illustrated below.


Figure 2: Integration and full regression testing is never error free. Late stage defects cause schedule and cost overruns.

This situation has resulted in the adoption of what has been coined "shift-right testing," in which organizations continue to test their application into the deployment phase. The intention of shift-right testing is to augment and extend testing efforts, with testing best-suited in the deployment phase such as API monitoring, toggling features in production, retrieving feedback from real life operation.

Shift Right Testing

The difficulties in reproducing realistic test environments and using real data and traffic in testing led teams to using production environments to monitor and test their applications. There are benefits to this, for example, being able to test applications with live production traffic supporting fault tolerance and performance improvements. A common use case is the so-called canary release, in which a new version of the software is released to a small subset of customers first, and then rolled out to an increasingly larger group as bugs are reported and fixed. Roku, for example, does this for updating their device firmware.

Shift-right testing relies on a development infrastructure that can roll back a release in the event of critical defects. For example, a severe security vulnerability in a canary release means rolling back the release until a new updated release is ready, as you can see in the illustration here:


Figure 3: Shift right testing relies on solid development operations infrastructure to roll back releases in the face of critical defects.

But there are risks to using production environments to monitor and test software, and of course, the intention of shift-right testing was never to replace unit, API and UI testing practices before deployment! Shift-right testing is a complementarypractice, that extends the philosophy of continuous testing into production. Despite this, organizations can easily abuse the concept to justify doing even less unit and API testing during development. In order to prevent this, we need to make testing during development phases to be easier, more productive and produce better quality software.

Read Part 2: Testing Smarter, Not Harder, by Focusing Your Testing

Mark Lambert is VP of Products at Parasoft
Share this

Industry News

June 01, 2020

IT Revolution announced a full conference agenda for DevOps Enterprise Summit London, June 23-25, 2020.

June 01, 2020

Caltech CTME announced that Simplilearn, a global provider of digital skills training, will collaborate with CTME (Caltech's Center for Technology and Management Education) to offer a specialized Post Graduate Program in DevOps software engineering.

June 01, 2020

DevOps Institute, a global member-based association for advancing the human elements of DevOps, announced the introduction of its SKILup Playbook Library, a dynamic collective body of knowledge (cBok) that aligns thought leadership from industry experts with a set of dynamic, orchestrated artifacts, research and assets.

May 28, 2020

Docker has extended its strategic collaboration with Microsoft to simplify code to cloud application development for developers and development teams by more closely integrating with Azure Container Instances (ACI).

May 28, 2020

Eggplant announced updates to its Digital Automation Intelligence (DAI) platform.

May 28, 2020

Aptum launched its Managed DevOps Service in partnership with CloudOps, a cloud consulting and professional services company specializing in DevOps.

May 27, 2020

Red Hat announced an expansion of its application services portfolio with the addition of Quarkus as a fully supported framework in Red Hat Runtimes.

May 27, 2020

Couchbase has completed a $105 million all-equity Series G round of fundraising.

May 27, 2020

Aqua Security closed a Series D round of $30M led by Greenspring Associates.

May 26, 2020

GitLab is releasing 13.0 of its DevSecOps platform to enable organizations to efficiently adapt and respond to new and dynamic business challenges.

May 26, 2020

Solo.io announced the availability of the Istio Developer Portal to streamline the developer onboarding process for improved developer experience and increased productivity with added security features.

May 26, 2020

WhiteHat Security will offer free application scanning services to any education institution to support secure online learning.

May 21, 2020

Exadel announced the Grand Prize winner of the “Appery.io COVID-19 Virtual Hackathon.”

May 21, 2020

CloudBees announced significant advances for its Software Delivery Management (SDM) platform – integrations with additional continuous integration and continuous delivery (CI/CD) engines, including Google Cloud Build and Tekton, and extension of the availability of CloudBees’ SDM Preview Program.

May 21, 2020

OutSystems is announcing over 70 development accelerators that ensure web and mobile applications created on the OutSystems low-code development platform can comply with the highest accessibility standards and regulations.