Test Smarter Not Harder: Shift Testing Left and Right with Test Impact Analysis - Part 1
August 15, 2019

Mark Lambert
Parasoft

Confirmed frequently by industry surveys and reports, software testing is still a bottleneck, even after the implementation of modern development processes like Agile, DevOps, and Continuous Integration/Deployment. In some cases, software teams aren't testing nearly enough and have to deal with bugs and security vulnerabilities at the later stages of the development cycle, which creates a false assumption that these new processes can't deliver on their promise. One solution to certain classes of issues is shift right testing, which relies on monitoring the application in a production environment, but it requires a rock solid infrastructure to roll back new changes if a critical defect arises.

As a result, organizations are still missing deadlines, and quality and security is suffering. But there's a better way! To test smarter, organizations are using technology called test impact analysis to understand exactly what to test. This data-driven approach supports both shift left and right testing.

Agile and DevOps and the Testing Bottleneck

Testing in any iterative process is a compromise of how much testing can be done in a limited cycle time. In most projects, it's impossible to do a full regression on each iteration. Instead, a limited set of testing is performed, and exactly what to test is based on best guesses. Testing is also back-loaded in the cycle since there isn't usually enough completed new features to test. The resulting effort vs. time graph ends up like a saw tooth, as shown below. In each cycle only a limited set of tests are executed until the final cycle where a full regression test is performed.


Figure 1: Agile processes result in a "saw tooth" of testing activity. Only the full regression cycle is able to do a "complete" test.

Unfortunately, no project reaches the final cycle with zero bugs and zero security vulnerabilities. Finding defects at this stage adds delays as bugs are fixed and retested. And even with those delays and all, many bugs still make their way into the deployed product, as illustrated below.


Figure 2: Integration and full regression testing is never error free. Late stage defects cause schedule and cost overruns.

This situation has resulted in the adoption of what has been coined "shift-right testing," in which organizations continue to test their application into the deployment phase. The intention of shift-right testing is to augment and extend testing efforts, with testing best-suited in the deployment phase such as API monitoring, toggling features in production, retrieving feedback from real life operation.

Shift Right Testing

The difficulties in reproducing realistic test environments and using real data and traffic in testing led teams to using production environments to monitor and test their applications. There are benefits to this, for example, being able to test applications with live production traffic supporting fault tolerance and performance improvements. A common use case is the so-called canary release, in which a new version of the software is released to a small subset of customers first, and then rolled out to an increasingly larger group as bugs are reported and fixed. Roku, for example, does this for updating their device firmware.

Shift-right testing relies on a development infrastructure that can roll back a release in the event of critical defects. For example, a severe security vulnerability in a canary release means rolling back the release until a new updated release is ready, as you can see in the illustration here:


Figure 3: Shift right testing relies on solid development operations infrastructure to roll back releases in the face of critical defects.

But there are risks to using production environments to monitor and test software, and of course, the intention of shift-right testing was never to replace unit, API and UI testing practices before deployment! Shift-right testing is a complementarypractice, that extends the philosophy of continuous testing into production. Despite this, organizations can easily abuse the concept to justify doing even less unit and API testing during development. In order to prevent this, we need to make testing during development phases to be easier, more productive and produce better quality software.

Read Part 2: Testing Smarter, Not Harder, by Focusing Your Testing

Mark Lambert is VP of Products at Parasoft
Share this

Industry News

September 12, 2019

Rafay Systems announced the general availability of its turnkey, SaaS-based offering designed to confront a complex set of ongoing challenges enterprises and service providers face when modernizing their applications.

September 12, 2019

StackRox announced the availability of the StackRox App for the Sumo Logic Continuous Intelligence Platform.

September 12, 2019

Lacework is receiving $42 million from Sutter Hill Ventures and Liberty Global Ventures.

September 11, 2019

Clubhouse released a fully featured Free Plan that offers the full power of its flagship product to teams up to 10 people.

September 11, 2019

Sectigo released integrations with five of the most popular DevOps configuration management and container orchestration platforms.

September 11, 2019

Kong announced the release of a new open source project called Kuma.

September 10, 2019

Parasoft is excited to announce that Parasoft SOAtest, an API and UI functional testing solution, has won a 2019 API Award in the Best in Microservices Infrastructure category.

September 10, 2019

Cohesity announced the launch of Cohesity Agile Dev and Test, a new solution that addresses a key bottleneck organizations face in building applications at speed.

September 10, 2019

Split Software announced the addition of Feature Monitoring, an automated detection capability for its feature delivery platform that reduces detection times of errors in a code release.

September 09, 2019

US Signal announced the launch of its managed Website and Application Security Solution.

September 09, 2019

Tasktop announced that Jama Software is now offering the cloud version of its Tasktop Integration Hub for Jama Connect to automate and visualize the flow of product-critical information across the software delivery value stream.

September 09, 2019

Mesosphere announced a significant expansion in strategy and product portfolio as well as a new company name - D2iQ.

September 05, 2019

Redgate launched SQL Monitor 9.1, the latest version of the world’s most popular tool for monitoring SQL Server estates.

September 05, 2019

US Signal announced the launch of its managed Website and Application Security Solution.

September 05, 2019

Pulumi Corporation announced the general availability of version 1.0 of its modern Infrastructure as Code platform.