Test Smarter Not Harder: Shift Testing Left and Right with Test Impact Analysis - Part 2
August 19, 2019

Mark Lambert
Parasoft

Software testing is still a bottleneck, even after the implementation of modern development processes like Agile, DevOps, and Continuous Integration/Deployment. In some cases, software teams aren't testing nearly enough and have to deal with bugs and security vulnerabilities at the later stages of the development cycle, which creates a false assumption that these new processes can't deliver on their promise. One solution to certain classes of issues is shift right testing, which relies on monitoring the application in a production environment, but it requires a rock solid infrastructure to roll back new changes if a critical defect arises.

Start with Test Smarter Not Harder: Shift Testing Left and Right with Test Impact Analysis - Part 1

Testing Smarter, Not Harder, by Focusing Your Testing

Most software isn't fully tested, and the decision of what to test is essentially based on developers' best guesses about what is critical functionality. During a SCRUM sprint, or an iteration in other processes, it's difficult to determine what to test, because, of course, "test everything" isn't an option. Since timelines are short, only parts of the software that were updated by the latest functionality can be tested, but exactly what code is impacted is usually unknown. Test automation helps, but without understanding exactly where and what to test, test coverage is inadequate.

These shortcomings can be overcome by using test impact analysis, which is a multivariate analysis of test coverage, code changes, and dependencies that pinpoints exactly what code needs to be tested. In addition, these exact tests can be scheduled and executed automatically.

Test impact analysis works at the developer level within the IDE, collecting information about which code is exercised by which tests, and applies that information within the developer's IDE as the developer is changing code, enabling the developer to easily identify and execute the specific tests that need to be run to verify that the changed code doesn't break any tests. Also, keeping track of which affected tests have been run, which have passed, and which have failed, makes it easy for the developer to determine which tests still need to be run, or which tests have failed and need to be addressed. Once all tests have been run and are passing, the developer knows that it's safe to commit their code and move on.

Test impact analysis works within a CI/CD process by integrating seamlessly into a project's build system, to get immediate feedback on changes. Test impact analysis identifies which code has changed since the baseline build (i.e. the last nightly build), determines which tests need to be run to exercise that code, and then runs just that subset of tests. This workflow enables teams to set up CI jobs that only run tests based on the most recent code changes, shrinking the amount of time it takes to run a CI job from hours to minutes.

Test impact analysis provides the following key benefits:

Understand what each test covers: Automatically correlating test execution data with test coverage data identifies which tests need to be run, based on the code currently being developed. Users save time without having to run unnecessary tests, and teams benefit from immediate feedback during development and after code check-in.

Understand what has changed: Developers often don't know which tests to run to validate code changes, so they either check their code in without running any tests (a very bad practice), or they run only one or two tests that they know about (which easily misses some), or they run all of their tests (which wastes time). Test impact analysis immediately identifies which tests are related to which code changes and are automatically executed. Checked-in code becomes more stable since it has been thoroughly tested prior to check-in.

Focus on tests that validate changes and impacted dependencies: Identifying and running just the set of tests needed to verify all of the code changes, and affected dependencies, that have been committed since the last baseline build (usually the nightly build), significantly decreasing the amount of time it takes to run CI. This allows teams to benefit from a true CI process.

Immediate and Ongoing Feedback: Identifying not just direct dependencies between tests and code, but indirect dependencies as well, test impact analysis helps teams understand as soon as possible after code is checked in whether the code broke any tests.

Summary

To greatly decrease the testing bottleneck in development to improve the efficiency of the "saw tooth" effort that testers put into every iteration, development teams can benefit from test impact analysis technology. Test automation with test impact analysis means focusing testing specifically on changes made during each iteration, and testing exactly what needs to be tested, automatically. These teams optimize their in-development testing effort with instant feedback on what needs to be done, what code fails testing, and what other code is impacted by new changes.

Mark Lambert is VP of Products at Parasoft
Share this

Industry News

June 01, 2020

IT Revolution announced a full conference agenda for DevOps Enterprise Summit London, June 23-25, 2020.

June 01, 2020

Caltech CTME announced that Simplilearn, a global provider of digital skills training, will collaborate with CTME (Caltech's Center for Technology and Management Education) to offer a specialized Post Graduate Program in DevOps software engineering.

June 01, 2020

DevOps Institute, a global member-based association for advancing the human elements of DevOps, announced the introduction of its SKILup Playbook Library, a dynamic collective body of knowledge (cBok) that aligns thought leadership from industry experts with a set of dynamic, orchestrated artifacts, research and assets.

May 28, 2020

Docker has extended its strategic collaboration with Microsoft to simplify code to cloud application development for developers and development teams by more closely integrating with Azure Container Instances (ACI).

May 28, 2020

Eggplant announced updates to its Digital Automation Intelligence (DAI) platform.

May 28, 2020

Aptum launched its Managed DevOps Service in partnership with CloudOps, a cloud consulting and professional services company specializing in DevOps.

May 27, 2020

Red Hat announced an expansion of its application services portfolio with the addition of Quarkus as a fully supported framework in Red Hat Runtimes.

May 27, 2020

Couchbase has completed a $105 million all-equity Series G round of fundraising.

May 27, 2020

Aqua Security closed a Series D round of $30M led by Greenspring Associates.

May 26, 2020

GitLab is releasing 13.0 of its DevSecOps platform to enable organizations to efficiently adapt and respond to new and dynamic business challenges.

May 26, 2020

Solo.io announced the availability of the Istio Developer Portal to streamline the developer onboarding process for improved developer experience and increased productivity with added security features.

May 26, 2020

WhiteHat Security will offer free application scanning services to any education institution to support secure online learning.

May 21, 2020

Exadel announced the Grand Prize winner of the “Appery.io COVID-19 Virtual Hackathon.”

May 21, 2020

CloudBees announced significant advances for its Software Delivery Management (SDM) platform – integrations with additional continuous integration and continuous delivery (CI/CD) engines, including Google Cloud Build and Tekton, and extension of the availability of CloudBees’ SDM Preview Program.

May 21, 2020

OutSystems is announcing over 70 development accelerators that ensure web and mobile applications created on the OutSystems low-code development platform can comply with the highest accessibility standards and regulations.