How AI is Making Test Automation Smarter
November 22, 2021

Mark Lambert
Applitools

From facial recognition to self-driving cars, Artificial Intelligence (AI) and machine learning (ML) have become commonplace for many industries in recent years. In parallel, the software development industry has undergone a transformation of its own.

As customers look to engage more through digital experiences, businesses have been forced to evolve faster than ever before. Enticing and delighting customers in every aspect of product delivery has become "business critical," determining if the customer chooses, and continues, to do business with you over a competitor. Although the discipline of Quality Engineering has remained unchanged, every aspect of how quality is delivered has evolved. Businesses can no longer trade off quality vs. speed, as both quality and speed must be achieved for modern digital-first businesses.

Recently, two reports came out that speak directly to the intersection of these two trends and discuss how industry leaders are leveraging AI to modernize their approach to Quality Engineering in 2021 and beyond.

The first, from EMA (Enterprise Management Associates), is titled Disrupting the Economics of Software Testing Through AI. In this report, author Torsten Volk, Managing Research Director at EMA, discusses the reasons why traditional approaches to software quality cannot scale to meet the needs of modern software delivery. He highlights 5 key categories of AI and 6 critical pain points of test automation that AI addresses.

In addition, over the last couple of months Sogeti has been releasing sections of their State of AI applied to Quality Engineering 2021-22 report (with still more sections to come through February 2022). This comprehensive report is created in partnership with leading technology providers to provide a detailed examination of the current state of artificial intelligence across many use-cases in the field of Quality Engineering and centers around a key question — how can AI make quality validation smarter?

As the application of AI to testing continues to advance, it is important to understand its potential and how it can help improve the quality, velocity, and efficiency of Quality Engineering activities. Below, I’ll discuss some of the highlights from both of these reports and what they define as the future of Quality Engineering.

The Emergence of Modern Quality Engineering and the Need for AI

First, let’s talk about the reason why traditional approaches to software quality and test automation are no longer sufficient. The first section of the Sogeti report gives an overview of the business pressure to release faster and increasingly complex technical environments. As Torsten discusses in his report, modern software development teams are faced with many challenges that have driven up the complexity and cost of quality, such as the explosion of device/browser combinations and application complexity. Multiply this by the number of releases per month and you can quickly see that the traditional test automation tools can no longer scale to the challenges of modern software delivery.


Source: Disrupting the Economics of Software Testing Through AI, September 2021, EMA Research

The biggest problem with the traditional approach to test automation is that it scales linearly. The more, or faster, you need to test the more human and non-human resources you need — which only works if you have an infinite amount of resources (do you?).

With this in mind, the EMA and Sogeti reports discuss the ways modern organizations can leverage AI/ML to streamline their test automation practices and scale to meet the increased pace of software delivery.

AI Scales Test Automation to Identify Issues Before They Impact Users

When it comes to Quality Engineering, certain tools are capable of controlling the graphical user interface (GUI) or an application programming interface (API). Others analyze coverage and recommend additional actions, and some analyze log files in search of specific behaviors. These are just a few examples. But to increase developer productivity, there needs to be an understanding of each of these tasks and how they can be optimized.

How does this relate to AI? AI has the ability to apply algorithms and approaches used in tools to perform human-like tasks. For example, a developer can reason by examining an application to determine whether or not it has been properly tested. If the testing cycle has fallen short, they can then determine what additional testing needs to be done. AI has the potential to act in a similar manner. Although AI may require some training, once it has been trained it has the potential to continue to test the function even as the application evolves.

The EMA report details five key AI capabilities that can help organizations streamline and automate parts of their quality and testing workflow:

Test Creation/Smart Crawling: Automatic discovery of new and changed test requirements through the continuous analysis of changes in the application and natural language process (NLP) of documented requirements

Self-Healing: Continuous and automated remediation of broken test workflows

Visual Inspection: Training of deep learning models to inspect the application through the eyes of the end user

Coverage Detection: Automatic detection of the different paths that end users can take through the application and reporting of gaps in code coverage

: Automatic detection of system behavior that is inconsistent with the predictions of the AI/ML model


Source: Disrupting the Economics of Software Testing Through AI, September 2021, EMA Research

The report highlights the key advantage of each capability and then details how the capabilities can bridge the gap between the "ideal scenario" and "in real life" situation for six critical pain points of Test Automation: false positives, test maintenance, insufficient feedback, application complexity, device/use-case coverage and toolchain complexity.

Each capability is assigned a rating, ranking their current impact in 2021 and predicting their future impact in 2024. Visual inspection, implemented with Visual AI, has the highest rating for both current and future impact with the key advantage that it "Provides complete and accurate coverage of the user experience. It learns and adapts to new situations without the need to write and maintain code-based rules."

The EMA report goes on to add that "Smart crawling, self-healing, anomaly detection, and coverage detection each are point solutions that help organizations lower their risk of blind spots while decreasing human workload. Visual inspection (Visual AI) goes further compared to these point solutions by aiming to understand application workflows and business requirements."

As discussed in the most recent section of the Sogeti report, Shorten Release Cycles with Visual AI, Visual AI is already a mature technology, currently being adopted by leading brands across industries to accelerate the delivery of their digital experiences. The high levels of accuracy, and ability to handle dynamic and shifting content, ensures teams do not get overwhelmed with false positives. The automated grouping and categorization of regressions, coupled with root cause analysis, accelerates feedback and reduces test maintenance efforts. Visual AI provides test engineers with an additional "pair of eyes," leaving them to focus on areas that really need human intelligence — the power and impact of this approach is enormous.

From AI assistance to Autonomous Testing

Currently the industry is focused on having AI remove repetitive and mundane tasks, freeing humans to focus on the creative/complex tasks that require human intelligence. And as Torsten mentions in his report, "AI-based test automation technologies can deliver real ROI today and have the potential to address, and ultimately eliminate, today’s critical automation bottlenecks."

The ROI will further increase as we look to the future and the next big innovation for Quality Engineering, Autonomous Testing. Autonomous Testing will change the role of developers and testers from testing the application to training the AI how to use the application, leaving it to perform the testing activities, and then reviewing the results. This change will deliver a fundamental increase in team efficiency, reducing the overall cost of quality and enabling businesses to establish truly scalable Quality Engineering practices.

Mark Lambert is VP and Evangelist at Applitools
Share this

Industry News

December 02, 2021

Mirantis announced DevOpsCare, powered by Lens, a vendor-agnostic, fully-managed CI/CD (continuous integration/continuous deployment) product for any Kubernetes environment, offering developers higher levels of productivity more quickly.

December 02, 2021

The D2iQ Kubernetes Platform (DKP) is now available in AWS Marketplace, a digital catalog with thousands of software listings from independent software vendors that make it easy to find, test, buy, and deploy software that runs on Amazon Web Services, Inc. (AWS).

December 01, 2021

Bugcrowd announced the availability of Bugcrowd's cybersecurity solutions on the AWS Marketplace, providing customers with easy access, simplified billing, quick deployment, and streamlined license management.

December 01, 2021

Kublr received Microsoft Azure Arc-enabled Kubernetes validation, including for Azure Arc-enabled Kubernetes for Data Services.

December 01, 2021

CloudSphere achieved Amazon Web Services (AWS) Migration and Modernization Competency for discovering, planning, and helping enterprise customers move business services to AWS to reduce cost, increase agility and improve security.

November 30, 2021

JFrog introduced a new container registry and package manager for running JFrog Artifactory with Kubernetes clusters on-premises, in the cloud, or both.

November 30, 2021

Docker announced the availability of Docker Official Images directly from Amazon Web Services (AWS).

November 30, 2021

Weaveworks announced the general availability of Weave GitOps Enterprise, a GitOps platform that automates continuous application delivery and Kubernetes operations at any scale.

November 30, 2021

Amazon Web Services announced AWS Mainframe Modernization, a new service that makes it faster and easier for customers to migrate mainframe and legacy workloads to the cloud, and enjoy the superior agility, elasticity, and cost savings of AWS.

November 29, 2021

Quali announced the newest release of Torque Enterprise, which includes enhanced integration with Terraform, new custom tagging capabilities, and improved cost visibility dashboards, unleashing an entirely new level of self-service access to application environments on demand.

November 29, 2021

Vertical Relevance (VR), a financial services-focused consulting firm, achieved Amazon Web Services (AWS) DevOps Competency status.

November 18, 2021

Loft Labs announced the launch of Loft version 2 with a focus on ease of use that overcomes the major complaint that Kubernetes is complex and hard to set up.

November 18, 2021

Perforce Software announced new functionality to speed remediation of discovered defects in automated scans.

November 18, 2021

Lacework raised $1.3 billion in growth funding at a valuation of $8.3 billion.

November 17, 2021

Parasoft announced the 2021.2 release of Parasoft C/C++test, the unified C and C++ development testing solution for embedded applications.