Optimize CI Pipelines with AI Risk-Based Testing
June 06, 2023

Johnston Harris
Appsurify

DevOps teams are pressured to have their builds running smoothly … but we all know that's hard to achieve. CI optimization is a constant battle against expectations and resources. With the number of automated tests that run in the pipeline, to flaky tests … to simply not enough CPU, DevOps teams are rarely 100% satisfied with their build process.


A build that used to take 10 minutes to finish, once you add automated tests into the picture, suddenly takes 45 minutes to complete. And if failed tests are set to "retry" — that 45-min build can easily double in time!

So enters AI to save the day! We can't get away with a blog post without mentioning AI these days, can we? Apparently not. Because it really does help DevOps teams with their CI optimization.

Pressure to keep the pipeline moving is fierce, and so keeping workflows running optimally is always top of mind for any Devops leader. Although there seems to be an abundant amount of tooling circling around the market, few exist that help optimize workflows to achieve a streamlined build process.

Here are the areas we see AI helping DevOps leaders with CI optimization.

How to Optimize a Pipeline Running a lot of Automated Tests

When automated tests come into the pipeline, the historic answer has always been: add CPU and run those tests in parallel, that should fix the problem. Before the team knows it, DevOps is reciting the famous words from Scotty in Star Trek "Captain, I'm giving her all she's got!"
Parallel threads clamp down on test run times initially, however, teams easily hit a maximum thread count where it's either cost prohibitive to add more CPU or that parallelization is maxed out and no longer reduces build times.

What's the next option?

This is where AI has come fully into picture. There are promising developments in the AI risk-based testing space that are helping DevOps teams with their CI optimization efforts. Leveraging AI, DevOps can set their CI pipeline to auto-select and execute only the automated tests impacted by recent developer changes on a per Commit basis.

Rather than running 100% of the testsuite in maximum parallel threads, the CI Pipeline executes a smart subset, such as 20% of tests, while still catching the regressions. Therefore, the CI Pipeline doesn't require as many resources to find regressions and can do so in a much more fast and efficient way.

Example for CI Optimization

An E2E testsuite with 1,000 tests running through a CI pipeline in 10 parallel threads takes 60 minutes to complete. 60 minutes is a long time. So how can they get 60 minutes down farther without throwing more hardware at the issue?

Inserting an AI risk-based testing tool into the pipeline would allow it to auto-select and execute only the tests relevant to developer changes, and thereby reduce the number of tests needing to be executed.

In this example, if Smart Test Selection is set at 10%, it would prioritize the top 100 E2E tests for execution based on the recent developer commits. The execution time would drop from 60 minutes down to approximately 6 minutes to complete.

By adding an AI risk-based testing tool into the CI pipeline, the team can now run against the full 1,000 tests while AI picks the top 100 tests given a recent change on a per PR basis to save time and resources. A worthy option to consider over just throwing more hardware at the issue.

How to Optimize a Pipeline with Flaky Tests

Flaky tests are the bane of existence for both DevOps and QA teams alike. These tests are a constant distraction, from breaking builds to chasing false positives. A test that fails due to no underlying code changes is simply a test that will cause someone to waste time.

There are a few avenues DevOps leaders can pursue to prevent flaky test distraction, such as retries. For example, if a test fails, retry it 3 to 5 times again to ensure it fails consistently each time to warrant raising a real defect.

The downside to this methodology, once again, it prolongs the CI build process!

A 10-min build with 5-min worth of automated tests and a high degree of flakiness can be extended to 25-min in some cases. Retries are good to help determine flakiness, but there is a tradeoff, which is time and resources.

What's the next option?

Giving clean feedback to developers is hard. AI can do this and block out flaky distraction. AI can recognize trends in test failures that pick up whether a test failed due to a real defect or not.

AI-models can allow DevOps teams to only fail CI builds on real failures, and avoid the flaky noise inherent in so many testsuites. If there's a degree of flakiness in every run, the builds can Pass/Fail only on real defects, thereby giving clean signals to Developers.

How does AI know to prevent Flaky Tests?

The AI trains from historical data, learning about the tests and defects, and builds out an AI Model that can determine when a test failed on a real defect or is a false positive.

For example

A UI testsuite with 1,000 tests has 5% inherent flakiness. After every run, the team must review 50 tests failures that are flaky — a major distraction that may cause developers to ignore the results (which is a very bad thing).

When the AI Model is trained on this testsuite, if there are no real defects introduced, the build, even with 5% flakiness, can be set to Pass. Only on which the AI Model determines to be a Real Defect, will the Build Fail.

Tooling such as this, which is highly configurable to team preference, can be leveraged to save team distraction, keep builds running smoothly, and optimize CI pipelines.

Conclusion

The advent of CI pipelines has accelerated team output to deliver faster than ever, however, as more weight is put on these pipelines, optimization strategies need to be put in place to maintain speed and efficiency.

Automated Tests and Flaky Tests can burden the CI Build process, however, new AI technologies, such as an AI-Powered Risk Based Testing tool, exist to help counteract these CI bottlenecks to keep workflows streamlined and efficient.

Johnston Harris is CEO and Co-Founder of Appsurify
Share this

Industry News

September 21, 2023

Red Hat and Oracle announced the expansion of their alliance to offer customers a greater choice in deploying applications on Oracle Cloud Infrastructure (OCI). As part of the expanded collaboration, Red Hat OpenShift, the industry’s leading hybrid cloud application platform powered by Kubernetes for architecting, building, and deploying cloud-native applications, will be supported and certified to run on OCI.

September 21, 2023

Harness announced the availability of Gitness™, a freely available, fully open source Git platform that brings a new era of collaboration, speed, security, and intelligence to software development.

September 20, 2023

Oracle announced new application development capabilities to enable developers to rapidly build and deploy applications on Oracle Cloud Infrastructure (OCI).

September 20, 2023

Sonar announced zero-configuration, automatic analysis for programming languages C and C++ within SonarCloud.

September 20, 2023

DataStax announced a new JSON API for Astra DB – the database-as-a-service built on the open source Apache Cassandra® – delivering on one of the most highly requested user features, and providing a seamless experience for Javascript developers building AI applications.

September 19, 2023

Oracle announced the availability of Java 21.

September 19, 2023

Mirantis launched Lens AppIQ, available directly in Lens Desktop and as (Software as a Service) SaaS.

September 19, 2023

Buildkite announced the company has entered into a definitive agreement to acquire Packagecloud, a cloud-based software package management platform, in an all stock deal.

September 19, 2023

CrowdStrike has agreed to acquire Bionic, a provider of Application Security Posture Management (ASPM).

September 18, 2023

Perforce Software announces BlazeMeter's Test Data Pro, the latest addition to its continuous testing platform.

September 18, 2023

CloudBees announced a new cloud native DevSecOps platform that places platform engineers and developer experience front and center.

September 18, 2023

Akuity announced a new open source tool, Kargo, to implement change promotions across many application life cycle stages using GitOps principles.

September 14, 2023

CloudBees announced significant performance and scalability breakthroughs for Jenkins® with new updates to its CloudBees Continuous Integration (CI) software.