Beginning Your Test Automation Journey
Part 1 of a three-part series on introducing and building test automation into your application development and deployment pipeline
April 01, 2019

Drew Horn
Applause

Many companies are embracing test automation in an effort to move quicker and more efficiently, while simultaneously saving on costs. Test automation continues to experience growth year over year. In fact, in 2017 KPMG found an 85 percent increase in test automation in a two-year period across all industry domains.

Automation is a tool that all companies should prioritize when operating in an Agile development environment. However, while one of the main goals of test automation is to move faster, maturing test automation within a deployment pipeline should be a progressive, iterative process. If you try to automate too much, or automate the wrong things, you can end up slowing down and even hurting development and testing efforts.

The success you have will hinge on how you set up your automation practice, so the more attention you pay to the early stages, the more benefits you will realize in the long run.

In your journey to mature test automation within your deployment pipeline, you will move through three distinct stages: beginner, intermediate, and expert. We will explore the beginner stage in part one of this series.

Taking the First Step

At its core, automation is all about doing things better, faster, and cheaper. It improves coverage by increasing the number of tests you can run and improves test quality through repeatability. However, you need to crawl before you can walk and eventually run, so, like most things, it is advisable to start small with your test automation efforts.

The beginner stage of test automation starts with assessing the maturity of your testing organization and defining a goal of where you want to go. You’ll also need to develop a (or integrate an existing) framework on which to run your unit, integration, and functional tests. You can assess your testing maturity by self-evaluating across five key criteria:

1. Team – What is the team makeup? Where are the expertise gaps? Are QA and Dev teams working in silos?

2. Technology – What does the deployment pipeline look like? How are automated tests triaged? How is automation integrated with a VSC/TCM/BTS?

3. Process – When are tests written? How are bugs triaged and tests updated? How fast are sprint cycles? How does feedback guide test strategy?

4. Reporting – What is the test coverage? What are the common devices used? How are test results viewed? How is a "go/no-go" decision made? What is the automation ROI?

5. Pain – What are the existing pain points? What bugs have been missed?

Once you’ve assessed your maturity and developed a framework, you will want to focus on one main thing: Creating quick wins for your team. This will help to build confidence in your automation practice and get the full weight of the team involved.

Your development team can begin by checking in unit tests to immediately receive pass/fail feedback. After seeing these unit tests pass on a consistent basis, the team should then begin to prioritize developing a core set of functional smoke tests.

How often a set of smoke tests should be run initially will depend on your team's experience and bandwidth. A good rule of thumb is to start with a nightly smoke test that you can review each morning. This allows you to focus on code quality, test reliability, and process as opposed to performance and optimization. Eventually, though, you will want these automated tests up and running as part of your overarching build efforts so they are automatically triggered upon a successful build.

Steady iteration, where success is built on success, will help you gain valuable experience with your automation. Once you see your early unit and smoke tests passing consistently, you can consider adding more smoke tests to your automation suite. In general, you will want to keep the runtime of your tests to 15 minutes or less. As you advance your automation practice, you will find additional ways to run more smoke tests in a 15-minute period, such as test parallelization. We'll cover these strategies in the next article.

Though it’s not the most exciting part of test automation, setting up a documentation process is an important step in building a successful automation practice. The entire team should agree on a definition of "done" for automation tasks that need to be written out during sprint planning. While developing and documenting your automation processes, you should also look to answer the following questions: When should tests be run initially? How is the team notified when tests fail? Who is responsible for fixing failed tests? What is the action for triaging failures and logging bugs? Document the answers to the questions so everyone understands the protocol, otherwise you can expect to run into problems in the future as you scale up the speed and breadth of your practice.

How long you spend in the beginner stage of your automation journey will depend on a number of factors. In general, the hardest part of this stage is just getting the ball rolling. Try to set a goal of about eight weeks to stand up your initial test automation practice. This is a good pace that will all you to keep momentum. When time constraints, lack of prior experience, and competing priorities are large factors, consider pulling in turn-key solutions to get up and running quickly. Remember that your strategy can always be adjusted later. It's much easier make tweaks with an existing practice in place that can provide value to the team almost immediately.

Read Part 2: Building Confidence in Automation.

Drew Horn is Senior Director of Automation at Applause
Share this

Industry News

July 14, 2020

Perfecto by Perforce announced the launch of Android emulators and iOS simulators in its cloud testing platform.

July 14, 2020

SmartBear released new DevOps features in SwaggerHub, the company's API design and documentation platform.

July 14, 2020

Traceable, a new end-to-end application security monitoring platform, launched from stealth today with $20M in series A funding from Unusual Ventures and BIG Labs.

July 13, 2020

Docker announced a collaboration with Amazon Web Services (AWS) to simplify the lives of developers by allowing them to focus on application development, streamlining the process of deploying and managing containers in AWS from their local development environment.

July 13, 2020

Perforce Software announced the release of a combined JRebel and XRebel plugin for the Eclipse IDE.

July 13, 2020

Spectro Cloud announced that its first product—Spectro Cloud—is now generally available.

July 09, 2020

ShiftLeft released a new version of NextGen Static Analysis (NG SAST), including new workflows, purpose-built for developers that significantly improve security, while enhancing productivity.

July 09, 2020

RunSafe Security announced a partnership with JFrog that will enable RunSafe to supercharge binary protections via a simple plugin that JFrog users can deploy within their Artifactory repositories and instantly protect binaries and containers.

July 09, 2020

LeanIX closed $80 million in Series D funding led by new investor Goldman Sachs Growth.

July 08, 2020

Afi.ai introduced Afi Data Platform, a cloud-based replication and resiliency service that helps to monitor, predict downtime and recover K8s applications.

July 08, 2020

D2iQ announced the release of Conductor, a new interactive learning platform that enables enterprises to access hands-on cloud native courses and training.

July 08, 2020

SUSE entered into a definitive agreement to acquire Rancher Labs.

July 07, 2020

Micro Focus announced AI-powered enhancements to the intelligent testing capabilities of the UFT Family, a unified set of solutions designed to reduce the overall complexity of automating the functional testing processes.

July 07, 2020

Push Technology announced the launch of a new Service API capability for Diffusion Cloud, Push’s Real-Time API Management Cloud Platform.

July 07, 2020

Lightrun exited stealth and announced $4M in seed funding for the first complete continuous debugging and observability platform for production applications.