Building Confidence in Automation
Part 2 of a three-part series on introducing and building test automation into your application development and deployment pipeline
April 15, 2019

Drew Horn
Applause

In Part 1 of this three-part series, we covered the first steps of introducing automated testing into your software development lifecycle. Now that you've done the early work of codifying manual tests into an automation framework and achieved some quick wins with initial smoke tests, you can continue to build confidence in test automation.

Start with Part 1: Beginning Your Test Automation Journey

Working with Test Case Management Tools

The most important competency of this intermediate SDLC stage is that you establish a test case management system (TCM) and reporting structure. It's key to make sure that all of your results are going to a single place to be seen from a single view. By doing that, you have a consistent view into any failures and can comfortably decide whether or not to deploy.

Furthermore, this approach allows you to merge your manual and automation testing efforts into a single system, with a single source of truth. As these two processes are merged together (which we will cover later in this article) you will enable your practice to scale properly without hitting unnecessary bottlenecks.

With a single TCM in place, you can now more effectively put quality gates in place as part of your deployment pipeline. Each testing stage in the pipeline (e.g. smoke testing, manual regression, test automation) should have a quality gate defined that determines whether or not the build should continue through the pipeline for additional testing. Implementing quality gates at each stage of the process helps your team identify build issues earlier. The earlier build issues are identified, the more cost-optimized your practice will be. This is especially important as you scale your practice and increase test coverage.

Unifying Manual and Automated Testing

After your test cases have been merged into a single repository and an assessment has been completed to determine which tests should be run manually, and which should be automated, it becomes much easier to combine your manual and automation testing efforts together and embed them into your deployment pipeline. This is a must for any production-grade QA practice looking to scale.

While the process varies from team to team, in general, embedding automated and manual testing together into your deployment pipeline can be seen as a four-step process:

1. Define a change set given your evaluation and goals– More specifically, what technical changes to your deployment pipeline and/or your manual processes need to be implemented or documented in order to embed all testing into the pipeline? For example, once an automated smoke test is complete, should a QA lead be notified so that they can initial manual testing? When manual testing is done, how does a key stakeholder review the results and determine if the quality gate should allow the build to the next step in the pipeline for additional downstream (possibly non-functional) testing? These are the types of questions that should be answered so you have a clear playbook on how to move forward.

2. Test the solution out-of-band– Changes to your pipeline and processes should also be tested. A great way to test your new process without impacting the current workflow is to do it out-of-band. For example, you could build a job on your CI server that runs automated regression tests but does not impact the existing pipeline flow. Doing so allows you to review the process and iterate as needed until all teams are ready to move such a process directly inline.

3. Train your team on the process– There will almost always be manual processes, so it is important to solidify these processes by training your team throughout the development, integration, staging, production, and feedback stages.

4. Implement changes into CI pipeline– Finally, once the changes to the pipeline and processes have been vetted and all teams are trained, you can make the switch.

Scaling

The last component of getting your automation out of an initial or beginner stage is starting to scale. With the above tools and processes in place, you should feel comfortable adding more automated tests and platforms. As your test matrix increases and the frequency of runs increases, executing tests in parallel becomes a high priority. Ideally, your automation framework should support the ability to execute tests in parallel. A challenge is making sure the tests that your team has developed work well when run at the same time. To do this, make sure your tests are as atomic and idempotent as possible. The state of the application after each test should (if possible) be the same as when it started. If this isn't possible, try to set up some test data for your tests in a way that each test relies on its own data. If test data used in one test can impact another, you will have a very difficult time debugging test failures.

If your framework doesn't support running tests in parallel, you could also set up separate jobs on your CI server to run groups of tests at the same time. This works, but generally adds additional complexity to your pipeline that could instead be encapsulated in your framework.

Read Part 3: Achieving Expert Status in Test Automation.

Drew Horn is Senior Director of Automation at Applause
Share this

Industry News

February 25, 2021

Red Hat announced Red Hat OpenShift 4.7, the latest version of the company’s enterprise Kubernetes platform.

February 25, 2021

Granulate announced the release of its open-source platform, the G-Profiler, a production profiling solution that measures the performance of code in production applications to facilitate compute optimization.

February 25, 2021

Checkmarx announced the launch of KICS (Keeping Infrastructure as Code Secure), an open source static analysis solution that enables developers to write more secure infrastructure as code (IaC).

February 24, 2021

Applause launched its Product Excellence Platform (PEP).

February 24, 2021

Mabl announced the beta release of their new native desktop application that empowers users to easily automate testing for browsers, mobile browsers, and APIs.

February 24, 2021

D2iQ announced the general availability of D2iQ Kaptain, the cloud native end-to-end platform for running ML workloads on Kubernetes.

February 23, 2021

Edge Delta announced new capabilities for dynamically processing and routing logs, metrics, and traces -- allowing DevOps, Security, and SRE teams to analyze large volumes of streaming data without the cost, delay, or complexity of requiring data to be indexed, helping customers decouple where machine data is analyzed from where it is stored.

February 23, 2021

env0 announced a remote run SaaS solution for the automation of Terragrunt workloads across multiple Terraform modules.

February 23, 2021

Platform9 closed its Series-D funding with an additional $12.5 million for a total of $37.5 million.

February 22, 2021

IT Revolution opened its Call for Presentations for both DevOps Enterprise Summit 2021 events in Europe and US.

February 22, 2021

Solo.io announced the general availability of Gloo Mesh Enterprise, an enterprise-grade, Kubernetes-native solution for service mesh management.

February 22, 2021

Copado closed $96 million in Series B funding, bringing the total funds invested in the company to $117 million.

February 18, 2021

ShiftLeft released ShiftLeft Illuminate, a new solution that leverages ShiftLeft technology to identify insider attacks, offer remediation advice and reduce overall risk to organizations’ software code base.

February 18, 2021

CircleCI introduced new platform updates to increase the control, protection, privacy, and confidence of today’s engineering teams.

February 18, 2021

OutSystems announced a $150 million capital raise.