2020 was one of the most transformative years for software testing to date. Teams were forced to adapt to completely new work environments and learn to develop, test and innovate at warp speed. At Perfecto by Perforce, we were intrigued by the rollercoaster that was 2020 and wanted to glean more insights into the unique testing trends and challenges that surfaced as a result. As such, we surveyed more than 700 DevOps professionals for our 2021 State of Test Automation Report.
At a time when releases needed to happen faster, findings reported that teams actually slowed down — only 9% released daily, down 14% from the previous year. Digging into these numbers even more, we uncovered several key takeaways from the data, but the overarching theme was clear: teams need intelligent test automation and they need it now.
Teams Are Making Good Progress, but Struggling to Adopt Test Automation Across the Board
In an increasingly digital world, it's critical for teams to get on board with automation, as it allows them to do more testing in less time. Not only that, but it reduces the need for repetitive, manual tasks and can also deliver better results because of its ability to increase test coverage.
However, teams are struggling to fully get onboard. According to the report, 10% of teams are manually testing all tests. To date, 20% just started automating tests and nearly 50% are automating fewer than half of their them. Across the board, teams cited the lack of resources for creating test automation and proper coverage for test automation scenarios as being their top two biggest hurdles.
For those teams rarely automating tests, they were more likely to release once or twice per year and less likely to achieve continuous testing, integration and delivery. On the other hand, those with more mature automation practices were more likely to release daily or weekly, but also were challenged by false negatives, test flakiness and instability.
The "Shift Left" Mentality Is Taking Off, but Manual Testing Is Holding Teams Back
Development teams are really striving to "shift left" and begin testing as early in the lifecycle as possible so they can release more, faster. However, a majority find it difficult to get to that point. Our research found that 71% of organizations still struggle to fit test automation into the build process. Instead, they wait to test after the build and release less as a result.
Mature DevOps requires automated checkpoints throughout the build process. A major pain point for testing earlier can again be attributed to manual testing. Asked about the biggest obstacles to shifting left, 29% said manual testing, followed by 20% who said test cycle analysis and 18% who said test environment setup.
The smaller percentage of respondents who have been successful in shifting left faster are those who have democratized quality assurance across the entire organization, and implemented such practices as feature flags, A/B testing and cloud-based testing.
Top Priorities for the Coming Year Are Mobile Driven as a Result of the Pandemic
If 2020 showed us anything, it's how dependent we are on our mobile phones. According to an eMarketer report, US adults were projected to spend an additional 23 minutes per day on their phones amidst the pandemic. As a result, more than 60% of respondents to our survey said native mobile and mobile web apps are important or very important to their testing strategy in 2021. Within, a growing 13% are increasingly interested in foldable phones.
Other priority trends for 2021 include incorporating artificial intelligence (AI) and machine learning (ML) into codeless testing (61%), followed by 5G (29%) and progressive web apps (PWAS) (21%).
As a Result of Testing Challenges, Teams Are Putting Emphasis on Automation Investment
Digital transformation had been a long time coming even before the pandemic drastically expedited the transition. In less than a year, organizations had to go almost entirely digital as their sole way of engaging with customers, employees and partners alike. Accordingly, a greater emphasis has been put on web and mobile technologies, which require more rigorous testing, greater coverage and continuous monitoring.
The DevOps teams that succeed in this new digital age understand the value of test automation, and are actively addressing the automation skills gap by vetting more behavior-driven development (BDD) and scriptless solutions. Despite all the barriers to automation and shifting left, nearly half of all respondents said they plan to invest in test automation solutions within the year.
For those planning to invest, over 40% have an interest in using intelligent test automation solutions. The frameworks at top of mind include codeless (44%), Cypress (40%), Flutter (19%), Playwright (10%) and other (31%).
However, there is still a large population — 29% — that has no plans to invest at all.
Looking at Test Automation in 2021 and Beyond
Overall, the report revealed that the majority of organizations understand and support the implementation of automation and continuous testing, yet few are actually hitting it out of the park. In many instances, there are even misalignments within these organizations in which executives believe they are more automated than developers report.
Now that more testing is performed on digital platforms than ever before, enterprise teams must close the gap and embrace test automation tools if they want to keep up with their competition — and getting the C-suite on board is the first step.
