The Different Types of Software Tests
November 22, 2022

Terence Wong
Octopus Deploy

For software teams, testing makes sense, applications should be screened for bugs. But why is testing important for your business and how does it fit into DevOps?

Testing is part of Continuous Delivery that assures quality at each stage of the delivery pipeline before moving on to the next stage. DevOps is an iterative cycle of building, testing, and releasing software in short iterations. A comprehensive testing environment helps each iteration of the DevOps loop strengthen the quality of the product. A weak testing phase can mean defects progress to release, and developers need to fix bugs while the product is live. Development teams fall on both sides of the testing spectrum.

A surveyby Mabel on the state of testing in DevOps indicates that automated testing (at least 4–5 different types of tests) is key to customer happiness. The 2021 State of DevOps DORA Report reveals that continuous testing is an indicator of success, with elite performers who meet their reliability targets being 3.7 times more likely to use continuous testing.

In this post, I discuss automated and manual tests, and 2 common types of tests: functional and non-functional.

Manual and automated tests

Software tests can be manual or automatic. If you've used an app on your device and reported a bug, you've carried out a manual test. Automated tests are scripted in advance and executed by a machine, they compare an expected result with the actual result. Both methods of testing have their place in a software application, however manuals tests are slower and require an environment for the testers.

As developers write automated tests in advance, the errors found in manual testing can inform automated tests to strengthen the test suite. Manual tests are suitable when opinion and nuance play a role, like user experience. There is no pre-determined result for an automated test to check in these cases.

Automated tests are near-instant and execute in the hundreds or thousands at runtime. Automated tests check for functionality and make sure every line of code and feature works as intended. In a DevOps process, automated tests enable Continuous Delivery by giving a test coverage of the application. If you want to set up your application with a test coverage, you install automated tests for every component of the application.

When you add new features to a release, you can run the tests to identify whether test coverage has decreased. You can use the results to identify bugs for the new release. Automated tests complement a Continuous Delivery DevOps strategy. The more tests you automate, the faster your application can iterate and cycle through the DevOps loop of building, testing, and releasing.

Functional and non-functional tests

There are many tests you can perform on your application. One way to categorize tests is functional and non-functional.

Functional tests ask questions like:

■ Does this button work?

■ Does one module work with another module?

■ Does the user journey work from the start of the experience to the end?

Maintenance testing checks if the application has retained all functionality from version to version. It asks whether any functionality in the application has regressed between versions. I include maintenance testing under functional tests because it relates to functionality, though some sources list it as a third type.

Non-functional tests check the way a system operates rather than the functions of the system. Non-functional tests asks questions like:

■ How secure is the application?

■ How much load can the application handle?

■ Can the application scale if needed?

Examples of functional tests

Unit tests
Unit tests test an individual unit of code for functionality. In the example below, we test a function, which is a block of code that does one task.

1. Test if a function works. This function averages the weather for the previous 14 days.
2. Give the function the weather inputs for the previous 14 days.
3. Check whether the expected output matches the actual output.

Integration tests
Integration tests verify the functionality between 2 or more modules. This example tests the integration between the e-commerce storefront and the shopping cart module.

1. Load the e-commerce store and add some items to the cart, and go to checkout.
2. Check that the correct number of items are in the cart and the listed price is correct.

Smoke tests
Smoke testing is preliminary testing to reveal failures that can result in rejection of a release.

1. Does the web server return a 200 OK response?
2. Can I ping the database?

Acceptance tests
Acceptance testing confirms your application is working according to a requirements specification. In this example, there's a requirement for a rewards system to work with an application. The tests check for the expected behavior of a rewards system.

1. If a user tries to purchase a product with rewards points and they have enough points, the purchase price should be $0.
2. If a user tries to purchase a product with rewards points and they do not have enough points, the purchase price should be $20.

Examples of non-functional tests

Load and performance tests
Load and performance tests check metrics like speed, response time, and resource usage of the application.

1. Measure the loading time of the home page and flag if it exceeds a threshold value.
2. Measure database response time when handling 100 or more concurrent requests. Flag if response time exceeds threshold value.
3. Under increasing load, test how many nodes an application needs to recover.

Security tests
Security tests check for security related weaknesses in the system.

1. Scan log files for sensitive information. eg. credit card numbers or email addresses.
2. Scanning for long running sessions as a sign that session handling isn't working as expected.

Scalability tests
Scalability tests test for issues related to scaling the application.

1. Under increasing load, test how many nodes an application needs to recover.
2. Test the length of time required for more nodes to be added and the application to recover. Chart how quickly you can expect 90% of scale up events to complete in.


Testing is essential to DevOps processes, making sure your software meets quality requirements before moving to the next stage. Research has shown that automated testing is a strong indicator of customer happiness and successful teams.

Tests can be run manually or be automated, and there are 2 main types of tests: functional and non-functional.

A robust testing environment fits well with Octopus Deploy. Octopus takes care of the release and deployment, and automates sections of the DevOps lifecycle, making complex deployments easier. To learn more about the importance of testing, read our post about why you should track vulnerabilities after deployment.

Terence Wong is a Technical Content Creator at Octopus Deploy
Share this

Industry News

March 20, 2023

To meet the growing demand for Oracle Container Engine for Kubernetes (OKE) with global organizations, Oracle Cloud Infrastructure (OCI) is introducing new capabilities that can boost the reliability and efficiency of large-scale Kubernetes environments while simplifying operations and reducing costs.

March 20, 2023

Perforce Software joined the Amazon Web Services (AWS) Independent Software Vendor (ISV) Accelerate Program and listed its free Enhanced Studio Pack (ESP) in AWS Marketplace.

March 20, 2023

Aembit, an identity platform that lets DevOps and Security teams discover, manage, enforce, and audit access between federated workloads, announced its official launch alongside $16.6M in seed financing from cybersecurity specialist investors Ballistic Ventures and Ten Eleven Ventures.

March 16, 2023

Hyland released Alfresco Content Services 7.0 – a cloud-native content services platform, optimized for content model flexibility and performance at scale.

March 16, 2023

CAST AI has announced the closing of a $20M investment round.

March 15, 2023

Check Point® Software Technologies introduced Infinity Global Services, an all-encompassing security solution that will empower organizations of all sizes to fortify their systems, from cloud to network to endpoint.

March 15, 2023

OpsCruise's Kubernetes and Cloud Service observability platform is certified to run on the Red Hat OpenShift Kubernetes platform.

March 14, 2023 released an update to the platform, delivering productivity for data teams.

March 14, 2023

CoreStack and Zensar announced a strategic global partnership. CoreStack will provide its AI-powered NextGen cloud governance and FinOps capabilities, complementing Zensar’s composable cloud operations offering.

March 14, 2023

Delinea introduced the Delinea Platform, a cloud-native foundation for Delinea's PAM solutions that empowers end-to-end visibility, dynamic privilege controls, and adaptive security.

March 13, 2023

Sysdig announced a new foundation that will serve as the long-term custodian of the Wireshark open source project.

March 13, 2023

Talend announced the latest update to Talend Data Fabric, its end-to-end platform for data discovery, transformation, governance, and sharing.

March 13, 2023

Descope has raised $53M in seed funding and emerged from stealth to launch a frictionless, secure, and developer-friendly authentication and user management platform.

March 09, 2023

Loft Labs announced Loft v3 with new capabilities and flexibility for platform teams to build and enable their development teams with a self-service Kubernetes.

March 09, 2023

AWS Application Composer is now generally available.