Applying Chaos Engineering to Traditional Systems
September 01, 2021

Tony Perez
Skytap

Imagine: it's 2011 and Netflix has introduced Chaos Monkey, a tool that injects arbitrary failures into their cloud architecture to pinpoint design flaws. Today, resiliency engineering has advanced so much that "Chaos Engineer" is an actual job title. Enterprises such as Amazon, Facebook and Google now use chaos to understand their architectures and distributed systems.

While chaos engineering is usually performed on cloud-native software, it can also be used to strengthen the dependability of traditional data center applications that may never move to the cloud. What kind of tests might you run on these applications? Some might be:

■ Low network bandwidth and/or high latency

■ Disc volumes full

■ Application code failure

■ Database/server down

■ Expired certificate(s)

■ Hardware failures

This can be accomplished by using the cloud, which allows IT to create a production-like environment that includes the original's exact application components. All technical infrastructure encapsulating a depiction of an application is called an "environment." Chaos testing can be performed on the cloud replica without affecting production code.

It isn't necessary to change components to "cloud-native" with this approach; simply lift-and-shift, keep the same lines of application code, and use the same servers as the original. To get the most value from chaos testing, reuse the RFC-1918 address spaces you're using on-prem in the cloud. Every major cloud service has some type of network address translation (NAT) system, enabling each cloud-based environment that could be using cloned address spaces to communicate with other on-prem resources to prevent IP address collision.

Setting Up Your Chaos Testing Workflow

One reason to use the cloud for chaos testing on-prem applications is the ability to do a fast reset of the system between test rounds. Your goal should be quickly resetting or re-creating the system in the cloud, allowing you to quickly run several chaos test scenarios without wasting time resetting between each one. Here is a workflow to prepare for chaos testing.

1. Import your on-prem environment to the cloud.

2. Once running, save your application so you can recreate on-demand clones.

When importing the on-prem environment to the cloud, the goal is to duplicate the original on-prem system exactly. All the volume's data, networks, VMs and storage must be included.

Next, your test workflow is:

1. Deploy a duplicate application from your template/scripts.

2. Run your chaos tests and collect the results.

3. Once tests are complete, delete the entire test environment.

4. When you're ready for the next test, return to step #1.

While cloud-based infrastructure won't totally mirror on-prem environments, there are some workarounds. Say the design and size of your storage array (SAN) can't be duplicated in the cloud, meaning you won't be allowed to test due to "failing the SAN." In this instance, you could disconnect or alter a disk linked to a VM to mimic a failure, all in the cloud.

Resetting Your Test Environments After Use

By replicating your traditional on-prem application in the cloud, you can run aggressive tests to determine solutions to common issues, thereby extending the life of the application. However, when the testing eventually ruins the cloud-based application clone, how will you reset for future test rounds? Manually fixing things can take ages, but with cloud-based testing you unlock an unending supply of clones.

Different clouds approach this in different ways. No matter the strategy, the aim is to quickly rebuild a ready-to-use set of infrastructure and application components representing the original application. Companies already doing "infrastructure-as-code" may have the tooling and scripts to replicate the system from nothing.

Note that cloning IP address space is hard to do on-prem; don't be tempted to "Re-IP" (re-assign IP addresses and hostnames) to servers to prevent collision with the originals. This approach means you've essentially changed the original system's representation, so your chaos tests may produce incorrect results due to mismatched hostnames and IP addresses.

What once seemed impossible is actually a simple, elegant approach to improving on-prem applications that will never see the cloud. The cloud provides a 24/7 sandbox for you to create and destroy things, then quickly recover without risking your original systems. This approach works for original application systems of record, disaster recovery systems, and software development pipelines, making it a one-stop testing shop for traditional applications.

Tony Perez is a Cloud Solutions Architect at Skytap
Share this

Industry News

January 26, 2023

Ubuntu Pro, Canonical’s comprehensive subscription for secure open source and compliance, is now generally available.

January 26, 2023

Mirantis, freeing developers to create their most valuable code, today announced that it has acquired the Santa Clara, California-based Shipa to add automated application discovery, operations, security, and observability to the Lens Kubernetes Platform.

January 25, 2023

SmartBear has integrated the powerful contract testing capabilities of PactFlow with SwaggerHub.

January 25, 2023

Venafi introduced TLS Protect for Kubernetes.

January 25, 2023

Tricentis announced the general availability of Tricentis Test Automation, a cloud-based test automation solution that simplifies test creation, orchestration, and scalable test execution for easier collaboration among QA teams and their business stakeholders and faster, higher-quality, and more durable releases of web-based applications and business processes.

January 24, 2023

Harness announced the acquisition of Propelo.

January 23, 2023

Couchbase announced its Couchbase Capella Database-as-a-Service (DBaaS) offering on Azure.

January 23, 2023

Mendix and Software Improvement Group (SIG) have announced the release of Mendix Quality & Security Management (QSM), a new cybersecurity solution that provides continuous deep-dive insights into security and code quality to immediately address risks and vulnerabilities.

January 23, 2023

Trunk announces the public launch of CI Analytics.

January 23, 2023

Panaya announced a new Partnership Program in response to ongoing growth within its partner network over the past year.

January 23, 2023

Cloudian closed $60 million in new funding, bringing the company’s total funding to $233 million.

January 19, 2023

Progress announced the R1 2023 release of Progress Telerik and Progress Kendo UI.

January 19, 2023

Wallarm announced the early release of the Wallarm API Leak Management solution, an enhanced API security technology designed to help organizations identify and remediate attacks exploiting leaked API keys and secrets, while providing on-going protection against hacks in the event of a leak.

January 19, 2023

ThreatModeler launched Threat Model Marketplace, a cybersecurity asset marketplace offering pre-built, field-tested threat models to be downloaded — free for a limited time — and incorporated into new and ongoing threat modeling initiatives.

January 18, 2023

Software AG has launched new updates to its webMethods platform that will simplify the process by which developers can find, work on and deploy new APIs and integration tools or capabilities.