How Continuous Deployment Improves (and Combats) Rollbacks
June 14, 2023

Stephen Atwell

Today's business environment is fast-paced and overwhelmingly digital, making it more critical than ever for organizations to frequently and successfully deploy innovative software. However, as organizations navigate the intricacies of modern software development, they often face code-breaking bugs and errors that only arise post-deployment. The costliness of these deployment errors makes it paramount for developers to retain access to rollback capabilities.

Rollbacks are a critical safety net in software deployment because they allow developers to revert code to its previous state if an issue arises. Rollbacks provide an immediate solution to user-facing functionality errors and minimize the impact of costly disruptions. Consider the following:

■ According to Information Technology Intelligent Consulting (ITIC), an hour of downtime costs enterprises anywhere from $1 million to $5 million.

■ Meanwhile, small and medium businesses (SMBs) that experience an hour's downtime rack up around $10,000 of losses per hour — more than enough to shutter certain SMBs.

Given these figures, it's essential for software organizations of all sizes to have an effective rollback strategy in place.

Yet frequent rollbacks also delay the implementation of new features and waste developer time, causing frustration. Thus, while DevOps teams should prioritize easy rollback capabilities, they should ultimately work toward a future with more sustainable deployment practices and tools.

One such practice DevOps teams can prioritize today is continuous deployment (CD). CD significantly reduces the need for rollbacks by testing and validating code pre-deployment while also enabling teams to seamlessly revert to older working code versions when necessary. Let's unpack how CD can mitigate the necessity of rollbacks and, when necessary, ensure rollback processes are seamless.

How Continuous Deployment Reduces Your Organization's Reliance on Rollbacks

Although rollbacks are a necessary component of any deployment strategy, they are suboptimal for apparent reasons: they waste developer time and resources and delay the frequency of essential code updates. According to Armory research, 52% of engineering and operations leaders measure success by deployment frequency. Rollbacks jeopardize this leading indicator of success.

Developers who rely on CD improve productivity and the overall efficiency of their organization's software development lifecycle — eradicating the need for rollbacks in most situations. Leading CD tools offer policy enforcement, permissions and entitlements management, and architectural best practice enforcement. They also leverage progressive rollout strategies such as:

Blue/green deployments — This technique allows developers to run two versions simultaneously in the production environment. One operates as the live version and the other as a test version, with the capability to change which version traffic reaches as necessary. Developers test new features on the test version, and that code only receives production traffic after validation. Because both the new and old version are running, if a problem is encountered, traffic can be instantly redirected back to the old version, rolling back the change. Blue/green deployment simplifies testing and reduces risk by empowering developers to continuously integrate new features and deploy code changes with zero downtime.

Canary deployments — To stress-test code changes while decreasing their blast radius, developers can send the new version a small subset of traffic. As with blue/green deployment, canary deployments run old and new versions in parallel. With a canary strategy, instead of redirecting traffic all-at-once, traffic is slowly shifted to the new version in small increments for beta testing. This provides developers with crucial information about code viability in a live environment. Once developers deem code changes robust, they can deploy them to the next batch of users, eventually transitioning all users to the new application. Leading canary deployment tools also ensure that rollbacks — when necessary — are instantaneous. For example, some canary strategies only upgrade a subset of copies of the application (e.g. servers, vms, containers or pods) to the new version. If three pods are running the old version and one is running the new version, then the new version is receiving 25% of traffic. But controlling the traffic split via the number of running copies limits rollback speed because the application must recall previous versions to execute a rollback. Leading canary strategies have the capability to switch to a service mesh approach, as required by the client. This approach typically provides more fine-grained control of the traffic split, while also allowing for instantaneous rollbacks.

Progressive deployment strategies such as these are complex, but automation allows even the smallest teams to successfully employ such practices. Many modern CD tools come with off-the-shelf implementations of these strategies, enabling your DevOps team to benefit from them while focusing crucial time on implementing functionalities unique to your application.

The Power of Automated Rollbacks

As discussed, every developer's goal should be rollback reduction. However, there are instances in which a rollback is the only immediate solution to a significant bug or code break.

For example, say a top client relies on an organization's software to prepare their board presentation materials. While preparing for a next-day presentation, this client encounters a system-breaking bug. They submit an urgent support ticket, but the organization's DevOps team cannot address root-cause issues until the next day, at the earliest.

In cases like this, developers must use a reliable rollback tool — ideally, one with automated capabilities and one-click rollbacks, which can automatically restore a previous version of code without developer involvement when an update reaches a specific threshold of predetermined criteria (for instance, once user traffic has crashed a critical piece of your organization's software). These tools improve software consistency and reliability, reverting users to a previous application version complete with all dependencies and mechanisms required for service delivery.

And quick restoration is key. In the above example, the client's retainer likely represents a significant proportion of the software organization's revenue. When trust is lost in this relationship, it can translate to millions of dollars in losses. A painless, expedient rollback may be the linchpin in maintaining this robust relationship.

Moreover, leading CD tools have other avenues to prevent costly, ill-fated deployments, such as the automated rollback of containerized applications. For example, developers can roll back Kubernetes deployments based on their complexity and the number of deployed artifacts. Additionally, developers can use CD tools to guard against unwanted rollbacks through conditional checks. This feature ensures a rollback is strictly necessary by first verifying that an unrelated deployment pipeline error (e.g., network interference) hasn't occurred.

Reaching the Deployment "Sweet Spot"

Ideally, an organization would never rely on rollbacks. But realistically, no organization — nor any one developer — is immune to code-breaking bugs. The "sweet spot" of deployment, then, is a toolkit that prioritizes sustainable development tools like CD while also adopting leading, automation-based rollback capabilities. By focusing on preventative measures and leveraging the proper deployment tools, organizations can enhance their efficiency, reduce downtime and ultimately deliver their end users a much more reliable, robust product.

Stephen Atwell is Principal Product Manager at
Share this

Industry News

December 06, 2023

ngrok unveiled its JavaScript and Python SDKs, enabling developers to programmatically serve their applications and manage traffic by embedding ingress with a single line of code.

December 06, 2023

Data Theorem introduced API Attack Path Visualization capabilities for the protection of APIs and the software supply chain.

December 05, 2023

Security Journey announced support for WCAG, SCIM and continued compliance with SOC2 Type 2, which are leading industry standards.

December 05, 2023

Vercel announced a new suite of features for its Developer Experience (DX) Platform, made for enterprise teams with large codebases.

December 04, 2023

Atlassian Corporation has completed the acquisition of Loom, a video messaging platform that helps users communicate through instantly shareable videos.

December 04, 2023

Orca Security announced that the Orca Cloud Security Platform has achieved the Amazon Web Services (AWS) Built-in Competency.

November 30, 2023

Parasoft, a global leader in automated software testing solutions, today announced complete support for MISRA C++ 2023 with the upcoming release of Parasoft C/C++test 2023.2.

November 30, 2023 achieved the Amazon Elastic Kubernetes Service (Amazon EKS) Ready designation from Amazon Web Services (AWS).

November 29, 2023

CircleCI implemented a gen2 GPU resource class, leveraging Amazon Elastic Compute Cloud (Amazon EC2) G5 instances, offering the latest generation of NVIDIA GPUs and new images tailored for artificial intelligence/machine learning (AI/ML) workflows.

November 29, 2023

XM Cyber announced new capabilities that provide complete and continuous visibility into risks and vulnerabilities in Kubernetes environments.

November 29, 2023

PerfectScale has achieved the Amazon Elastic Kubernetes Service (Amazon EKS) Ready designation from Amazon Web Services (AWS).

November 28, 2023

BMC announced two new product innovations, BMC AMI DevX Code Insights and BMC AMI zAdviser Enterprise.

November 28, 2023

Rafay Systems announced the availability of the Rafay Cloud Automation Platform — the evolution of its Kubernetes Operations Platform — to enable platform teams to deliver automation and self-service capabilities to developers, data scientists and other cloud users.

November 28, 2023

Bitrise is integrating with Amazon Web Services (AWS) to provide compliance-conscious companies with greater access to CI/CD capabilities for mobile app development.

November 28, 2023

Armory announced a new unified declarative deployment capability for AWS Lambda.