What Netflix Teaches Us About DevOps Culture
August 30, 2021

Nate Nelson
Headspring Software Consulting

Netflix didn't set out to create a "DevOps culture." They didn't follow a set of predefined rules. They didn't have endless strategy meetings. They didn't have to hire DevOps consultants.

Instead, they developed a DevOps culture organically. And it all started with the worst outage in their history.


(image by Thibault Penin)

Their then Head of IT Operations, Mike Osier, recalled how the 2008 disaster began: "On Monday, 8/11, our monitors flagged a database corruption event in our shipping system. Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down."

Netflix was, to this point, still largely reliant on physical DVD sales. A takedown of all 55 of their distribution centers meant that no small portion of their entire business was frozen.

It was going to be a long night.

The following day, the company still couldn't ship DVDs. On Wednesday, it was the same. And Thursday.


(image via CNET)

It was the event that changed everything. "That is when we realized," wrote their VP of Cloud Computing, "that we had to move [. . .] in the cloud."

Except it wasn't a sudden realization, or a snap decision. In the "Towering Inferno" — a rather ordinary top floor conference room in their Los Gatos, CA headquarters — engineers and executives got together and hashed out a plan. It took several meetings, and long hours of discussion. The goal was to never let something like what they just experienced happen again.

But how? This was 2008, remember, and the cloud was still very new. Most people didn't even know it existed. Amazon — Netflix's chosen provider — had only been offering their AWS service for a couple of years. There was no rulebook on how to migrate; how to implement.

And it wasn't just a technical challenge, either: it was philosophical, cultural. Netflix stood at a crossroads of what kind of company they were going to be in the future.

Down one path, they'd continue, broadly speaking, as they were before. They could do a simple "lift-and-shift": pick up the Netflix app, dump it onto the cloud, and fix whatever was necessary to make it fit. Past that, everything would remain as usual. Simple. Easy.

They chose the other path.

"We thought: ‘Let's rethink this completely, go back to ‘first principles'," recalled Neil Hunt, Chief Product Officer at the time. "First principles" meant fundamentally rebuilding their application — how it worked, and how they worked on it — from the ground up. In so doing, they stumbled upon what we as a DevOps consulting company frequently champion as a "DevOps culture."

DevOps can take many forms, but, in any case, it requires:

■ Bringing development and IT operations teams together (obviously)

■ Systematizing the software release cycle: e.g. who does what, in what order, how, etc…

■ Automation (where possible)

The goals of DevOps are to:

■ Save time and resources

■ Release software faster

■ Maintain a high quality standard

■ Achieve the goals of the business


(The "Towering Inferno"; image via Business Insider)

DevOps culture is much broader than that, though. It's about building an environment that supports collaboration — getting everyone on board with one, shared mission. It's about being Agile. Sometimes, like in the case of Netflix, it involves making up your own rules, or breaking existing ones.

The first step in their transformation was to move from a monolith to a microservices architecture. With microservices, developers could operate independently. And, with CI/CD pipelines, they could work continuously to apply changes to the app without any need for wholesale updates.

Still, CI/CD isn't enough on its own. Proper DevOps requires strict, systematic quality control in order to ensure that every new release works and, importantly, works within the whole. So, as their CPO, told Increment Magazine, they didn't just build their new app on the cloud, or build it on-prem and then move it to the cloud:

"We would run our existing infrastructure, and side by side run our AWS infrastructure, and migrate one piece at a time, from one system to another."

Two apps, side by side, being constantly and rigorously tested in order to ensure quality.

And yet it wasn't enough. After all, you can test software all you'd like, but there's no way to cover against every potential failure. The new cloud app was resilient to the kind of error that embarrassed the company in 2008, but what about other unforeseen errors that could cause equivalent or even worse damage? This was the fear that spawned the "Simian Army."

The Simian Army embodies many of the core principles of DevOps: automation, quality assurance, thinking of what the business needs and then working backwards. And yet, it's something few other companies would ever consider — because it's essentially a squad of software designed to break things. The first recruit in this army was Chaos Monkey, whose name, according to Netflix:

"...comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables -- all the while we continue serving our customers without interruption."

We're talking about a set of tools designed to run 24/7, in all environments, causing chaos by randomly triggering errors and shutdowns. Chaos Monkey randomly disables common production instances. Chaos Gorilla simulates what would happen if an entire region of the AWS cloud went completely offline. And there are half a dozen more. The goal, ultimately, is to prepare for every possibility by actively causing it.

"So next time an instance fails at 3 am on a Sunday, we won't even notice."

This wasn't just a hypothetical. On April 21st, 2011, one of Amazon's premier data centers — US-East-1 — cut out, bringing a number of popular websites down with it. On the 25th of September, 2014, a full 10% of Amazon's web servers worldwide shut down for a maintenance update. In neither case did anybody streaming Breaking Bad, or Orange is the New Black, notice a difference, because, as Netflix pointed out, the "systems are designed explicitly for these sorts of failures."

Not every company can pull off what Netflix did in terms of DevOps. Some have failed, "because they simply don't have the engineering muscle that Netflix does. Even though they may want to deliver faster and more efficiently and are even OK with taking on a little more risk, they can't. They need the team to make it happen." Or the help of a DevOps advisory services company.

Either way, the lesson is the same: DevOps isn't about following strict rules. It's about creating a culture. Netflix created their own rules, before there were rules, and then they broke some old ones. But they created a culture where everyone had a seat in the conference room--where developers could build well, test often, and edit their work, and operations had the backing they needed to ensure top quality.

The next time you stream your favorite show in full HD, uninterrupted, thank DevOps.

Nate Nelson is a freelance writer for Headspring Software Consulting
Share this

Industry News

December 06, 2021

Ascend.io announced support for Amazon Redshift Serverless powered on Amazon Web Services, Inc. (AWS), a fully managed petabyte-scale cloud data warehouse.

December 06, 2021

Neosec formed a strategic partnership with Kong Inc. to integrate its API security platform with Kong Gateway to provide a complete enterprise-class solution for managing and securing APIs and microservices.

December 02, 2021

Mirantis announced DevOpsCare, powered by Lens, a vendor-agnostic, fully-managed CI/CD (continuous integration/continuous deployment) product for any Kubernetes environment, offering developers higher levels of productivity more quickly.

December 02, 2021

The D2iQ Kubernetes Platform (DKP) is now available in AWS Marketplace, a digital catalog with thousands of software listings from independent software vendors that make it easy to find, test, buy, and deploy software that runs on Amazon Web Services, Inc. (AWS).

December 01, 2021

Bugcrowd announced the availability of Bugcrowd's cybersecurity solutions on the AWS Marketplace, providing customers with easy access, simplified billing, quick deployment, and streamlined license management.

December 01, 2021

Kublr received Microsoft Azure Arc-enabled Kubernetes validation, including for Azure Arc-enabled Kubernetes for Data Services.

December 01, 2021

CloudSphere achieved Amazon Web Services (AWS) Migration and Modernization Competency for discovering, planning, and helping enterprise customers move business services to AWS to reduce cost, increase agility and improve security.

November 30, 2021

JFrog introduced a new container registry and package manager for running JFrog Artifactory with Kubernetes clusters on-premises, in the cloud, or both.

November 30, 2021

Docker announced the availability of Docker Official Images directly from Amazon Web Services (AWS).

November 30, 2021

Weaveworks announced the general availability of Weave GitOps Enterprise, a GitOps platform that automates continuous application delivery and Kubernetes operations at any scale.

November 30, 2021

Amazon Web Services announced AWS Mainframe Modernization, a new service that makes it faster and easier for customers to migrate mainframe and legacy workloads to the cloud, and enjoy the superior agility, elasticity, and cost savings of AWS.

November 29, 2021

Quali announced the newest release of Torque Enterprise, which includes enhanced integration with Terraform, new custom tagging capabilities, and improved cost visibility dashboards, unleashing an entirely new level of self-service access to application environments on demand.

November 29, 2021

Vertical Relevance (VR), a financial services-focused consulting firm, achieved Amazon Web Services (AWS) DevOps Competency status.

November 18, 2021

Loft Labs announced the launch of Loft version 2 with a focus on ease of use that overcomes the major complaint that Kubernetes is complex and hard to set up.

November 18, 2021

Perforce Software announced new functionality to speed remediation of discovered defects in automated scans.