What Netflix Teaches Us About DevOps Culture
August 30, 2021

Nate Nelson
Headspring Software Consulting

Netflix didn't set out to create a "DevOps culture." They didn't follow a set of predefined rules. They didn't have endless strategy meetings. They didn't have to hire DevOps consultants.

Instead, they developed a DevOps culture organically. And it all started with the worst outage in their history.


(image by Thibault Penin)

Their then Head of IT Operations, Mike Osier, recalled how the 2008 disaster began: "On Monday, 8/11, our monitors flagged a database corruption event in our shipping system. Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down."

Netflix was, to this point, still largely reliant on physical DVD sales. A takedown of all 55 of their distribution centers meant that no small portion of their entire business was frozen.

It was going to be a long night.

The following day, the company still couldn't ship DVDs. On Wednesday, it was the same. And Thursday.


(image via CNET)

It was the event that changed everything. "That is when we realized," wrote their VP of Cloud Computing, "that we had to move [. . .] in the cloud."

Except it wasn't a sudden realization, or a snap decision. In the "Towering Inferno" — a rather ordinary top floor conference room in their Los Gatos, CA headquarters — engineers and executives got together and hashed out a plan. It took several meetings, and long hours of discussion. The goal was to never let something like what they just experienced happen again.

But how? This was 2008, remember, and the cloud was still very new. Most people didn't even know it existed. Amazon — Netflix's chosen provider — had only been offering their AWS service for a couple of years. There was no rulebook on how to migrate; how to implement.

And it wasn't just a technical challenge, either: it was philosophical, cultural. Netflix stood at a crossroads of what kind of company they were going to be in the future.

Down one path, they'd continue, broadly speaking, as they were before. They could do a simple "lift-and-shift": pick up the Netflix app, dump it onto the cloud, and fix whatever was necessary to make it fit. Past that, everything would remain as usual. Simple. Easy.

They chose the other path.

"We thought: ‘Let's rethink this completely, go back to ‘first principles'," recalled Neil Hunt, Chief Product Officer at the time. "First principles" meant fundamentally rebuilding their application — how it worked, and how they worked on it — from the ground up. In so doing, they stumbled upon what we as a DevOps consulting company frequently champion as a "DevOps culture."

DevOps can take many forms, but, in any case, it requires:

■ Bringing development and IT operations teams together (obviously)

■ Systematizing the software release cycle: e.g. who does what, in what order, how, etc…

■ Automation (where possible)

The goals of DevOps are to:

■ Save time and resources

■ Release software faster

■ Maintain a high quality standard

■ Achieve the goals of the business


(The "Towering Inferno"; image via Business Insider)

DevOps culture is much broader than that, though. It's about building an environment that supports collaboration — getting everyone on board with one, shared mission. It's about being Agile. Sometimes, like in the case of Netflix, it involves making up your own rules, or breaking existing ones.

The first step in their transformation was to move from a monolith to a microservices architecture. With microservices, developers could operate independently. And, with CI/CD pipelines, they could work continuously to apply changes to the app without any need for wholesale updates.

Still, CI/CD isn't enough on its own. Proper DevOps requires strict, systematic quality control in order to ensure that every new release works and, importantly, works within the whole. So, as their CPO, told Increment Magazine, they didn't just build their new app on the cloud, or build it on-prem and then move it to the cloud:

"We would run our existing infrastructure, and side by side run our AWS infrastructure, and migrate one piece at a time, from one system to another."

Two apps, side by side, being constantly and rigorously tested in order to ensure quality.

And yet it wasn't enough. After all, you can test software all you'd like, but there's no way to cover against every potential failure. The new cloud app was resilient to the kind of error that embarrassed the company in 2008, but what about other unforeseen errors that could cause equivalent or even worse damage? This was the fear that spawned the "Simian Army."

The Simian Army embodies many of the core principles of DevOps: automation, quality assurance, thinking of what the business needs and then working backwards. And yet, it's something few other companies would ever consider — because it's essentially a squad of software designed to break things. The first recruit in this army was Chaos Monkey, whose name, according to Netflix:

"...comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables -- all the while we continue serving our customers without interruption."

We're talking about a set of tools designed to run 24/7, in all environments, causing chaos by randomly triggering errors and shutdowns. Chaos Monkey randomly disables common production instances. Chaos Gorilla simulates what would happen if an entire region of the AWS cloud went completely offline. And there are half a dozen more. The goal, ultimately, is to prepare for every possibility by actively causing it.

"So next time an instance fails at 3 am on a Sunday, we won't even notice."

This wasn't just a hypothetical. On April 21st, 2011, one of Amazon's premier data centers — US-East-1 — cut out, bringing a number of popular websites down with it. On the 25th of September, 2014, a full 10% of Amazon's web servers worldwide shut down for a maintenance update. In neither case did anybody streaming Breaking Bad, or Orange is the New Black, notice a difference, because, as Netflix pointed out, the "systems are designed explicitly for these sorts of failures."

Not every company can pull off what Netflix did in terms of DevOps. Some have failed, "because they simply don't have the engineering muscle that Netflix does. Even though they may want to deliver faster and more efficiently and are even OK with taking on a little more risk, they can't. They need the team to make it happen." Or the help of a DevOps advisory services company.

Either way, the lesson is the same: DevOps isn't about following strict rules. It's about creating a culture. Netflix created their own rules, before there were rules, and then they broke some old ones. But they created a culture where everyone had a seat in the conference room--where developers could build well, test often, and edit their work, and operations had the backing they needed to ensure top quality.

The next time you stream your favorite show in full HD, uninterrupted, thank DevOps.

Nate Nelson is a freelance writer for Headspring Software Consulting
Share this

Industry News

March 18, 2024

Kubiya.ai announces the launch of its DevOps Digital Agents.

March 18, 2024

Aviatrix® introduced Aviatrix Distributed Cloud Firewall for Kubernetes, a distributed cloud networking and network security solution for containerized enterprise applications and workloads.

March 18, 2024

Stride announces the general availability of Stride Conductor, its new autonomous coding product that transforms the software development landscape.

March 14, 2024

CircleCI unveiled CircleCI releases, which enables developers to automate the release orchestration process directly from the CircleCI UI.

March 13, 2024

Fermyon™ Technologies announces Fermyon Platform for Kubernetes, a WebAssembly platform for Kubernetes.

March 13, 2024

Akuity announced a new offer targeted at Enterprises and businesses where security and compliance are key.

March 13, 2024

New Relic launched new capabilities for New Relic IAST (Interactive Application Security Testing), including proof-of-exploit reporting for application security testing.

March 12, 2024

OutSystems announced AI Agent Builder, a new solution in the OutSystems Developer Cloud platform that makes it easy for IT leaders to incorporate generative AI (GenAI) powered applications into their digital transformation strategy, as well as govern the use of AI to ensure standardization and security.

March 12, 2024

Mirantis announced significant updates to Lens Desktop that makes working with Kubernetes easier by simplifying operations, improving efficiency, and increasing productivity. Lens 2024 Early Access is now available to Lens users.

March 12, 2024

Codezero announced a $3.5 million seed-funding round led by Ballistic Ventures, the venture capital firm dedicated exclusively to funding entrepreneurs and innovations in cybersecurity.

March 11, 2024

Prismatic launched a code-native integration building experience.

March 07, 2024

Check Point® Software Technologies Ltd. announced its Check Point Infinity Platform has been ranked as the #1 Zero Trust Platform in the latest Miercom Zero Trust Platform Assessment.

March 07, 2024

Tricentis announced the launch and availability of SAP Test Automation by Tricentis as an SAP Solution Extension.

March 07, 2024

Netlify announced the general availability of the AI-enabled deploy assist.

March 07, 2024

DataStax announced a new integration with Airbyte that simplifies the process of building production-ready GenAI applications with structured and unstructured data.