What Netflix Teaches Us About DevOps Culture
August 30, 2021

Nate Nelson
Headspring Software Consulting

Netflix didn't set out to create a "DevOps culture." They didn't follow a set of predefined rules. They didn't have endless strategy meetings. They didn't have to hire DevOps consultants.

Instead, they developed a DevOps culture organically. And it all started with the worst outage in their history.


(image by Thibault Penin)

Their then Head of IT Operations, Mike Osier, recalled how the 2008 disaster began: "On Monday, 8/11, our monitors flagged a database corruption event in our shipping system. Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down."

Netflix was, to this point, still largely reliant on physical DVD sales. A takedown of all 55 of their distribution centers meant that no small portion of their entire business was frozen.

It was going to be a long night.

The following day, the company still couldn't ship DVDs. On Wednesday, it was the same. And Thursday.


(image via CNET)

It was the event that changed everything. "That is when we realized," wrote their VP of Cloud Computing, "that we had to move [. . .] in the cloud."

Except it wasn't a sudden realization, or a snap decision. In the "Towering Inferno" — a rather ordinary top floor conference room in their Los Gatos, CA headquarters — engineers and executives got together and hashed out a plan. It took several meetings, and long hours of discussion. The goal was to never let something like what they just experienced happen again.

But how? This was 2008, remember, and the cloud was still very new. Most people didn't even know it existed. Amazon — Netflix's chosen provider — had only been offering their AWS service for a couple of years. There was no rulebook on how to migrate; how to implement.

And it wasn't just a technical challenge, either: it was philosophical, cultural. Netflix stood at a crossroads of what kind of company they were going to be in the future.

Down one path, they'd continue, broadly speaking, as they were before. They could do a simple "lift-and-shift": pick up the Netflix app, dump it onto the cloud, and fix whatever was necessary to make it fit. Past that, everything would remain as usual. Simple. Easy.

They chose the other path.

"We thought: ‘Let's rethink this completely, go back to ‘first principles'," recalled Neil Hunt, Chief Product Officer at the time. "First principles" meant fundamentally rebuilding their application — how it worked, and how they worked on it — from the ground up. In so doing, they stumbled upon what we as a DevOps consulting company frequently champion as a "DevOps culture."

DevOps can take many forms, but, in any case, it requires:

■ Bringing development and IT operations teams together (obviously)

■ Systematizing the software release cycle: e.g. who does what, in what order, how, etc…

■ Automation (where possible)

The goals of DevOps are to:

■ Save time and resources

■ Release software faster

■ Maintain a high quality standard

■ Achieve the goals of the business


(The "Towering Inferno"; image via Business Insider)

DevOps culture is much broader than that, though. It's about building an environment that supports collaboration — getting everyone on board with one, shared mission. It's about being Agile. Sometimes, like in the case of Netflix, it involves making up your own rules, or breaking existing ones.

The first step in their transformation was to move from a monolith to a microservices architecture. With microservices, developers could operate independently. And, with CI/CD pipelines, they could work continuously to apply changes to the app without any need for wholesale updates.

Still, CI/CD isn't enough on its own. Proper DevOps requires strict, systematic quality control in order to ensure that every new release works and, importantly, works within the whole. So, as their CPO, told Increment Magazine, they didn't just build their new app on the cloud, or build it on-prem and then move it to the cloud:

"We would run our existing infrastructure, and side by side run our AWS infrastructure, and migrate one piece at a time, from one system to another."

Two apps, side by side, being constantly and rigorously tested in order to ensure quality.

And yet it wasn't enough. After all, you can test software all you'd like, but there's no way to cover against every potential failure. The new cloud app was resilient to the kind of error that embarrassed the company in 2008, but what about other unforeseen errors that could cause equivalent or even worse damage? This was the fear that spawned the "Simian Army."

The Simian Army embodies many of the core principles of DevOps: automation, quality assurance, thinking of what the business needs and then working backwards. And yet, it's something few other companies would ever consider — because it's essentially a squad of software designed to break things. The first recruit in this army was Chaos Monkey, whose name, according to Netflix:

"...comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables -- all the while we continue serving our customers without interruption."

We're talking about a set of tools designed to run 24/7, in all environments, causing chaos by randomly triggering errors and shutdowns. Chaos Monkey randomly disables common production instances. Chaos Gorilla simulates what would happen if an entire region of the AWS cloud went completely offline. And there are half a dozen more. The goal, ultimately, is to prepare for every possibility by actively causing it.

"So next time an instance fails at 3 am on a Sunday, we won't even notice."

This wasn't just a hypothetical. On April 21st, 2011, one of Amazon's premier data centers — US-East-1 — cut out, bringing a number of popular websites down with it. On the 25th of September, 2014, a full 10% of Amazon's web servers worldwide shut down for a maintenance update. In neither case did anybody streaming Breaking Bad, or Orange is the New Black, notice a difference, because, as Netflix pointed out, the "systems are designed explicitly for these sorts of failures."

Not every company can pull off what Netflix did in terms of DevOps. Some have failed, "because they simply don't have the engineering muscle that Netflix does. Even though they may want to deliver faster and more efficiently and are even OK with taking on a little more risk, they can't. They need the team to make it happen." Or the help of a DevOps advisory services company.

Either way, the lesson is the same: DevOps isn't about following strict rules. It's about creating a culture. Netflix created their own rules, before there were rules, and then they broke some old ones. But they created a culture where everyone had a seat in the conference room--where developers could build well, test often, and edit their work, and operations had the backing they needed to ensure top quality.

The next time you stream your favorite show in full HD, uninterrupted, thank DevOps.

Nate Nelson is a freelance writer for Headspring Software Consulting
Share this

Industry News

May 12, 2022

Red Hat introduced Red Hat Enterprise Linux 9, the Linux operating system designed to drive more consistent innovation across the open hybrid cloud, from bare metal servers to cloud providers and the farthest edge of enterprise networks.

May 12, 2022

Couchbase announced version 7.1 of Couchbase Server.

May 12, 2022

Copado added Copado Robotic Testing to Copado Essentials.

May 11, 2022

Red Hat announced new advancements within its Red Hat Cloud Services portfolio, delivering a fully-managed and streamlined user experience as organizations build, deploy, manage and scale cloud-native applications across hybrid environments.

May 11, 2022

JFrog introduced a new Docker Desktop Extension for JFrog Xray that allows organizations to automatically scan Docker Containers for vulnerabilities and violations early in the development process.

May 11, 2022

Progress announced a series of updates in Progress Telerik and Progress Kendo UI.

May 11, 2022

Vultr announces that Vultr Kubernetes Engine (VKE) is generally available.

May 10, 2022

Docker announced new features and partnerships to increase developer productivity. Specifically, the company announced Docker Extensions which allow developers to discover and add complementary development tools to Docker Desktop.

May 10, 2022

Red Hat announced the general availability of Red Hat Ansible Automation Platform on Microsoft Azure, pairing hybrid cloud automation with the convenience and support of a managed offering.

May 10, 2022

The Fedora Project, a community-driven open source collaboration sponsored by Red Hat, announced the general availability of Fedora Linux 36, the latest version of the fully open source Fedora operating system.

May 10, 2022

Progress announced the release of Progress Chef Cloud Security, extending DevSecOps with compliance support for native cloud assets and enabling end-to-end management of all on premise, cloud and native cloud resources.

This new offering is complemented with new capabilities across the Chef portfolio targeting DevOps success in the most demanding and complex enterprise deployments.

May 10, 2022

Platform9 announced new platform capabilities in Platform9 5.5 that make it easier for cloud-native development and operations teams to build, scale, and operate apps and Kubernetes clusters in the cloud, on-premises, and at the edge.

May 09, 2022

Red Hat and Accenture have expanded their nearly 12 year strategic partnership to further power open hybrid cloud innovation for enterprises worldwide.

May 09, 2022

Opsera has partnered with Mindtree.

May 09, 2022

Mendix announced that Mendix Workflow for process automation is now generally available.