We Don't Need Another Hero
May 27, 2016

Dave Josephsen
Librato

Someone recently asked me this question: "What's the first thing that comes to your mind when you hear the word 'DevOps'?"

A loaded question, I agree, and of course I lied, and made up something about the "first way." I mean really, if you want a possibly embarrassing answer to a loaded question you really should confront me face to face with it in a public place. If the asker of that question had done so, I would have had to answer honestly that the first thing I think about when I hear the word DevOps is Brent from The Phoenix Project.

If you haven't read it, Brent is probably you. That person who knows how all the stuff actually works, and who everyone depends on to fix things when they go sideways. Brent is a hero, and DevOps abhors hero's. They constrain the productivity of the system overall and create single points of failure. They enable broken systems to limp along instead of allowing them to fail and be replaced. So in the book, Brent is basically a huge organizational problem.

I was deeply hurt by this plot device on my first reading. In fact if I hadn't been trapped on a plane with nothing else to read, I probably wouldn't have finished The Phoenix Project because of it (which totally would have been my loss). DevOps is an emergent property of failure in classical IT management, and it's just hard to wrap your head around the fact that sometimes failure needs to happen before we can pick up the pieces and make something great from them.

Further, the DevOps end-game is hard to visualize, because in many ways it's orthogonal to what preceded it. Many organizations just can't get there until they've failed. Getting there isn't easy either way, so we talk a lot about improvement-kata and "getting there", and why you'd want to be there in the first place. What it looks like day-to-day once you've arrived is something you almost never hear about.

What does that mean? Well to understand that I need to digress a little bit into what we make for a living, which happens to be time-series databases.

One big problem with designing time-series databases is that the pattern of reads and writes is very different. Typically we want to optimize for writes, because we write a lot more than we read, but reads also need to be pretty efficient or no-one will want to use the UI. One way we manage to minimize read latency is to use a rotating row key. That is, we simply preface the name of a given metric with a key that changes every so often, which in turn forces the DB to create a new row to store the data. Breaking the data up like this helps us keep the rows down to a predictable size, and allows us to make queries of large datasets parallel (ie we can read out a bunch of chunks of predictably sized data at the same time rather than one huge chunk).

Predictable is something of a key word there, because really what row-keys buy you is a heap of different kinds of very important predictability. With row keys, we know our rows will always be the size of a measurement times the number of measurements in a row-key interval. From there we can extrapolate all kinds of stuff like how much data we'll need to put on the wire for client reads, and how much storage and processing power our data tier will require and etc. The point is, we can make really important decisions by relying on the predictability that row-keys provide.

But here's the rub. We're a multi-tenant system. In other words, we have users, and users have the power to name things. Uh-oh. If users can put an arbitrarily-sized label on a data point, then we longer have a predictably sized data structure anymore. That's ok, we can isolate that variable by creating a predictably sized UID which maps back to the end-user's human-readable label.

So when you use a row key to buy predictable quantities in the data tier, you also often have to pay some taxes in the form of lookup-tables, or indexes if you prefer. In this example we're going to need two indexes, one to keep track of where in our storage tier a given 6-hour block of measurements is stored (because our rows are named after rotating numbers now, so we need some way to actually find the right data when a user asks for 60-minutes of metric:foo), and another to map user-generated variable length names into either hashes or some other unique identifier (so we know what to even search for in the first index when a user asks for 60 minutes of metric:foo).

Ok, now we know pretty much everything we need to re-examine the problem. Ben, the ops guy in that conversation, is tracking what he considers to be a resource allocation problem. The symptom Ben is reacting to here is high CPU utilization.

Dave Josephsen is the Developer Evangelist for Librato.

Share this

Industry News

September 23, 2021

Shortcut, the collaborative home for modern software teams, announced new Team-to-Workflow functionality.

September 23, 2021

Portainer announced the launch of its Portainer Business Charmed Operator, allowing for seamless integration with Canonical’s Charmed Kubernetes distribution.

September 22, 2021

Parasoft announced the launch and extension of its static application security testing (SAST) and API testing platform with penetration testing, shifting security testing into developer workflows.

September 22, 2021

Synopsys announced a new partnership with The Chertoff Group, a global advisory services firm that applies security expertise, technology insights, and policy intelligence to help clients build resilient organizations, gain competitive advantage, and accelerate growth.

September 21, 2021

ShiftLeft announced that its unified code security platform, ShiftLeft CORE, is now available on the AWS Marketplace.

September 21, 2021

JFrog announced its JFrog Artifactory and JFrog Xray solutions are now accredited in Iron Bank and available via Platform One.

September 21, 2021

GitKraken acquired BigBrassBand and its popular Git Integration for Jira.

September 20, 2021

D2iQ announced version 2.0 of the D2iQ Kubernetes Platform (DKP).

September 20, 2021

Platform9 joined Intel's Open Retail Initiative (ORI) and launched a new software-defined store solution.

September 20, 2021

FireMon has acquired DisruptOps.

September 16, 2021

SnapLogic released the latest version of its new SnapLogic Flows solution.

September 16, 2021

Postman launched a new version of its API Management platform.

September 16, 2021

Progress announced its R3 2021 release of Progress Telerik and Progress Kendo UI.

September 15, 2021

Progress announced the availability of Progress Sitefinity DX 14.0.

September 15, 2021

Launchable, the intelligence platform layer for all software testing, announced their latest addition to the platform, Flaky Tests Insights (beta).