Battle Plan 2018: Illuminate Blind Spots and Unknown Unknowns
December 27, 2017

Josh Gray
Cedexis

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.

Bonus points if you know who came up with that tongue twister. He was talking about terrorists, but we're here to discuss a different sort of war — the Battle for Bandwidth. These days, application and content delivery requires special tactics, an integrated strategy, and well-sourced intelligence. And the unknown unknowns are the true enemy because they inevitably lead to outages, slowdowns, and mutinous customers.

In early November, a major outage caused by a minor configuration error (a route leak, to be exact) at global backbone provider Level 3 created widespread connection issues on both U.S. coasts. Comcast, Verizon, Cox, and Vonage customers were particularly affected.

One small error can have mighty ripple effects, and the cause isn't always apparent to network admins and enterprise customers. The time it took to return the Down Detector maps from angry red to mellow yellow could have been shortened by looking at Real User Measurements (crowdsourced telemetry), realizing it wasn't a single site or ISP, and following a logic tree to find the culprit.

With Global Server Load Balancing, your delivery network is smart enough to see the barricade around the corner and switch routes on the fly — saving the day (and making the other guys look a bit dazed and confused).

Blind spots can be hiding more than outages. Your crack team of DevOps commandos can't run successful release missions if they can't check what's really going on in the field. You don't want them dashing around in the dark without a robust tactical plan based on all the parameters you can assess — when you turn unknown unknowns into known knowns from your various data streams, you can put them to work.

Continuous deployment isn't for the faint of heart — you better have your Kevlar and your night vision goggles. Companies like Salesforce are releasing updates dozens of times a day; but even a handful a week requires a careful strategy. You can use RUM to test an update by initially limiting roll-out to one data center. Check for 40x/50x errors. If you're seeing problems, you can check both user experience with your app (non-updated versions) in other places, and user experience at the same data center where you are testing the updated version, to deduce the source of trouble.

One of the biggest unknown unknowns in traffic management is what's going on in places you haven't served recently. If a story about Boise causes traffic to spike there, and that's not normally an audience hotspot for your service, chances are you won't have any measurements of your own to go on. Community intelligence turns these dark corners of your empire into known knowns through automated crowdsourcing of quality of experience metrics. When combined with real-time server health checks and third-party data streams, you have a powerful ability to make efficient, economical routing decisions, even for destinations you don't have any history with.

The more insight and intelligence can be used to accelerate the acquisition of known knowns, the better it is for your business and your bottom line. In the New Year, we should be less accepting of blind spots. They're expensive — they cost us time, money, and customers. Nobody has enough human problem solvers around to keep putting out fires and rigging up one-off workarounds. Our best talent should be working on the next release, the next big idea, or the next major dilemma (Net Neutrality game changers, anyone?) — not floundering around trying to guess what's holding up traffic. You can't control what you can't see, and on the hybrid IT battlefield, control keeps you on top of the hill. We're pretty sure Donald Rumsfeld would agree.

Josh Gray is Chief Architect at Cedexis
Share this

Industry News

June 27, 2022

Delinea announced the latest release of DevOps Secrets Vault.

June 27, 2022

Jit announced a $38.5 million seed funding round and launched a free beta version which automates product security.

June 27, 2022

Platform.sh raised $140 million in Series D funding.

June 23, 2022

Akana by Perforce now offers BlazeMeter to customers, previously a solution with Broadcom Layer7.

June 23, 2022

Coder announced the release of a new open source project that gives developers and data scientists a consistent, secure, yet flexible way to create cloud workspaces in minutes.

June 23, 2022

GitGuardian is announcing a series of new features to address developer experience in securing the software development lifecycle.

June 22, 2022

OctoML released a major platform expansion to accelerate the development of AI-powered applications by eliminating bottlenecks in machine learning deployment.

June 22, 2022

Snow Software announced new functionality and integrations for Snow Atlas, a purpose-built platform that provides a framework to accelerate data-driven technology decision-making.

June 22, 2022

Traefik Labs launched Traefik Hub, a new cloud service that eliminates the complexity of management and automation of Kubernetes and Docker networking at scale.

June 21, 2022

The Linux Foundation, the nonprofit organization enabling mass innovation through open source, announced the new Open Programmable Infrastructure (OPI) Project.

June 21, 2022

Docker announced the acquisition of Atomist, a company founded to improve developer productivity and keep cloud native applications safe.

June 21, 2022

SmartBear released BitBar, an all-in-one web and native mobile app testing solution.

June 16, 2022

Armory announced general availability of Armory Continuous Deployment-as-a-Service.

June 16, 2022

Infragistics announced the launch of App Builder On-Prem.

June 16, 2022

LambdaTest launched Test-at-Scale (TAS), a test intelligence and observability platform, to help development teams with shift-left testing.