Battle Plan 2018: Illuminate Blind Spots and Unknown Unknowns
December 27, 2017

Josh Gray
Cedexis

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.

Bonus points if you know who came up with that tongue twister. He was talking about terrorists, but we're here to discuss a different sort of war — the Battle for Bandwidth. These days, application and content delivery requires special tactics, an integrated strategy, and well-sourced intelligence. And the unknown unknowns are the true enemy because they inevitably lead to outages, slowdowns, and mutinous customers.

In early November, a major outage caused by a minor configuration error (a route leak, to be exact) at global backbone provider Level 3 created widespread connection issues on both U.S. coasts. Comcast, Verizon, Cox, and Vonage customers were particularly affected.

One small error can have mighty ripple effects, and the cause isn't always apparent to network admins and enterprise customers. The time it took to return the Down Detector maps from angry red to mellow yellow could have been shortened by looking at Real User Measurements (crowdsourced telemetry), realizing it wasn't a single site or ISP, and following a logic tree to find the culprit.

With Global Server Load Balancing, your delivery network is smart enough to see the barricade around the corner and switch routes on the fly — saving the day (and making the other guys look a bit dazed and confused).

Blind spots can be hiding more than outages. Your crack team of DevOps commandos can't run successful release missions if they can't check what's really going on in the field. You don't want them dashing around in the dark without a robust tactical plan based on all the parameters you can assess — when you turn unknown unknowns into known knowns from your various data streams, you can put them to work.

Continuous deployment isn't for the faint of heart — you better have your Kevlar and your night vision goggles. Companies like Salesforce are releasing updates dozens of times a day; but even a handful a week requires a careful strategy. You can use RUM to test an update by initially limiting roll-out to one data center. Check for 40x/50x errors. If you're seeing problems, you can check both user experience with your app (non-updated versions) in other places, and user experience at the same data center where you are testing the updated version, to deduce the source of trouble.

One of the biggest unknown unknowns in traffic management is what's going on in places you haven't served recently. If a story about Boise causes traffic to spike there, and that's not normally an audience hotspot for your service, chances are you won't have any measurements of your own to go on. Community intelligence turns these dark corners of your empire into known knowns through automated crowdsourcing of quality of experience metrics. When combined with real-time server health checks and third-party data streams, you have a powerful ability to make efficient, economical routing decisions, even for destinations you don't have any history with.

The more insight and intelligence can be used to accelerate the acquisition of known knowns, the better it is for your business and your bottom line. In the New Year, we should be less accepting of blind spots. They're expensive — they cost us time, money, and customers. Nobody has enough human problem solvers around to keep putting out fires and rigging up one-off workarounds. Our best talent should be working on the next release, the next big idea, or the next major dilemma (Net Neutrality game changers, anyone?) — not floundering around trying to guess what's holding up traffic. You can't control what you can't see, and on the hybrid IT battlefield, control keeps you on top of the hill. We're pretty sure Donald Rumsfeld would agree.

Josh Gray is Chief Architect at Cedexis
Share this

Industry News

April 23, 2025

Kubernetes 1.33 was released today.

Kubernetes 1.33 Release Information

April 23, 2025

Docker announced a major expansion of its AI initiative with the upcoming Docker MCP Catalog and Docker MCP Toolkit.

April 23, 2025

Perforce Software announced the release of its latest platform update for Puppet Enterprise Advanced, designed to streamline DevSecOps practices and fortify enterprise security postures.

April 23, 2025

Azul announced JVM Inventory, a new feature of Azul Intelligence Cloud designed to address the complexity and risk of migrating off Oracle Java.

April 23, 2025

LaunchDarkly announced the acquisition of Highlight, a powerful, open source, full-stack application monitoring platform known for its error monitoring, logging, distributed tracing and session replay capabilities.

April 22, 2025

O’Reilly announced AI Codecon—a groundbreaking virtual conference series dedicated to exploring the rapidly evolving world of AI-assisted software development.

April 22, 2025

Veracode unveiled new capabilities offering proactive risk mitigation and automated security at enterprise scale.

April 22, 2025

Snyk launched Snyk API & Web, delivering a dynamic application security testing (DAST) solution designed to meet the growing demands of modern and increasingly AI-powered software development.

April 21, 2025

Postman announced new releases designed to help organizations build APIs faster, more securely, and with less friction.

April 21, 2025

SnapLogic announced AgentCreator 3.0, an evolution in agentic AI technology that eliminates the complexity of enterprise AI adoption.

April 17, 2025

GitLab announced the general availability of GitLab Duo with Amazon Q.

April 17, 2025

Perforce Software and Liquibase announced a strategic partnership to enhance secure and compliant database change management for DevOps teams.

April 17, 2025

Spacelift announced the launch of Saturnhead AI — an enterprise-grade AI assistant that slashes DevOps troubleshooting time by transforming complex infrastructure logs into clear, actionable explanations.

April 16, 2025

CodeSecure and FOSSA announced a strategic partnership and native product integration that enables organizations to eliminate security blindspots associated with both third party and open source code.