Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 2
July 14, 2022

Sayandeb Saha

As containers become the default choice for developing and distributing modern applications and Kubernetes (k8s) the de-facto platform for deploying, running, and scaling such applications, enterprises need to scale their Kubernetes environments rapidly to keep up. However, rapidly scaling Kubernetes environments can be challenging and create complexities that may be hard for you to address and difficult to resolve without a clear strategy. Part 2 of this blog specifies a few more common techniques that you can use to navigate the complexities of managing scaled-out Kubernetes environments.

Start with: Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 1

Keeping Up with Kubernetes Updates

Kubernetes is a thriving open-source project delivering rapid innovation with releases three times a year. If using fully managed Kubernetes from public cloud providers, be prepared for Kubernetes service life cycles that are aggressive. Test your applications with newer versions of Kubernetes as they are released to minimize upgrade-related downtime. If possible, avoid in-place upgrades of Kubernetes clusters — create new clusters, clone your applications to the new clusters, divert traffic to the new clusters, and retire the old clusters. Proactively adopt more recent versions of Kubernetes for running your business-critical applications to prevent public cloud providers from upgrading your Kubernetes control plane version after the end of life of a particular version of the Kubernetes control plane.

For self-managed Kubernetes platforms, vendors also release aggressively to keep up with upstream innovation. You will have more control over when to upgrade, but you do not want to fall behind as it becomes difficult to upgrade if you are too far back and vendors discontinue support for the versions you are on.

Most Kubernetes providers document their life cycle. Read, understand, and take the necessary actions to keep up with rapid releases and subsequent end-of-life schedules.

Reduce or Eliminate Application/Cluster Downtime

Like all other applications and environments, Kubernetes applications and clusters can also experience service-impacting disasters or outages, which can be self-inflicted or accidental. To keep up with the rapid upgrades as explained in the previous section and recover from unplanned outages, use commercially licensed or open-source Kubernetes data protection solutions that provide backup, DR, and mobility for Kubernetes applications. While adopting such solutions look for ones can handle scaled out multi-cluster environments providing a single pane of glass for your K8s protection needs.

GitOps for Application Life-Cycle Management

Releasing applications on Kubernetes can be challenging and even more daunting in scaled-out environments. GitOps, which leverages the power of Git, a popular software version control tool, to provide both revision and change control for applications within the Kubernetes platform, is a best practice that you should consider adopting in large Kubernetes environments.

This model stores the system's desired state in a software version control system like Git. Developers make changes to the configuration files representing the desired state instead of using CLI or GUI to directly make changes on the K8s clusters. A delta between the desired state stored in Git and the system's actual state indicates the changeset that needs to be deployed. These changesets can be reviewed and approved (or rejected) through standard Git processes such as pull requests, code reviews, and merges to master. Approved and merged changesets to the main branch are applied to K8s clusters for changing the system's current state to the desired state based on the configuration stored in Git.

You can quickly and easily release applications using this practice and roll back as needed if things don't go according to plan. Using GitOps for change control leverages Kubernetes' core functionality as a reconciliation engine. This process provides an implicit audit trail of actions taken while releasing applications enabling easier troubleshooting and root cause analyses in large K8s environments.

Comprehensive Observability

Rich observability is essential for maintaining large Kubernetes environments so that you can proactively and reactively mitigate issues that can otherwise become a revenue and/or productivity impacting outage. Kubernetes observability is complex as Kubernetes constitutes multiple layers of infrastructure and several distinct, highly distributed services, each producing its own set of monitoring data with no single master source/log.

To maintain large Kubernetes environments, you must implement:

■ Monitoring of K8s infrastructure (cluster, nodes, namespaces, pods, etc.) and application resources (CPU, memory, storage, networking)

■ Log collection and management for all Kubernetes services and infrastructure

■ Alerts and notifications

Monitoring data generated from various sources need to be collected separately, correlated, and sometimes analyzed to provide the full context of each event or change to an admin, who can understand it, and take corrective action(s) as needed to keep your environment humming without disruption.


If you have started dabbling into Kubernetes or have small/medium K8s environments, it's only a matter of time you will be managing a large K8s environment as developers embrace containers and Kubernetes for new apps and refactor existing apps. Adopting a few strategies outlined here can reduce some of your pains that are associated with large K8s estates. Seek solutions that can help with your data management needs for large scale Kubernetes environments making upgrades easier, recover from disasters faster, and backup your precious application data with support for "Namespace-as-a-Service" operating models commonly used in such environments.

Sayandeb Saha is Sr. Director, Product Management, at NetApp
Share this

Industry News

July 18, 2024

Mission Cloud announced the launch of Mission Cloud Engagements - DevOps, a platform designed to transform how businesses manage and execute their AWS DevOps projects.

July 18, 2024

Accelario announces the release of its free TDM solution, including database virtualization and data anonymization.

July 18, 2024

RAVEL (formerly StratusCore) introduced RAVEL Orchestrate’s new Bare Metal Build Station functionality, which empowers IT and DevOps teams in SMBs or enterprises to intelligently prepare and deploy customized images to any physical machine connected to a network.

July 17, 2024

OpenText™ announced its solution to speed the triage and remediation of vulnerabilities throughout the stages of code development, OpenText Fortify Aviator, an AI-powered code security solution, saves developers significant time by enabling faster and easier auditing and remediation of static application security testing (SAST) vulnerabilities—all within a single solution​.

July 17, 2024

Tricentis announced the acquisition of SeaLights, a SaaS-based, software quality intelligence platform.

July 17, 2024

CAST is now available as software as a service (SaaS).

July 16, 2024

OpenText announced its latest product innovations with Cloud Editions (CE) 24.3.

July 16, 2024

Red Hat introduced new capabilities and enhancements for Red Hat OpenShift, the hybrid cloud application platform powered by Kubernetes, as well as the general availability of Red Hat Advanced Cluster Security Cloud Service.

July 16, 2024

DevEx Connect launched as a community-driven independent research, analyst and events organization focusing on everything under the DevEx umbrella, including DevOps, SRE and Platform Engineering.

July 15, 2024

Elastic announced support for Amazon Bedrock-hosted models in Elasticsearch Open Inference API and Playground.

July 11, 2024

Progress announced new and powerful enhancements in the latest release of Progress® LoadMaster® 360, its cloud-based unified application delivery platform. These enhancements help organizations protect their web applications against increasingly sophisticated cyberattacks and provide customers with an optimal application experience.

July 11, 2024

Virtusa announced a strategic partnership with Quality Clouds, a provider of SaaS governance solutions for Salesforce and ServiceNow platforms.

July 11, 2024

Zesty launched its newest offering, Commitment Manager for Amazon RDS (Relational Database Service).

July 10, 2024

MacStadium unveiled Orka Desktop, a free, local macOS virtualization tool.