Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 2
July 14, 2022

Sayandeb Saha

As containers become the default choice for developing and distributing modern applications and Kubernetes (k8s) the de-facto platform for deploying, running, and scaling such applications, enterprises need to scale their Kubernetes environments rapidly to keep up. However, rapidly scaling Kubernetes environments can be challenging and create complexities that may be hard for you to address and difficult to resolve without a clear strategy. Part 2 of this blog specifies a few more common techniques that you can use to navigate the complexities of managing scaled-out Kubernetes environments.

Start with: Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 1

Keeping Up with Kubernetes Updates

Kubernetes is a thriving open-source project delivering rapid innovation with releases three times a year. If using fully managed Kubernetes from public cloud providers, be prepared for Kubernetes service life cycles that are aggressive. Test your applications with newer versions of Kubernetes as they are released to minimize upgrade-related downtime. If possible, avoid in-place upgrades of Kubernetes clusters — create new clusters, clone your applications to the new clusters, divert traffic to the new clusters, and retire the old clusters. Proactively adopt more recent versions of Kubernetes for running your business-critical applications to prevent public cloud providers from upgrading your Kubernetes control plane version after the end of life of a particular version of the Kubernetes control plane.

For self-managed Kubernetes platforms, vendors also release aggressively to keep up with upstream innovation. You will have more control over when to upgrade, but you do not want to fall behind as it becomes difficult to upgrade if you are too far back and vendors discontinue support for the versions you are on.

Most Kubernetes providers document their life cycle. Read, understand, and take the necessary actions to keep up with rapid releases and subsequent end-of-life schedules.

Reduce or Eliminate Application/Cluster Downtime

Like all other applications and environments, Kubernetes applications and clusters can also experience service-impacting disasters or outages, which can be self-inflicted or accidental. To keep up with the rapid upgrades as explained in the previous section and recover from unplanned outages, use commercially licensed or open-source Kubernetes data protection solutions that provide backup, DR, and mobility for Kubernetes applications. While adopting such solutions look for ones can handle scaled out multi-cluster environments providing a single pane of glass for your K8s protection needs.

GitOps for Application Life-Cycle Management

Releasing applications on Kubernetes can be challenging and even more daunting in scaled-out environments. GitOps, which leverages the power of Git, a popular software version control tool, to provide both revision and change control for applications within the Kubernetes platform, is a best practice that you should consider adopting in large Kubernetes environments.

This model stores the system's desired state in a software version control system like Git. Developers make changes to the configuration files representing the desired state instead of using CLI or GUI to directly make changes on the K8s clusters. A delta between the desired state stored in Git and the system's actual state indicates the changeset that needs to be deployed. These changesets can be reviewed and approved (or rejected) through standard Git processes such as pull requests, code reviews, and merges to master. Approved and merged changesets to the main branch are applied to K8s clusters for changing the system's current state to the desired state based on the configuration stored in Git.

You can quickly and easily release applications using this practice and roll back as needed if things don't go according to plan. Using GitOps for change control leverages Kubernetes' core functionality as a reconciliation engine. This process provides an implicit audit trail of actions taken while releasing applications enabling easier troubleshooting and root cause analyses in large K8s environments.

Comprehensive Observability

Rich observability is essential for maintaining large Kubernetes environments so that you can proactively and reactively mitigate issues that can otherwise become a revenue and/or productivity impacting outage. Kubernetes observability is complex as Kubernetes constitutes multiple layers of infrastructure and several distinct, highly distributed services, each producing its own set of monitoring data with no single master source/log.

To maintain large Kubernetes environments, you must implement:

■ Monitoring of K8s infrastructure (cluster, nodes, namespaces, pods, etc.) and application resources (CPU, memory, storage, networking)

■ Log collection and management for all Kubernetes services and infrastructure

■ Alerts and notifications

Monitoring data generated from various sources need to be collected separately, correlated, and sometimes analyzed to provide the full context of each event or change to an admin, who can understand it, and take corrective action(s) as needed to keep your environment humming without disruption.


If you have started dabbling into Kubernetes or have small/medium K8s environments, it's only a matter of time you will be managing a large K8s environment as developers embrace containers and Kubernetes for new apps and refactor existing apps. Adopting a few strategies outlined here can reduce some of your pains that are associated with large K8s estates. Seek solutions that can help with your data management needs for large scale Kubernetes environments making upgrades easier, recover from disasters faster, and backup your precious application data with support for "Namespace-as-a-Service" operating models commonly used in such environments.

Sayandeb Saha is Sr. Director, Product Management, at NetApp
Share this

Industry News

September 21, 2023

Red Hat and Oracle announced the expansion of their alliance to offer customers a greater choice in deploying applications on Oracle Cloud Infrastructure (OCI). As part of the expanded collaboration, Red Hat OpenShift, the industry’s leading hybrid cloud application platform powered by Kubernetes for architecting, building, and deploying cloud-native applications, will be supported and certified to run on OCI.

September 21, 2023

Harness announced the availability of Gitness™, a freely available, fully open source Git platform that brings a new era of collaboration, speed, security, and intelligence to software development.

September 20, 2023

Oracle announced new application development capabilities to enable developers to rapidly build and deploy applications on Oracle Cloud Infrastructure (OCI).

September 20, 2023

Sonar announced zero-configuration, automatic analysis for programming languages C and C++ within SonarCloud.

September 20, 2023

DataStax announced a new JSON API for Astra DB – the database-as-a-service built on the open source Apache Cassandra® – delivering on one of the most highly requested user features, and providing a seamless experience for Javascript developers building AI applications.

September 19, 2023

Oracle announced the availability of Java 21.

September 19, 2023

Mirantis launched Lens AppIQ, available directly in Lens Desktop and as (Software as a Service) SaaS.

September 19, 2023

Buildkite announced the company has entered into a definitive agreement to acquire Packagecloud, a cloud-based software package management platform, in an all stock deal.

September 19, 2023

CrowdStrike has agreed to acquire Bionic, a provider of Application Security Posture Management (ASPM).

September 18, 2023

Perforce Software announces BlazeMeter's Test Data Pro, the latest addition to its continuous testing platform.

September 18, 2023

CloudBees announced a new cloud native DevSecOps platform that places platform engineers and developer experience front and center.

September 18, 2023

Akuity announced a new open source tool, Kargo, to implement change promotions across many application life cycle stages using GitOps principles.

September 14, 2023

CloudBees announced significant performance and scalability breakthroughs for Jenkins® with new updates to its CloudBees Continuous Integration (CI) software.