Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 2
July 14, 2022

Sayandeb Saha

As containers become the default choice for developing and distributing modern applications and Kubernetes (k8s) the de-facto platform for deploying, running, and scaling such applications, enterprises need to scale their Kubernetes environments rapidly to keep up. However, rapidly scaling Kubernetes environments can be challenging and create complexities that may be hard for you to address and difficult to resolve without a clear strategy. Part 2 of this blog specifies a few more common techniques that you can use to navigate the complexities of managing scaled-out Kubernetes environments.

Start with: Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 1

Keeping Up with Kubernetes Updates

Kubernetes is a thriving open-source project delivering rapid innovation with releases three times a year. If using fully managed Kubernetes from public cloud providers, be prepared for Kubernetes service life cycles that are aggressive. Test your applications with newer versions of Kubernetes as they are released to minimize upgrade-related downtime. If possible, avoid in-place upgrades of Kubernetes clusters — create new clusters, clone your applications to the new clusters, divert traffic to the new clusters, and retire the old clusters. Proactively adopt more recent versions of Kubernetes for running your business-critical applications to prevent public cloud providers from upgrading your Kubernetes control plane version after the end of life of a particular version of the Kubernetes control plane.

For self-managed Kubernetes platforms, vendors also release aggressively to keep up with upstream innovation. You will have more control over when to upgrade, but you do not want to fall behind as it becomes difficult to upgrade if you are too far back and vendors discontinue support for the versions you are on.

Most Kubernetes providers document their life cycle. Read, understand, and take the necessary actions to keep up with rapid releases and subsequent end-of-life schedules.

Reduce or Eliminate Application/Cluster Downtime

Like all other applications and environments, Kubernetes applications and clusters can also experience service-impacting disasters or outages, which can be self-inflicted or accidental. To keep up with the rapid upgrades as explained in the previous section and recover from unplanned outages, use commercially licensed or open-source Kubernetes data protection solutions that provide backup, DR, and mobility for Kubernetes applications. While adopting such solutions look for ones can handle scaled out multi-cluster environments providing a single pane of glass for your K8s protection needs.

GitOps for Application Life-Cycle Management

Releasing applications on Kubernetes can be challenging and even more daunting in scaled-out environments. GitOps, which leverages the power of Git, a popular software version control tool, to provide both revision and change control for applications within the Kubernetes platform, is a best practice that you should consider adopting in large Kubernetes environments.

This model stores the system's desired state in a software version control system like Git. Developers make changes to the configuration files representing the desired state instead of using CLI or GUI to directly make changes on the K8s clusters. A delta between the desired state stored in Git and the system's actual state indicates the changeset that needs to be deployed. These changesets can be reviewed and approved (or rejected) through standard Git processes such as pull requests, code reviews, and merges to master. Approved and merged changesets to the main branch are applied to K8s clusters for changing the system's current state to the desired state based on the configuration stored in Git.

You can quickly and easily release applications using this practice and roll back as needed if things don't go according to plan. Using GitOps for change control leverages Kubernetes' core functionality as a reconciliation engine. This process provides an implicit audit trail of actions taken while releasing applications enabling easier troubleshooting and root cause analyses in large K8s environments.

Comprehensive Observability

Rich observability is essential for maintaining large Kubernetes environments so that you can proactively and reactively mitigate issues that can otherwise become a revenue and/or productivity impacting outage. Kubernetes observability is complex as Kubernetes constitutes multiple layers of infrastructure and several distinct, highly distributed services, each producing its own set of monitoring data with no single master source/log.

To maintain large Kubernetes environments, you must implement:

■ Monitoring of K8s infrastructure (cluster, nodes, namespaces, pods, etc.) and application resources (CPU, memory, storage, networking)

■ Log collection and management for all Kubernetes services and infrastructure

■ Alerts and notifications

Monitoring data generated from various sources need to be collected separately, correlated, and sometimes analyzed to provide the full context of each event or change to an admin, who can understand it, and take corrective action(s) as needed to keep your environment humming without disruption.


If you have started dabbling into Kubernetes or have small/medium K8s environments, it's only a matter of time you will be managing a large K8s environment as developers embrace containers and Kubernetes for new apps and refactor existing apps. Adopting a few strategies outlined here can reduce some of your pains that are associated with large K8s estates. Seek solutions that can help with your data management needs for large scale Kubernetes environments making upgrades easier, recover from disasters faster, and backup your precious application data with support for "Namespace-as-a-Service" operating models commonly used in such environments.

Sayandeb Saha is Sr. Director, Product Management, at NetApp
Share this

Industry News

March 21, 2023

OpenText launched the latest version of ValueEdge -- an innovative modular, cloud-based DevOps and value stream management (VSM) platform.

March 21, 2023

Oracle announced the availability of Java 20, the latest version of the programming language and development platform.

March 21, 2023

Rafay Systems introduced Environment Manager, a solution that empowers enterprise platform teams to improve the developer experience by delivering self-service capabilities for provisioning full-stack environments.

March 20, 2023

To meet the growing demand for Oracle Container Engine for Kubernetes (OKE) with global organizations, Oracle Cloud Infrastructure (OCI) is introducing new capabilities that can boost the reliability and efficiency of large-scale Kubernetes environments while simplifying operations and reducing costs.

March 20, 2023

Perforce Software joined the Amazon Web Services (AWS) Independent Software Vendor (ISV) Accelerate Program and listed its free Enhanced Studio Pack (ESP) in AWS Marketplace.

March 20, 2023

Aembit, an identity platform that lets DevOps and Security teams discover, manage, enforce, and audit access between federated workloads, announced its official launch alongside $16.6M in seed financing from cybersecurity specialist investors Ballistic Ventures and Ten Eleven Ventures.

March 16, 2023

Hyland released Alfresco Content Services 7.0 – a cloud-native content services platform, optimized for content model flexibility and performance at scale.

March 16, 2023

CAST AI has announced the closing of a $20M investment round.

March 15, 2023

Check Point® Software Technologies introduced Infinity Global Services, an all-encompassing security solution that will empower organizations of all sizes to fortify their systems, from cloud to network to endpoint.

March 15, 2023

OpsCruise's Kubernetes and Cloud Service observability platform is certified to run on the Red Hat OpenShift Kubernetes platform.

March 14, 2023 released an update to the platform, delivering productivity for data teams.

March 14, 2023

CoreStack and Zensar announced a strategic global partnership. CoreStack will provide its AI-powered NextGen cloud governance and FinOps capabilities, complementing Zensar’s composable cloud operations offering.

March 14, 2023

Delinea introduced the Delinea Platform, a cloud-native foundation for Delinea's PAM solutions that empowers end-to-end visibility, dynamic privilege controls, and adaptive security.

March 13, 2023

Sysdig announced a new foundation that will serve as the long-term custodian of the Wireshark open source project.

March 13, 2023

Talend announced the latest update to Talend Data Fabric, its end-to-end platform for data discovery, transformation, governance, and sharing.