Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 2
July 14, 2022

Sayandeb Saha

As containers become the default choice for developing and distributing modern applications and Kubernetes (k8s) the de-facto platform for deploying, running, and scaling such applications, enterprises need to scale their Kubernetes environments rapidly to keep up. However, rapidly scaling Kubernetes environments can be challenging and create complexities that may be hard for you to address and difficult to resolve without a clear strategy. Part 2 of this blog specifies a few more common techniques that you can use to navigate the complexities of managing scaled-out Kubernetes environments.

Start with: Navigating the Complexities of Operating Large-Scale Kubernetes Environments - 1

Keeping Up with Kubernetes Updates

Kubernetes is a thriving open-source project delivering rapid innovation with releases three times a year. If using fully managed Kubernetes from public cloud providers, be prepared for Kubernetes service life cycles that are aggressive. Test your applications with newer versions of Kubernetes as they are released to minimize upgrade-related downtime. If possible, avoid in-place upgrades of Kubernetes clusters — create new clusters, clone your applications to the new clusters, divert traffic to the new clusters, and retire the old clusters. Proactively adopt more recent versions of Kubernetes for running your business-critical applications to prevent public cloud providers from upgrading your Kubernetes control plane version after the end of life of a particular version of the Kubernetes control plane.

For self-managed Kubernetes platforms, vendors also release aggressively to keep up with upstream innovation. You will have more control over when to upgrade, but you do not want to fall behind as it becomes difficult to upgrade if you are too far back and vendors discontinue support for the versions you are on.

Most Kubernetes providers document their life cycle. Read, understand, and take the necessary actions to keep up with rapid releases and subsequent end-of-life schedules.

Reduce or Eliminate Application/Cluster Downtime

Like all other applications and environments, Kubernetes applications and clusters can also experience service-impacting disasters or outages, which can be self-inflicted or accidental. To keep up with the rapid upgrades as explained in the previous section and recover from unplanned outages, use commercially licensed or open-source Kubernetes data protection solutions that provide backup, DR, and mobility for Kubernetes applications. While adopting such solutions look for ones can handle scaled out multi-cluster environments providing a single pane of glass for your K8s protection needs.

GitOps for Application Life-Cycle Management

Releasing applications on Kubernetes can be challenging and even more daunting in scaled-out environments. GitOps, which leverages the power of Git, a popular software version control tool, to provide both revision and change control for applications within the Kubernetes platform, is a best practice that you should consider adopting in large Kubernetes environments.

This model stores the system's desired state in a software version control system like Git. Developers make changes to the configuration files representing the desired state instead of using CLI or GUI to directly make changes on the K8s clusters. A delta between the desired state stored in Git and the system's actual state indicates the changeset that needs to be deployed. These changesets can be reviewed and approved (or rejected) through standard Git processes such as pull requests, code reviews, and merges to master. Approved and merged changesets to the main branch are applied to K8s clusters for changing the system's current state to the desired state based on the configuration stored in Git.

You can quickly and easily release applications using this practice and roll back as needed if things don't go according to plan. Using GitOps for change control leverages Kubernetes' core functionality as a reconciliation engine. This process provides an implicit audit trail of actions taken while releasing applications enabling easier troubleshooting and root cause analyses in large K8s environments.

Comprehensive Observability

Rich observability is essential for maintaining large Kubernetes environments so that you can proactively and reactively mitigate issues that can otherwise become a revenue and/or productivity impacting outage. Kubernetes observability is complex as Kubernetes constitutes multiple layers of infrastructure and several distinct, highly distributed services, each producing its own set of monitoring data with no single master source/log.

To maintain large Kubernetes environments, you must implement:

■ Monitoring of K8s infrastructure (cluster, nodes, namespaces, pods, etc.) and application resources (CPU, memory, storage, networking)

■ Log collection and management for all Kubernetes services and infrastructure

■ Alerts and notifications

Monitoring data generated from various sources need to be collected separately, correlated, and sometimes analyzed to provide the full context of each event or change to an admin, who can understand it, and take corrective action(s) as needed to keep your environment humming without disruption.


If you have started dabbling into Kubernetes or have small/medium K8s environments, it's only a matter of time you will be managing a large K8s environment as developers embrace containers and Kubernetes for new apps and refactor existing apps. Adopting a few strategies outlined here can reduce some of your pains that are associated with large K8s estates. Seek solutions that can help with your data management needs for large scale Kubernetes environments making upgrades easier, recover from disasters faster, and backup your precious application data with support for "Namespace-as-a-Service" operating models commonly used in such environments.

Sayandeb Saha is Sr. Director, Product Management, at NetApp
Share this

Industry News

August 16, 2022

Canonical welcomes the .NET development platform, one of Microsoft’s earliest contributions to open source projects, as a native experience on Ubuntu hosts and container images, starting in Ubuntu 22.04 LTS.

August 16, 2022

Veracode announced the launch of the Veracode Velocity Partner Program.

August 16, 2022

Render announced a new monorepository feature that enables its customers to keep all of their code in one super repository instead of managing multiple smaller repositories.

August 15, 2022

Gadget announced Connections, a major new feature that gives app developers access to building blocks that enable them to build and scale ecommerce apps in a fraction of the time, at a fraction of the cost.

August 15, 2022

Opsera is on the Salesforce AppExchange to help enterprise customers shorten software delivery cycles, improve pipeline quality and security, lower operations costs and better align software delivery to business outcomes.

August 15, 2022

Virtusa Corporation earned the DevOps with GitHub on Microsoft Azure advanced specialization, a validation of a services partner's deep knowledge, extensive experience and proven success in implementing secure software development practices applying DevOps principles and using Azure and GitHub solutions.

August 15, 2022

Companies looking to reduce their cloud costs with automated optimization can now easily procure CAST AI via Google Cloud Marketplace using their existing committed spend.

August 11, 2022

Granulate, an Intel Company, announced the upcoming launch of its latest free cost-reduction solution, gMaestro, a continuous workload and pod rightsizing tool for Kubernetes cost optimization.

August 11, 2022

Rezilion announced the availability of MI-X, a newly created open-source tool developed by Rezilion's vulnerability research team.

August 11, 2022

Contrast Security announced its enhanced application programming interface (API) security capabilities within the Contrast Secure Code Platform.

August 10, 2022

Mirantis made it even easier to integrate Mirantis Container Cloud into developer workflows and provide developers and operators with easy access and visibility into the Kubernetes clusters with the Mirantis Container Cloud Lens Extension announced today.

August 10, 2022

ArmorCode announced an integration with Traceable AI which will bring its data into the ArmorCode platform and improve Application Security Posture from code to cloud.

August 10, 2022

Quali unveiled enhanced features for its Torque platform to unify infrastructure orchestration and governance.

August 09, 2022

Veracode announced the enhancement of its Continuous Software Security Platform with substantial improvements to its integrated developer experience.

August 09, 2022

Normalyze announced General Availability for its Freemium offering, a self-serve, free platform that democratizes data discovery and classification in all three public clouds, Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).