With Kubernetes Adoption at All Time High, Security and Compliance Concerns Still Inhibit Production Deployments
Even as DevSecOps becomes mainstream, container and Kubernetes security incidents are strikingly common
February 11, 2021

Ali Golshan
StackRox

We all wish we could build, deploy, and run our applications without the stress of security concerns. However, the reality is that most of us will run into serious security or compliance issues at one time or another. When that happens, an organization is likely to experience the frustration of delayed application deployments and stifled agility. Containers and Kubernetes promise faster development cycles, quicker bug fixes, and increased velocity, but when security is an afterthought, organizations risk the very gains that containerization promises, particularly agility.

The recent StackRox State of Container and Kubernetes Security Survey found that 44% of survey respondents had to delay an application rollout due to container or Kubernetes security issues. But the right combination of policies, processes and tooling will help mitigate these security and compliance concerns so that teams are more confident pushing container and Kubernetes deployments into production, which should be the culmination of any successful container strategy.

It's interesting to note how DevOps and security teams tend to deploy Kubernetes — roughly 50% of users self-manage their deployments, according to the most recent survey. In this model, these teams must be aware of the security pitfalls and the necessary requirements needed to meet compliance standards. Not that self-managed Kubernetes is less-secure, rather, it simply requires more diligence with updates and patching and more fine tuning of Kubernetes-native security controls. The alternative is tapping into the declarative power of a Kubernetes-native security platform that integrates with the existing tooling of a self-managed environment. Whether provider-managed or self-managed, Kubernetes users must take care to protect their applications and infrastructure from vulnerabilities and misconfigurations, ensure compliance with external and internal policies, and detect and stop runtime threats.

Two-thirds of survey respondents referenced security and compliance as their largest source of concern with their organizations' container strategies. Inadequate investment in security led the list (34%) while others felt their organizations did not take threats seriously enough (15%). There were additional concerns that compliance needs were not being accounted for (17%) as well.

Despite security concerns, organizations are increasingly adopting DevSecOp models that enable security to be tightly integrated with application development and operations starting in the build phase of the software development lifecycle. The survey found that the vast majority of respondents have some form of DevSecOps initiative underway. Only 17% continue to operate DevOps separately from security, proving that DevSecOps is more than a buzzword. This shows promise for the ability of organizations to continually shed concerns about security in order to optimize container and Kubernetes deployments for production.

However, organizational development and IT operations are only part of the equation. There are major container and Kubernetes security risks that exist beyond the realm of container strategies that must be addressed. According to the survey, 90% of respondents claimed to have experienced a security incident in their container and Kubernetes environment in the last 12 months. The most common types of security incidents as reported by respondents include misconfigurations and exposures, Kubernetes vulnerabilities, runtime threats, and failed compliance audits. As such, it's critical that organizations take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments further.

Exposures due to misconfigurations were some of the most worrisome security risks in container and Kubernetes environments, and for good reason as 67% of the survey respondents detected a serious misconfiguration in the last 12 months. These risks fall under the banner of configuration management, which pose a unique security challenge when using Kubernetes to orchestrate containerized applications. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration.

Teams generally know not to expose Kubernetes dashboards to the Internet but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right. Because of this, configuration management should be automated to control the risk of human error due to error-prone manual processes. The survey indicated that most misconfiguration-related security incidents in Kubernetes are caused by human error. Development and security teams must look closely at how they configure images, secrets, namespaces, runtime privileges, network policies, persistent storage, and the control-plane - especially for self-managed Kubernetes deployments.

Kubernetes vulnerabilities are another major source of risk, and one that continues to increase given the mounting frequency of CVEs impacting the ecosystem, for example CVE-2020-8554, a man-in-the-middle vulnerability announced in December 2020. Common exploits of known vulnerabilities include crypto mining and other malware installation as well as privilege escalation and host access. The problem is pervasive and while image scanning at the build stage is necessary, vulnerabilities also pose a security risk to deployments in production. Effective vulnerability management spans the entire container lifecycle, and should:

■ Identify vulnerabilities in images

■ Prevent images with risky vulnerabilities from getting pushed to production-accessible container registries

■ Fail builds containing fixable vulnerabilities above a certain severity

■ Leverage custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images

The runtime phase introduces additional threats for Kubernetes users from external adversaries. In this phase, software is in the wild, so there are additional tactics teams should use to defend against these risks. It starts with monitoring runtime activity and establishing a baseline of known application behavior, particularly the most security-relevant container activities such as process activity and network communications among and between containerized services and external clients and servers. Kubernetes’ declarative data is also critical in this phase. Using date from the build and deploy phases can help evaluate observed versus expected activity during runtime in order to identify suspicious activity. Limiting unnecessary network communication is another best practice. Runtime is when you can see what kind of network traffic is needed for the application to function, as opposed to what’s allowed, offering the opportunity to remove unnecessary connections and reduce complexity that can introduce risks. Finally, users should leverage process allow-lists after observing an application for a period of time, identifying processes that are executed in the normal course of its behavior, then using this as an allow list against future application behavior.

Compliance, being one of the primary drivers behind container and Kubernetes security initiatives, is one final area that must be taken very seriously. Sixteen percent of StackRox’s survey respondents admitted failing a compliance audit in the past year - often this can be attributed to security becoming an afterthought in the journey towards container adoption. There are several compliance standards specific to containers and Kubernetes that apply to all organizations — CIS Benchmark for Docker, CIS Benchmark for Kubernetes, NIST SP 800-190. Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A too-common mistake is to wait until containers are in production before considering their compliance requirements, or only focusing on compliance at the runtime phase. The best way to mitigate risk from failing a compliance audit is to implement security controls as early as possible in the container life cycle. Automating compliance checks and evidence reporting as much as possible will help to reduce operational overhead.

Organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments more than they already are. Given the lack of confidence in security and compliance being felt within DevOps and security teams, it’s natural to hold back on production deployments, but it will limit the benefits of containerization and Kubernetes adoption. Start small by looking for ways to gain visibility into cloud-native environments. Then try to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. With this information, security and DevOps can begin to enforce policies to reduce risks to an acceptable level and work towards fulfilling their container and Kubernetes adoption goals.

Ali Golshan is CTO and Co-Founder of StackRox
Share this

Industry News

May 12, 2022

Red Hat introduced Red Hat Enterprise Linux 9, the Linux operating system designed to drive more consistent innovation across the open hybrid cloud, from bare metal servers to cloud providers and the farthest edge of enterprise networks.

May 12, 2022

Couchbase announced version 7.1 of Couchbase Server.

May 12, 2022

Copado added Copado Robotic Testing to Copado Essentials.

May 11, 2022

Red Hat announced new advancements within its Red Hat Cloud Services portfolio, delivering a fully-managed and streamlined user experience as organizations build, deploy, manage and scale cloud-native applications across hybrid environments.

May 11, 2022

JFrog introduced a new Docker Desktop Extension for JFrog Xray that allows organizations to automatically scan Docker Containers for vulnerabilities and violations early in the development process.

May 11, 2022

Progress announced a series of updates in Progress Telerik and Progress Kendo UI.

May 11, 2022

Vultr announces that Vultr Kubernetes Engine (VKE) is generally available.

May 10, 2022

Docker announced new features and partnerships to increase developer productivity. Specifically, the company announced Docker Extensions which allow developers to discover and add complementary development tools to Docker Desktop.

May 10, 2022

Red Hat announced the general availability of Red Hat Ansible Automation Platform on Microsoft Azure, pairing hybrid cloud automation with the convenience and support of a managed offering.

May 10, 2022

The Fedora Project, a community-driven open source collaboration sponsored by Red Hat, announced the general availability of Fedora Linux 36, the latest version of the fully open source Fedora operating system.

May 10, 2022

Progress announced the release of Progress Chef Cloud Security, extending DevSecOps with compliance support for native cloud assets and enabling end-to-end management of all on premise, cloud and native cloud resources.

This new offering is complemented with new capabilities across the Chef portfolio targeting DevOps success in the most demanding and complex enterprise deployments.

May 10, 2022

Platform9 announced new platform capabilities in Platform9 5.5 that make it easier for cloud-native development and operations teams to build, scale, and operate apps and Kubernetes clusters in the cloud, on-premises, and at the edge.

May 09, 2022

Red Hat and Accenture have expanded their nearly 12 year strategic partnership to further power open hybrid cloud innovation for enterprises worldwide.

May 09, 2022

Opsera has partnered with Mindtree.

May 09, 2022

Mendix announced that Mendix Workflow for process automation is now generally available.