With Kubernetes Adoption at All Time High, Security and Compliance Concerns Still Inhibit Production Deployments
Even as DevSecOps becomes mainstream, container and Kubernetes security incidents are strikingly common
February 11, 2021

Ali Golshan
StackRox

We all wish we could build, deploy, and run our applications without the stress of security concerns. However, the reality is that most of us will run into serious security or compliance issues at one time or another. When that happens, an organization is likely to experience the frustration of delayed application deployments and stifled agility. Containers and Kubernetes promise faster development cycles, quicker bug fixes, and increased velocity, but when security is an afterthought, organizations risk the very gains that containerization promises, particularly agility.

The recent StackRox State of Container and Kubernetes Security Survey found that 44% of survey respondents had to delay an application rollout due to container or Kubernetes security issues. But the right combination of policies, processes and tooling will help mitigate these security and compliance concerns so that teams are more confident pushing container and Kubernetes deployments into production, which should be the culmination of any successful container strategy.

It's interesting to note how DevOps and security teams tend to deploy Kubernetes — roughly 50% of users self-manage their deployments, according to the most recent survey. In this model, these teams must be aware of the security pitfalls and the necessary requirements needed to meet compliance standards. Not that self-managed Kubernetes is less-secure, rather, it simply requires more diligence with updates and patching and more fine tuning of Kubernetes-native security controls. The alternative is tapping into the declarative power of a Kubernetes-native security platform that integrates with the existing tooling of a self-managed environment. Whether provider-managed or self-managed, Kubernetes users must take care to protect their applications and infrastructure from vulnerabilities and misconfigurations, ensure compliance with external and internal policies, and detect and stop runtime threats.

Two-thirds of survey respondents referenced security and compliance as their largest source of concern with their organizations' container strategies. Inadequate investment in security led the list (34%) while others felt their organizations did not take threats seriously enough (15%). There were additional concerns that compliance needs were not being accounted for (17%) as well.

Despite security concerns, organizations are increasingly adopting DevSecOp models that enable security to be tightly integrated with application development and operations starting in the build phase of the software development lifecycle. The survey found that the vast majority of respondents have some form of DevSecOps initiative underway. Only 17% continue to operate DevOps separately from security, proving that DevSecOps is more than a buzzword. This shows promise for the ability of organizations to continually shed concerns about security in order to optimize container and Kubernetes deployments for production.

However, organizational development and IT operations are only part of the equation. There are major container and Kubernetes security risks that exist beyond the realm of container strategies that must be addressed. According to the survey, 90% of respondents claimed to have experienced a security incident in their container and Kubernetes environment in the last 12 months. The most common types of security incidents as reported by respondents include misconfigurations and exposures, Kubernetes vulnerabilities, runtime threats, and failed compliance audits. As such, it's critical that organizations take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments further.

Exposures due to misconfigurations were some of the most worrisome security risks in container and Kubernetes environments, and for good reason as 67% of the survey respondents detected a serious misconfiguration in the last 12 months. These risks fall under the banner of configuration management, which pose a unique security challenge when using Kubernetes to orchestrate containerized applications. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration.

Teams generally know not to expose Kubernetes dashboards to the Internet but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right. Because of this, configuration management should be automated to control the risk of human error due to error-prone manual processes. The survey indicated that most misconfiguration-related security incidents in Kubernetes are caused by human error. Development and security teams must look closely at how they configure images, secrets, namespaces, runtime privileges, network policies, persistent storage, and the control-plane - especially for self-managed Kubernetes deployments.

Kubernetes vulnerabilities are another major source of risk, and one that continues to increase given the mounting frequency of CVEs impacting the ecosystem, for example CVE-2020-8554, a man-in-the-middle vulnerability announced in December 2020. Common exploits of known vulnerabilities include crypto mining and other malware installation as well as privilege escalation and host access. The problem is pervasive and while image scanning at the build stage is necessary, vulnerabilities also pose a security risk to deployments in production. Effective vulnerability management spans the entire container lifecycle, and should:

■ Identify vulnerabilities in images

■ Prevent images with risky vulnerabilities from getting pushed to production-accessible container registries

■ Fail builds containing fixable vulnerabilities above a certain severity

■ Leverage custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images

The runtime phase introduces additional threats for Kubernetes users from external adversaries. In this phase, software is in the wild, so there are additional tactics teams should use to defend against these risks. It starts with monitoring runtime activity and establishing a baseline of known application behavior, particularly the most security-relevant container activities such as process activity and network communications among and between containerized services and external clients and servers. Kubernetes’ declarative data is also critical in this phase. Using date from the build and deploy phases can help evaluate observed versus expected activity during runtime in order to identify suspicious activity. Limiting unnecessary network communication is another best practice. Runtime is when you can see what kind of network traffic is needed for the application to function, as opposed to what’s allowed, offering the opportunity to remove unnecessary connections and reduce complexity that can introduce risks. Finally, users should leverage process allow-lists after observing an application for a period of time, identifying processes that are executed in the normal course of its behavior, then using this as an allow list against future application behavior.

Compliance, being one of the primary drivers behind container and Kubernetes security initiatives, is one final area that must be taken very seriously. Sixteen percent of StackRox’s survey respondents admitted failing a compliance audit in the past year - often this can be attributed to security becoming an afterthought in the journey towards container adoption. There are several compliance standards specific to containers and Kubernetes that apply to all organizations — CIS Benchmark for Docker, CIS Benchmark for Kubernetes, NIST SP 800-190. Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A too-common mistake is to wait until containers are in production before considering their compliance requirements, or only focusing on compliance at the runtime phase. The best way to mitigate risk from failing a compliance audit is to implement security controls as early as possible in the container life cycle. Automating compliance checks and evidence reporting as much as possible will help to reduce operational overhead.

Organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments more than they already are. Given the lack of confidence in security and compliance being felt within DevOps and security teams, it’s natural to hold back on production deployments, but it will limit the benefits of containerization and Kubernetes adoption. Start small by looking for ways to gain visibility into cloud-native environments. Then try to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. With this information, security and DevOps can begin to enforce policies to reduce risks to an acceptable level and work towards fulfilling their container and Kubernetes adoption goals.

Ali Golshan is CTO and Co-Founder of StackRox
Share this

Industry News

December 06, 2021

Ascend.io announced support for Amazon Redshift Serverless powered on Amazon Web Services, Inc. (AWS), a fully managed petabyte-scale cloud data warehouse.

December 06, 2021

Neosec formed a strategic partnership with Kong Inc. to integrate its API security platform with Kong Gateway to provide a complete enterprise-class solution for managing and securing APIs and microservices.

December 02, 2021

Mirantis announced DevOpsCare, powered by Lens, a vendor-agnostic, fully-managed CI/CD (continuous integration/continuous deployment) product for any Kubernetes environment, offering developers higher levels of productivity more quickly.

December 02, 2021

The D2iQ Kubernetes Platform (DKP) is now available in AWS Marketplace, a digital catalog with thousands of software listings from independent software vendors that make it easy to find, test, buy, and deploy software that runs on Amazon Web Services, Inc. (AWS).

December 01, 2021

Bugcrowd announced the availability of Bugcrowd's cybersecurity solutions on the AWS Marketplace, providing customers with easy access, simplified billing, quick deployment, and streamlined license management.

December 01, 2021

Kublr received Microsoft Azure Arc-enabled Kubernetes validation, including for Azure Arc-enabled Kubernetes for Data Services.

December 01, 2021

CloudSphere achieved Amazon Web Services (AWS) Migration and Modernization Competency for discovering, planning, and helping enterprise customers move business services to AWS to reduce cost, increase agility and improve security.

November 30, 2021

JFrog introduced a new container registry and package manager for running JFrog Artifactory with Kubernetes clusters on-premises, in the cloud, or both.

November 30, 2021

Docker announced the availability of Docker Official Images directly from Amazon Web Services (AWS).

November 30, 2021

Weaveworks announced the general availability of Weave GitOps Enterprise, a GitOps platform that automates continuous application delivery and Kubernetes operations at any scale.

November 30, 2021

Amazon Web Services announced AWS Mainframe Modernization, a new service that makes it faster and easier for customers to migrate mainframe and legacy workloads to the cloud, and enjoy the superior agility, elasticity, and cost savings of AWS.

November 29, 2021

Quali announced the newest release of Torque Enterprise, which includes enhanced integration with Terraform, new custom tagging capabilities, and improved cost visibility dashboards, unleashing an entirely new level of self-service access to application environments on demand.

November 29, 2021

Vertical Relevance (VR), a financial services-focused consulting firm, achieved Amazon Web Services (AWS) DevOps Competency status.

November 18, 2021

Loft Labs announced the launch of Loft version 2 with a focus on ease of use that overcomes the major complaint that Kubernetes is complex and hard to set up.

November 18, 2021

Perforce Software announced new functionality to speed remediation of discovered defects in automated scans.