With Kubernetes Adoption at All Time High, Security and Compliance Concerns Still Inhibit Production Deployments
Even as DevSecOps becomes mainstream, container and Kubernetes security incidents are strikingly common
February 11, 2021

Ali Golshan

We all wish we could build, deploy, and run our applications without the stress of security concerns. However, the reality is that most of us will run into serious security or compliance issues at one time or another. When that happens, an organization is likely to experience the frustration of delayed application deployments and stifled agility. Containers and Kubernetes promise faster development cycles, quicker bug fixes, and increased velocity, but when security is an afterthought, organizations risk the very gains that containerization promises, particularly agility.

The recent StackRox State of Container and Kubernetes Security Survey found that 44% of survey respondents had to delay an application rollout due to container or Kubernetes security issues. But the right combination of policies, processes and tooling will help mitigate these security and compliance concerns so that teams are more confident pushing container and Kubernetes deployments into production, which should be the culmination of any successful container strategy.

It's interesting to note how DevOps and security teams tend to deploy Kubernetes — roughly 50% of users self-manage their deployments, according to the most recent survey. In this model, these teams must be aware of the security pitfalls and the necessary requirements needed to meet compliance standards. Not that self-managed Kubernetes is less-secure, rather, it simply requires more diligence with updates and patching and more fine tuning of Kubernetes-native security controls. The alternative is tapping into the declarative power of a Kubernetes-native security platform that integrates with the existing tooling of a self-managed environment. Whether provider-managed or self-managed, Kubernetes users must take care to protect their applications and infrastructure from vulnerabilities and misconfigurations, ensure compliance with external and internal policies, and detect and stop runtime threats.

Two-thirds of survey respondents referenced security and compliance as their largest source of concern with their organizations' container strategies. Inadequate investment in security led the list (34%) while others felt their organizations did not take threats seriously enough (15%). There were additional concerns that compliance needs were not being accounted for (17%) as well.

Despite security concerns, organizations are increasingly adopting DevSecOp models that enable security to be tightly integrated with application development and operations starting in the build phase of the software development lifecycle. The survey found that the vast majority of respondents have some form of DevSecOps initiative underway. Only 17% continue to operate DevOps separately from security, proving that DevSecOps is more than a buzzword. This shows promise for the ability of organizations to continually shed concerns about security in order to optimize container and Kubernetes deployments for production.

However, organizational development and IT operations are only part of the equation. There are major container and Kubernetes security risks that exist beyond the realm of container strategies that must be addressed. According to the survey, 90% of respondents claimed to have experienced a security incident in their container and Kubernetes environment in the last 12 months. The most common types of security incidents as reported by respondents include misconfigurations and exposures, Kubernetes vulnerabilities, runtime threats, and failed compliance audits. As such, it's critical that organizations take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments further.

Exposures due to misconfigurations were some of the most worrisome security risks in container and Kubernetes environments, and for good reason as 67% of the survey respondents detected a serious misconfiguration in the last 12 months. These risks fall under the banner of configuration management, which pose a unique security challenge when using Kubernetes to orchestrate containerized applications. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration.

Teams generally know not to expose Kubernetes dashboards to the Internet but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right. Because of this, configuration management should be automated to control the risk of human error due to error-prone manual processes. The survey indicated that most misconfiguration-related security incidents in Kubernetes are caused by human error. Development and security teams must look closely at how they configure images, secrets, namespaces, runtime privileges, network policies, persistent storage, and the control-plane - especially for self-managed Kubernetes deployments.

Kubernetes vulnerabilities are another major source of risk, and one that continues to increase given the mounting frequency of CVEs impacting the ecosystem, for example CVE-2020-8554, a man-in-the-middle vulnerability announced in December 2020. Common exploits of known vulnerabilities include crypto mining and other malware installation as well as privilege escalation and host access. The problem is pervasive and while image scanning at the build stage is necessary, vulnerabilities also pose a security risk to deployments in production. Effective vulnerability management spans the entire container lifecycle, and should:

■ Identify vulnerabilities in images

■ Prevent images with risky vulnerabilities from getting pushed to production-accessible container registries

■ Fail builds containing fixable vulnerabilities above a certain severity

■ Leverage custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images

The runtime phase introduces additional threats for Kubernetes users from external adversaries. In this phase, software is in the wild, so there are additional tactics teams should use to defend against these risks. It starts with monitoring runtime activity and establishing a baseline of known application behavior, particularly the most security-relevant container activities such as process activity and network communications among and between containerized services and external clients and servers. Kubernetes’ declarative data is also critical in this phase. Using date from the build and deploy phases can help evaluate observed versus expected activity during runtime in order to identify suspicious activity. Limiting unnecessary network communication is another best practice. Runtime is when you can see what kind of network traffic is needed for the application to function, as opposed to what’s allowed, offering the opportunity to remove unnecessary connections and reduce complexity that can introduce risks. Finally, users should leverage process allow-lists after observing an application for a period of time, identifying processes that are executed in the normal course of its behavior, then using this as an allow list against future application behavior.

Compliance, being one of the primary drivers behind container and Kubernetes security initiatives, is one final area that must be taken very seriously. Sixteen percent of StackRox’s survey respondents admitted failing a compliance audit in the past year - often this can be attributed to security becoming an afterthought in the journey towards container adoption. There are several compliance standards specific to containers and Kubernetes that apply to all organizations — CIS Benchmark for Docker, CIS Benchmark for Kubernetes, NIST SP 800-190. Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A too-common mistake is to wait until containers are in production before considering their compliance requirements, or only focusing on compliance at the runtime phase. The best way to mitigate risk from failing a compliance audit is to implement security controls as early as possible in the container life cycle. Automating compliance checks and evidence reporting as much as possible will help to reduce operational overhead.

Organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments more than they already are. Given the lack of confidence in security and compliance being felt within DevOps and security teams, it’s natural to hold back on production deployments, but it will limit the benefits of containerization and Kubernetes adoption. Start small by looking for ways to gain visibility into cloud-native environments. Then try to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. With this information, security and DevOps can begin to enforce policies to reduce risks to an acceptable level and work towards fulfilling their container and Kubernetes adoption goals.

Ali Golshan is CTO and Co-Founder of StackRox
Share this

Industry News

March 02, 2021

JFrog announced that its DevOps Platform tools – JFrog Artifactory and JFrog Xray – are available with native deployment templates for customers using AWS GovCloud (US) and Azure Government clouds.

March 02, 2021

Spectro Cloud announced support for existing Kubernetes environments, including clusters on public cloud services such as Amazon EKS, Azure AKS and Google GKE, has been added to the Spectro Cloud Kubernetes management platform.

March 02, 2021

Idera announced the acquisition of PreEmptive Solutions, LLC, a provider of application protection and security.

March 01, 2021

CloudBolt Software announced the launch of OneFuse Community Edition, a free version of its codeless integration platform for automating, integrating, and extending private and hybrid cloud infrastructures.

March 01, 2021

DBmaestro launched support for Snowflake, the Data Cloud company.

March 01, 2021

Platform9 closed Series-D funding with an additional $12.5 million for a total of $37.5 million.

February 25, 2021

Red Hat announced Red Hat OpenShift 4.7, the latest version of the company’s enterprise Kubernetes platform.

February 25, 2021

Granulate announced the release of its open-source platform, the G-Profiler, a production profiling solution that measures the performance of code in production applications to facilitate compute optimization.

February 25, 2021

Checkmarx announced the launch of KICS (Keeping Infrastructure as Code Secure), an open source static analysis solution that enables developers to write more secure infrastructure as code (IaC).

February 24, 2021

Applause launched its Product Excellence Platform (PEP).

February 24, 2021

Mabl announced the beta release of their new native desktop application that empowers users to easily automate testing for browsers, mobile browsers, and APIs.

February 24, 2021

D2iQ announced the general availability of D2iQ Kaptain, the cloud native end-to-end platform for running ML workloads on Kubernetes.

February 23, 2021

Edge Delta announced new capabilities for dynamically processing and routing logs, metrics, and traces -- allowing DevOps, Security, and SRE teams to analyze large volumes of streaming data without the cost, delay, or complexity of requiring data to be indexed, helping customers decouple where machine data is analyzed from where it is stored.

February 23, 2021

env0 announced a remote run SaaS solution for the automation of Terragrunt workloads across multiple Terraform modules.

February 23, 2021

Platform9 closed its Series-D funding with an additional $12.5 million for a total of $37.5 million.