With Kubernetes Adoption at All Time High, Security and Compliance Concerns Still Inhibit Production Deployments
Even as DevSecOps becomes mainstream, container and Kubernetes security incidents are strikingly common
February 11, 2021

Ali Golshan

We all wish we could build, deploy, and run our applications without the stress of security concerns. However, the reality is that most of us will run into serious security or compliance issues at one time or another. When that happens, an organization is likely to experience the frustration of delayed application deployments and stifled agility. Containers and Kubernetes promise faster development cycles, quicker bug fixes, and increased velocity, but when security is an afterthought, organizations risk the very gains that containerization promises, particularly agility.

The recent StackRox State of Container and Kubernetes Security Survey found that 44% of survey respondents had to delay an application rollout due to container or Kubernetes security issues. But the right combination of policies, processes and tooling will help mitigate these security and compliance concerns so that teams are more confident pushing container and Kubernetes deployments into production, which should be the culmination of any successful container strategy.

It's interesting to note how DevOps and security teams tend to deploy Kubernetes — roughly 50% of users self-manage their deployments, according to the most recent survey. In this model, these teams must be aware of the security pitfalls and the necessary requirements needed to meet compliance standards. Not that self-managed Kubernetes is less-secure, rather, it simply requires more diligence with updates and patching and more fine tuning of Kubernetes-native security controls. The alternative is tapping into the declarative power of a Kubernetes-native security platform that integrates with the existing tooling of a self-managed environment. Whether provider-managed or self-managed, Kubernetes users must take care to protect their applications and infrastructure from vulnerabilities and misconfigurations, ensure compliance with external and internal policies, and detect and stop runtime threats.

Two-thirds of survey respondents referenced security and compliance as their largest source of concern with their organizations' container strategies. Inadequate investment in security led the list (34%) while others felt their organizations did not take threats seriously enough (15%). There were additional concerns that compliance needs were not being accounted for (17%) as well.

Despite security concerns, organizations are increasingly adopting DevSecOp models that enable security to be tightly integrated with application development and operations starting in the build phase of the software development lifecycle. The survey found that the vast majority of respondents have some form of DevSecOps initiative underway. Only 17% continue to operate DevOps separately from security, proving that DevSecOps is more than a buzzword. This shows promise for the ability of organizations to continually shed concerns about security in order to optimize container and Kubernetes deployments for production.

However, organizational development and IT operations are only part of the equation. There are major container and Kubernetes security risks that exist beyond the realm of container strategies that must be addressed. According to the survey, 90% of respondents claimed to have experienced a security incident in their container and Kubernetes environment in the last 12 months. The most common types of security incidents as reported by respondents include misconfigurations and exposures, Kubernetes vulnerabilities, runtime threats, and failed compliance audits. As such, it's critical that organizations take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments further.

Exposures due to misconfigurations were some of the most worrisome security risks in container and Kubernetes environments, and for good reason as 67% of the survey respondents detected a serious misconfiguration in the last 12 months. These risks fall under the banner of configuration management, which pose a unique security challenge when using Kubernetes to orchestrate containerized applications. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration.

Teams generally know not to expose Kubernetes dashboards to the Internet but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right. Because of this, configuration management should be automated to control the risk of human error due to error-prone manual processes. The survey indicated that most misconfiguration-related security incidents in Kubernetes are caused by human error. Development and security teams must look closely at how they configure images, secrets, namespaces, runtime privileges, network policies, persistent storage, and the control-plane - especially for self-managed Kubernetes deployments.

Kubernetes vulnerabilities are another major source of risk, and one that continues to increase given the mounting frequency of CVEs impacting the ecosystem, for example CVE-2020-8554, a man-in-the-middle vulnerability announced in December 2020. Common exploits of known vulnerabilities include crypto mining and other malware installation as well as privilege escalation and host access. The problem is pervasive and while image scanning at the build stage is necessary, vulnerabilities also pose a security risk to deployments in production. Effective vulnerability management spans the entire container lifecycle, and should:

■ Identify vulnerabilities in images

■ Prevent images with risky vulnerabilities from getting pushed to production-accessible container registries

■ Fail builds containing fixable vulnerabilities above a certain severity

■ Leverage custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images

The runtime phase introduces additional threats for Kubernetes users from external adversaries. In this phase, software is in the wild, so there are additional tactics teams should use to defend against these risks. It starts with monitoring runtime activity and establishing a baseline of known application behavior, particularly the most security-relevant container activities such as process activity and network communications among and between containerized services and external clients and servers. Kubernetes’ declarative data is also critical in this phase. Using date from the build and deploy phases can help evaluate observed versus expected activity during runtime in order to identify suspicious activity. Limiting unnecessary network communication is another best practice. Runtime is when you can see what kind of network traffic is needed for the application to function, as opposed to what’s allowed, offering the opportunity to remove unnecessary connections and reduce complexity that can introduce risks. Finally, users should leverage process allow-lists after observing an application for a period of time, identifying processes that are executed in the normal course of its behavior, then using this as an allow list against future application behavior.

Compliance, being one of the primary drivers behind container and Kubernetes security initiatives, is one final area that must be taken very seriously. Sixteen percent of StackRox’s survey respondents admitted failing a compliance audit in the past year - often this can be attributed to security becoming an afterthought in the journey towards container adoption. There are several compliance standards specific to containers and Kubernetes that apply to all organizations — CIS Benchmark for Docker, CIS Benchmark for Kubernetes, NIST SP 800-190. Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A too-common mistake is to wait until containers are in production before considering their compliance requirements, or only focusing on compliance at the runtime phase. The best way to mitigate risk from failing a compliance audit is to implement security controls as early as possible in the container life cycle. Automating compliance checks and evidence reporting as much as possible will help to reduce operational overhead.

Organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments more than they already are. Given the lack of confidence in security and compliance being felt within DevOps and security teams, it’s natural to hold back on production deployments, but it will limit the benefits of containerization and Kubernetes adoption. Start small by looking for ways to gain visibility into cloud-native environments. Then try to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. With this information, security and DevOps can begin to enforce policies to reduce risks to an acceptable level and work towards fulfilling their container and Kubernetes adoption goals.

Ali Golshan is CTO and Co-Founder of StackRox
Share this

Industry News

October 03, 2023

Parasoft announced new advancements in its Continuous Quality Platform for functional solutions, which include Parasoft Virtualize, SOAtest, CTP, and DTP.

The latest releases introduce capabilities including:

- GenAI integration for API testing

- Comprehensive microservices code coverage

- Web accessibility testing

- Powerful learning mode for creating and updating virtual assets

These innovations are set to transform the landscape of software testing for enterprise application development and test teams.

October 03, 2023

LinearB announced the release of free DORA Metrics dashboards.

October 03, 2023

PerfectScale, a provider of Kubernetes optimization, has successfully closed $7.1 million in seed funding.

October 02, 2023

Spectro Cloud announced Palette EdgeAI to simplify how organizations deploy and manage AI workloads at scale across simple to complex edge locations, such as retail, healthcare, industrial automation, oil and gas, automotive/connected cars, and more.

September 28, 2023

Kong announced Kong Konnect Dedicated Cloud Gateways, the simplest and most cost-effective way to run Kong Gateways in the cloud fully managed as a service and on enterprise dedicated infrastructure.

September 28, 2023

Sisense unveiled the public preview of Compose SDK for Fusion.

September 28, 2023

Cloudflare announced Hyperdrive to make every local database global. Now developers can easily build globally distributed applications on Cloudflare Workers, the serverless developer platform used by over one million developers, without being constrained by their existing infrastructure.

September 27, 2023

Kong announced full support for Kong Mesh in Konnect, making Kong Konnect an API lifecycle management platform with built-in support for Kong Gateway Enterprise, Kong Ingress Controller and Kong Mesh via a SaaS control plane.

September 27, 2023

Vultr announced the launch of the Vultr GPU Stack and Container Registry to enable global enterprises and digital startups alike to build, test and operationalize artificial intelligence (AI) models at scale — across any region on the globe. \

September 27, 2023

Salt Security expanded its partnership with CrowdStrike by integrating the Salt Security API Protection Platform with the CrowdStrike Falcon® Platform.

September 26, 2023

Progress announced a partnership with Software Improvement Group (SIG), an independent technology and advisory firm for software quality, security and improvement, to help ensure the long-term maintainability and modernization of business-critical applications built on the Progress® OpenEdge® platform.

September 26, 2023

Solace announced a new version of its Solace Event Portal solution that gives organizations with Apache Kafka deployments better visibility into, and control over, their Kafka event streams, brokers and associated assets.

September 26, 2023

Reply launched a proprietary framework for generative AI-based software development, KICODE Reply.

September 26, 2023

Harness announced the industry-wide Engineering Excellence Collective™, an engineering leadership community.

September 25, 2023

Harness announced four new product modules on the Harness platform.