Leveraging Kubernetes to Drive Big Value From Big Data
October 04, 2021

Gary Ogasawara
Cloudian

The growing adoption of Kubernetes frameworks by VMware, Red Hat and other major infrastructure and cloud players is making it easier to support cloud-native apps for big data. This is an important development, as it enables organizations to unlock additional value from that data.

Big data has significant gravity. Therefore, it's easier and more cost-effective to run analytics or other applications close to the data rather than moving the data to the applications. That's why Kubernetes is key: one of its biggest benefits is extreme agility, and by leveraging Kubernetes, organizations can swiftly move workloads to their big data lakes.

Moreover, Kubernetes makes it easy to deploy the same workloads in different environments, including private clouds, public clouds and the edge. For example, a machine learning algorithm can be developed and deployed in the public cloud, then also deployed at the edge.

Kubernetes has significant potential to accelerate big data deployments. Despite being a relatively young technology, Kubernetes has matured rapidly over the past two years. Thanks to its inherent agility and portability, Kubernetes has seen increasing adoption to support an expanding range of use cases. According to data published late last year, its use among development teams grew from 27% in 2018 to 48% in 2020. In addition, a July 2021 Red Hat survey found that 74% or respondents were using Kubernetes in production environments.

However, for organizations to maximize Kubernetes' value for big data deployments, they need to build a storage strategy that successfully bridges the gap between data repositories and application workloads.

Key Storage Considerations

By adopting a common storage platform across the Kubernetes ecosystem, IT teams can deliver the persistent capabilities that application developers need, make life easier for IT administrators and minimize storage costs. For example, according to analysis from Enterprise Strategy Group (ESG), "maximizing the potential of container-based applications . . . mandates a modern persistent storage foundation — one that can deliver consistent cloud-like access across both on-premises datacenter infrastructure and the public cloud."

This is important if IT professionals are to address key challenges in container-based environments. According to ESG research data, the cost of storage infrastructure is seen as the biggest challenge (cited by 37% of respondents), followed by overall storage performance (36%). Of similar importance are managing container storage environments across a hybrid/multi-cloud environment (35%), ensuring data availability (33%) and backing up/protecting storage for containers (33%).

To address these and other priorities, Kubernetes applications require storage that's agile, scalable, S3-compliant, secure and compatible with today's cloud-native infrastructure technologies and services. On the one hand, this is critical to ensure that Kubernetes users avoid vendor lock-in, while on the other, they also need maximum application portability so developers can write code once and then deploy it where it's needed.

It's particularly important that storage provides S3 compatibility to support these deployments. Modern apps tend to grow rapidly, especially modern apps that support big data use cases. Kubernetes-based, cloud-native environments require highly scalable infrastructure to accommodate this massive growth. S3-based object storage provides limitless scalability because of its horizontal, scale-out architecture. This architecture enables organizations to increase deployments by adding nodes whenever and wherever needed. Because S3-based storage uses a single, global namespace, this scaling can also be done across multiple geographic sites at once.

In addition, S3 compliance is also important for bolstering Kubernetes' portability. As noted earlier, portability is one of the technology's key benefits — organizations look to Kubernetes to help deploy the same apps immediately across different environments, such as on-prem and public clouds. S3-based storage enables seamless integration between private and public cloud deployments, allowing enterprises to quickly move apps and data between the two. This maximizes the agility of Kubernetes apps, making it easier to overcome data gravity and move workloads to big data lakes.

Enterprise-grade security is also a crucial storage consideration for Kubernetes applications used for big data. Modern, cloud-native apps are more complex and have more attack surfaces than traditional apps. As a result, they're harder to secure. Generally, these apps are susceptible to the same vulnerabilities as traditional apps, including ransomware, arguably today's biggest data security threat. To protect Kubernetes apps against ransomware attacks, storage platforms should offer robust encryption capabilities and leverage immutable backup data. By encrypting all sensitive data, including data both in flight and at rest, it becomes impossible for cybercriminals to read or expose that data in any intelligible form. Meanwhile, the use of immutable backup data prevents cybercriminals from being able to alter or delete data and enables quick recovery through an uninfected backup copy.

Broadly speaking, to ensure their storage systems provide comprehensive protection for Kubernetes apps, organizations should look at storage platforms that have earned major security and compliance certifications. These include the Common Criteria for Information Technology Security Evaluation, the Federal Information Processing Standard (FIPS) and SEC Rule 17a-4, among others.

In addition to these considerations, organizations should focus on delivering self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets. This approach will yield a storage solution for Kubernetes that's both agile and standards-based.

Conclusion

Too often, big data remains inaccessible and organizations that are focused on fully exploiting its business potential require cloud-native flexibilities to run their workloads. As ESG explains, "Trying to address these . . . needs with traditional storage that simply 'bolts-on' cloud-like features, or with cloudlike storage that may be missing some essential enterprise features, will only unnecessarily burden and frustrate IT administrators while slowing down application development initiatives."

Organizations that address these issues via storage platforms that offer the inherent flexibility, security and scale-out properties required in a Kubernetes strategy will be well positioned to deliver on the potential of big data.

Gary Ogasawara is CTO at Cloudian
Share this

Industry News

October 06, 2022

Platform.sh announced it has partnered with MongoDB.

October 06, 2022

Veracode announced the enhancement of its Continuous Software Security Platform to include container security.

This early access program for Veracode Container Security is now underway for existing customers.

The new Veracode Container Security offering, designed to meet the needs of cloud-native software engineering teams, addresses vulnerability scanning, secure configuration, and secrets management requirements for container images.

October 06, 2022

Mirantis announced that Mirantis Container Runtime – latest generation of the Docker Enterprise Engine, the secure container runtime that forms the foundation of Mirantis Container Cloud and Mirantis Kubernetes Engine and is used at the heart of many other Kubernetes deployments – is now available in the Microsoft Azure Marketplace.

October 05, 2022

Perforce Software announced enhanced support for automated testing with the release of Helix ALM 2022.2.

October 05, 2022

Parasoft announced the latest releases of its API and microservices testing tools, including SOAtest, Virtualize, CTP, and Selenic.

October 05, 2022

Vaadin announced the release of four Acceleration Kits designed to make it faster and easier to build and modernize Java applications for enterprise use.

October 04, 2022

Pegasystems announced the latest release of Robot Studio, the robotic process automation (RPA) low-code authoring environment for Pega's intelligent automation platform.

October 04, 2022

EvolveWare announced the Agile Business Rules Extraction (Agile BRE) solution on its Intellisys platform.

October 04, 2022

Mabl announced new features that empower quality professionals to easily validate APIs as part of their integrated end-to-end tests.

October 03, 2022

Spectro Cloud announced a major new release of its Palette Edge platform.

October 03, 2022

Arcion announced agentless change data capture (CDC) for all of its supported databases and applications.

September 29, 2022

CloudBees announced the acquisition of ReleaseIQ to expand the company’s DevSecOps capabilities, empowering customers with a low-code, end-to-end release orchestration and visibility solution.

September 29, 2022

SmartBear continues expanding its commitment to the Atlassian Marketplace, adding Bugsnag for Jira and SwaggerHub Integration for Confluence.

Bugsnag developers monitoring application stability and documenting in Jira no longer need to interrupt their workflow to access the app. Developers working in SwaggerHub can use the macro to push API definitions and changes directly to other teams and business stakeholders that work within Confluence. By increasing the presence of SmartBear tools on the Atlassian Marketplace, the company continues meeting developers where they are.

September 29, 2022

Ox Security exited stealth today with $34M in funding led by Evolution Equity Partners, Team8, and M12, Microsoft's venture fund, with participation from Rain Capital.

September 29, 2022

cnvrg.io announced that the new Intel Developer Cloud is now available via the cnvrg.io Metacloud platform, providing a fully integrated software and hardware solution.