Leveraging Kubernetes to Drive Big Value From Big Data
October 04, 2021

Gary Ogasawara
Cloudian

The growing adoption of Kubernetes frameworks by VMware, Red Hat and other major infrastructure and cloud players is making it easier to support cloud-native apps for big data. This is an important development, as it enables organizations to unlock additional value from that data.

Big data has significant gravity. Therefore, it's easier and more cost-effective to run analytics or other applications close to the data rather than moving the data to the applications. That's why Kubernetes is key: one of its biggest benefits is extreme agility, and by leveraging Kubernetes, organizations can swiftly move workloads to their big data lakes.

Moreover, Kubernetes makes it easy to deploy the same workloads in different environments, including private clouds, public clouds and the edge. For example, a machine learning algorithm can be developed and deployed in the public cloud, then also deployed at the edge.

Kubernetes has significant potential to accelerate big data deployments. Despite being a relatively young technology, Kubernetes has matured rapidly over the past two years. Thanks to its inherent agility and portability, Kubernetes has seen increasing adoption to support an expanding range of use cases. According to data published late last year, its use among development teams grew from 27% in 2018 to 48% in 2020. In addition, a July 2021 Red Hat survey found that 74% or respondents were using Kubernetes in production environments.

However, for organizations to maximize Kubernetes' value for big data deployments, they need to build a storage strategy that successfully bridges the gap between data repositories and application workloads.

Key Storage Considerations

By adopting a common storage platform across the Kubernetes ecosystem, IT teams can deliver the persistent capabilities that application developers need, make life easier for IT administrators and minimize storage costs. For example, according to analysis from Enterprise Strategy Group (ESG), "maximizing the potential of container-based applications . . . mandates a modern persistent storage foundation — one that can deliver consistent cloud-like access across both on-premises datacenter infrastructure and the public cloud."

This is important if IT professionals are to address key challenges in container-based environments. According to ESG research data, the cost of storage infrastructure is seen as the biggest challenge (cited by 37% of respondents), followed by overall storage performance (36%). Of similar importance are managing container storage environments across a hybrid/multi-cloud environment (35%), ensuring data availability (33%) and backing up/protecting storage for containers (33%).

To address these and other priorities, Kubernetes applications require storage that's agile, scalable, S3-compliant, secure and compatible with today's cloud-native infrastructure technologies and services. On the one hand, this is critical to ensure that Kubernetes users avoid vendor lock-in, while on the other, they also need maximum application portability so developers can write code once and then deploy it where it's needed.

It's particularly important that storage provides S3 compatibility to support these deployments. Modern apps tend to grow rapidly, especially modern apps that support big data use cases. Kubernetes-based, cloud-native environments require highly scalable infrastructure to accommodate this massive growth. S3-based object storage provides limitless scalability because of its horizontal, scale-out architecture. This architecture enables organizations to increase deployments by adding nodes whenever and wherever needed. Because S3-based storage uses a single, global namespace, this scaling can also be done across multiple geographic sites at once.

In addition, S3 compliance is also important for bolstering Kubernetes' portability. As noted earlier, portability is one of the technology's key benefits — organizations look to Kubernetes to help deploy the same apps immediately across different environments, such as on-prem and public clouds. S3-based storage enables seamless integration between private and public cloud deployments, allowing enterprises to quickly move apps and data between the two. This maximizes the agility of Kubernetes apps, making it easier to overcome data gravity and move workloads to big data lakes.

Enterprise-grade security is also a crucial storage consideration for Kubernetes applications used for big data. Modern, cloud-native apps are more complex and have more attack surfaces than traditional apps. As a result, they're harder to secure. Generally, these apps are susceptible to the same vulnerabilities as traditional apps, including ransomware, arguably today's biggest data security threat. To protect Kubernetes apps against ransomware attacks, storage platforms should offer robust encryption capabilities and leverage immutable backup data. By encrypting all sensitive data, including data both in flight and at rest, it becomes impossible for cybercriminals to read or expose that data in any intelligible form. Meanwhile, the use of immutable backup data prevents cybercriminals from being able to alter or delete data and enables quick recovery through an uninfected backup copy.

Broadly speaking, to ensure their storage systems provide comprehensive protection for Kubernetes apps, organizations should look at storage platforms that have earned major security and compliance certifications. These include the Common Criteria for Information Technology Security Evaluation, the Federal Information Processing Standard (FIPS) and SEC Rule 17a-4, among others.

In addition to these considerations, organizations should focus on delivering self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets. This approach will yield a storage solution for Kubernetes that's both agile and standards-based.

Conclusion

Too often, big data remains inaccessible and organizations that are focused on fully exploiting its business potential require cloud-native flexibilities to run their workloads. As ESG explains, "Trying to address these . . . needs with traditional storage that simply 'bolts-on' cloud-like features, or with cloudlike storage that may be missing some essential enterprise features, will only unnecessarily burden and frustrate IT administrators while slowing down application development initiatives."

Organizations that address these issues via storage platforms that offer the inherent flexibility, security and scale-out properties required in a Kubernetes strategy will be well positioned to deliver on the potential of big data.

Gary Ogasawara is CTO at Cloudian
Share this

Industry News

May 19, 2022

Jellyfish announced the launch of Jellyfish Benchmarks, a way to add context around engineering metrics and performance by introducing a method for comparison.

May 19, 2022

Solo.io announced the addition and integration of Cilium networking into its Gloo Mesh platform, providing a complete application-networking solution for companies’ cloud-native digital transformation efforts.

May 19, 2022

Aqua Security announced multiple updates to Aqua Trivy, making it a unified scanner for cloud native security.

May 18, 2022

Red Hat unveiled updates across its portfolio of developer tools designed to help organizations build and deliver applications faster and more consistently across Kubernetes-based hybrid and multicloud environments.

May 18, 2022

Armory announced public early access to their new Continuous Deployment-as-a-Service product.

May 18, 2022

DataCore Software announced DataCore Bolt, enterprise-grade container-native storage software for DevOps.

May 17, 2022

DevOps Institute, a global professional association for advancing the human elements of DevOps, announced the release of the Upskilling IT 2022 report.

May 17, 2022

Replicated announced a host of new platform features and capabilities that enable their customers to accelerate enterprise adoption of their Kubernetes applications.

May 17, 2022

Codefresh announced that its flagship continuous delivery (CD) platform will be made accessible as a fully-hosted solution for DevOps teams seeking to quickly and easily achieve frictionless, GitOps-based continuous software delivery in the cloud.

May 16, 2022

Red Hat announced new capabilities and enhancements across its portfolio of open hybrid cloud solutions aimed at accelerating enterprise adoption of edge compute architectures through the Red Hat Edge initiative.

May 16, 2022

D2iQ announced a partnership with GitLab.

May 16, 2022

Kasten by Veeam announced the new Kasten by Veeam K10 V5.0 Kubernetes data management platform.

May 12, 2022

Red Hat introduced Red Hat Enterprise Linux 9, the Linux operating system designed to drive more consistent innovation across the open hybrid cloud, from bare metal servers to cloud providers and the farthest edge of enterprise networks.

May 12, 2022

Couchbase announced version 7.1 of Couchbase Server.

May 12, 2022

Copado added Copado Robotic Testing to Copado Essentials.