Leveraging Kubernetes to Drive Big Value From Big Data
October 04, 2021

Gary Ogasawara
Cloudian

The growing adoption of Kubernetes frameworks by VMware, Red Hat and other major infrastructure and cloud players is making it easier to support cloud-native apps for big data. This is an important development, as it enables organizations to unlock additional value from that data.

Big data has significant gravity. Therefore, it's easier and more cost-effective to run analytics or other applications close to the data rather than moving the data to the applications. That's why Kubernetes is key: one of its biggest benefits is extreme agility, and by leveraging Kubernetes, organizations can swiftly move workloads to their big data lakes.

Moreover, Kubernetes makes it easy to deploy the same workloads in different environments, including private clouds, public clouds and the edge. For example, a machine learning algorithm can be developed and deployed in the public cloud, then also deployed at the edge.

Kubernetes has significant potential to accelerate big data deployments. Despite being a relatively young technology, Kubernetes has matured rapidly over the past two years. Thanks to its inherent agility and portability, Kubernetes has seen increasing adoption to support an expanding range of use cases. According to data published late last year, its use among development teams grew from 27% in 2018 to 48% in 2020. In addition, a July 2021 Red Hat survey found that 74% or respondents were using Kubernetes in production environments.

However, for organizations to maximize Kubernetes' value for big data deployments, they need to build a storage strategy that successfully bridges the gap between data repositories and application workloads.

Key Storage Considerations

By adopting a common storage platform across the Kubernetes ecosystem, IT teams can deliver the persistent capabilities that application developers need, make life easier for IT administrators and minimize storage costs. For example, according to analysis from Enterprise Strategy Group (ESG), "maximizing the potential of container-based applications . . . mandates a modern persistent storage foundation — one that can deliver consistent cloud-like access across both on-premises datacenter infrastructure and the public cloud."

This is important if IT professionals are to address key challenges in container-based environments. According to ESG research data, the cost of storage infrastructure is seen as the biggest challenge (cited by 37% of respondents), followed by overall storage performance (36%). Of similar importance are managing container storage environments across a hybrid/multi-cloud environment (35%), ensuring data availability (33%) and backing up/protecting storage for containers (33%).

To address these and other priorities, Kubernetes applications require storage that's agile, scalable, S3-compliant, secure and compatible with today's cloud-native infrastructure technologies and services. On the one hand, this is critical to ensure that Kubernetes users avoid vendor lock-in, while on the other, they also need maximum application portability so developers can write code once and then deploy it where it's needed.

It's particularly important that storage provides S3 compatibility to support these deployments. Modern apps tend to grow rapidly, especially modern apps that support big data use cases. Kubernetes-based, cloud-native environments require highly scalable infrastructure to accommodate this massive growth. S3-based object storage provides limitless scalability because of its horizontal, scale-out architecture. This architecture enables organizations to increase deployments by adding nodes whenever and wherever needed. Because S3-based storage uses a single, global namespace, this scaling can also be done across multiple geographic sites at once.

In addition, S3 compliance is also important for bolstering Kubernetes' portability. As noted earlier, portability is one of the technology's key benefits — organizations look to Kubernetes to help deploy the same apps immediately across different environments, such as on-prem and public clouds. S3-based storage enables seamless integration between private and public cloud deployments, allowing enterprises to quickly move apps and data between the two. This maximizes the agility of Kubernetes apps, making it easier to overcome data gravity and move workloads to big data lakes.

Enterprise-grade security is also a crucial storage consideration for Kubernetes applications used for big data. Modern, cloud-native apps are more complex and have more attack surfaces than traditional apps. As a result, they're harder to secure. Generally, these apps are susceptible to the same vulnerabilities as traditional apps, including ransomware, arguably today's biggest data security threat. To protect Kubernetes apps against ransomware attacks, storage platforms should offer robust encryption capabilities and leverage immutable backup data. By encrypting all sensitive data, including data both in flight and at rest, it becomes impossible for cybercriminals to read or expose that data in any intelligible form. Meanwhile, the use of immutable backup data prevents cybercriminals from being able to alter or delete data and enables quick recovery through an uninfected backup copy.

Broadly speaking, to ensure their storage systems provide comprehensive protection for Kubernetes apps, organizations should look at storage platforms that have earned major security and compliance certifications. These include the Common Criteria for Information Technology Security Evaluation, the Federal Information Processing Standard (FIPS) and SEC Rule 17a-4, among others.

In addition to these considerations, organizations should focus on delivering self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets. This approach will yield a storage solution for Kubernetes that's both agile and standards-based.

Conclusion

Too often, big data remains inaccessible and organizations that are focused on fully exploiting its business potential require cloud-native flexibilities to run their workloads. As ESG explains, "Trying to address these . . . needs with traditional storage that simply 'bolts-on' cloud-like features, or with cloudlike storage that may be missing some essential enterprise features, will only unnecessarily burden and frustrate IT administrators while slowing down application development initiatives."

Organizations that address these issues via storage platforms that offer the inherent flexibility, security and scale-out properties required in a Kubernetes strategy will be well positioned to deliver on the potential of big data.

Gary Ogasawara is CTO at Cloudian
Share this

Industry News

March 18, 2024

Kubiya.ai announces the launch of its DevOps Digital Agents.

March 18, 2024

Aviatrix® introduced Aviatrix Distributed Cloud Firewall for Kubernetes, a distributed cloud networking and network security solution for containerized enterprise applications and workloads.

March 18, 2024

Stride announces the general availability of Stride Conductor, its new autonomous coding product that transforms the software development landscape.

March 14, 2024

CircleCI unveiled CircleCI releases, which enables developers to automate the release orchestration process directly from the CircleCI UI.

March 13, 2024

Fermyon™ Technologies announces Fermyon Platform for Kubernetes, a WebAssembly platform for Kubernetes.

March 13, 2024

Akuity announced a new offer targeted at Enterprises and businesses where security and compliance are key.

March 13, 2024

New Relic launched new capabilities for New Relic IAST (Interactive Application Security Testing), including proof-of-exploit reporting for application security testing.

March 12, 2024

OutSystems announced AI Agent Builder, a new solution in the OutSystems Developer Cloud platform that makes it easy for IT leaders to incorporate generative AI (GenAI) powered applications into their digital transformation strategy, as well as govern the use of AI to ensure standardization and security.

March 12, 2024

Mirantis announced significant updates to Lens Desktop that makes working with Kubernetes easier by simplifying operations, improving efficiency, and increasing productivity. Lens 2024 Early Access is now available to Lens users.

March 12, 2024

Codezero announced a $3.5 million seed-funding round led by Ballistic Ventures, the venture capital firm dedicated exclusively to funding entrepreneurs and innovations in cybersecurity.

March 11, 2024

Prismatic launched a code-native integration building experience.

March 07, 2024

Check Point® Software Technologies Ltd. announced its Check Point Infinity Platform has been ranked as the #1 Zero Trust Platform in the latest Miercom Zero Trust Platform Assessment.

March 07, 2024

Tricentis announced the launch and availability of SAP Test Automation by Tricentis as an SAP Solution Extension.

March 07, 2024

Netlify announced the general availability of the AI-enabled deploy assist.

March 07, 2024

DataStax announced a new integration with Airbyte that simplifies the process of building production-ready GenAI applications with structured and unstructured data.