Leveraging Kubernetes to Drive Big Value From Big Data
October 04, 2021

Gary Ogasawara
Cloudian

The growing adoption of Kubernetes frameworks by VMware, Red Hat and other major infrastructure and cloud players is making it easier to support cloud-native apps for big data. This is an important development, as it enables organizations to unlock additional value from that data.

Big data has significant gravity. Therefore, it's easier and more cost-effective to run analytics or other applications close to the data rather than moving the data to the applications. That's why Kubernetes is key: one of its biggest benefits is extreme agility, and by leveraging Kubernetes, organizations can swiftly move workloads to their big data lakes.

Moreover, Kubernetes makes it easy to deploy the same workloads in different environments, including private clouds, public clouds and the edge. For example, a machine learning algorithm can be developed and deployed in the public cloud, then also deployed at the edge.

Kubernetes has significant potential to accelerate big data deployments. Despite being a relatively young technology, Kubernetes has matured rapidly over the past two years. Thanks to its inherent agility and portability, Kubernetes has seen increasing adoption to support an expanding range of use cases. According to data published late last year, its use among development teams grew from 27% in 2018 to 48% in 2020. In addition, a July 2021 Red Hat survey found that 74% or respondents were using Kubernetes in production environments.

However, for organizations to maximize Kubernetes' value for big data deployments, they need to build a storage strategy that successfully bridges the gap between data repositories and application workloads.

Key Storage Considerations

By adopting a common storage platform across the Kubernetes ecosystem, IT teams can deliver the persistent capabilities that application developers need, make life easier for IT administrators and minimize storage costs. For example, according to analysis from Enterprise Strategy Group (ESG), "maximizing the potential of container-based applications . . . mandates a modern persistent storage foundation — one that can deliver consistent cloud-like access across both on-premises datacenter infrastructure and the public cloud."

This is important if IT professionals are to address key challenges in container-based environments. According to ESG research data, the cost of storage infrastructure is seen as the biggest challenge (cited by 37% of respondents), followed by overall storage performance (36%). Of similar importance are managing container storage environments across a hybrid/multi-cloud environment (35%), ensuring data availability (33%) and backing up/protecting storage for containers (33%).

To address these and other priorities, Kubernetes applications require storage that's agile, scalable, S3-compliant, secure and compatible with today's cloud-native infrastructure technologies and services. On the one hand, this is critical to ensure that Kubernetes users avoid vendor lock-in, while on the other, they also need maximum application portability so developers can write code once and then deploy it where it's needed.

It's particularly important that storage provides S3 compatibility to support these deployments. Modern apps tend to grow rapidly, especially modern apps that support big data use cases. Kubernetes-based, cloud-native environments require highly scalable infrastructure to accommodate this massive growth. S3-based object storage provides limitless scalability because of its horizontal, scale-out architecture. This architecture enables organizations to increase deployments by adding nodes whenever and wherever needed. Because S3-based storage uses a single, global namespace, this scaling can also be done across multiple geographic sites at once.

In addition, S3 compliance is also important for bolstering Kubernetes' portability. As noted earlier, portability is one of the technology's key benefits — organizations look to Kubernetes to help deploy the same apps immediately across different environments, such as on-prem and public clouds. S3-based storage enables seamless integration between private and public cloud deployments, allowing enterprises to quickly move apps and data between the two. This maximizes the agility of Kubernetes apps, making it easier to overcome data gravity and move workloads to big data lakes.

Enterprise-grade security is also a crucial storage consideration for Kubernetes applications used for big data. Modern, cloud-native apps are more complex and have more attack surfaces than traditional apps. As a result, they're harder to secure. Generally, these apps are susceptible to the same vulnerabilities as traditional apps, including ransomware, arguably today's biggest data security threat. To protect Kubernetes apps against ransomware attacks, storage platforms should offer robust encryption capabilities and leverage immutable backup data. By encrypting all sensitive data, including data both in flight and at rest, it becomes impossible for cybercriminals to read or expose that data in any intelligible form. Meanwhile, the use of immutable backup data prevents cybercriminals from being able to alter or delete data and enables quick recovery through an uninfected backup copy.

Broadly speaking, to ensure their storage systems provide comprehensive protection for Kubernetes apps, organizations should look at storage platforms that have earned major security and compliance certifications. These include the Common Criteria for Information Technology Security Evaluation, the Federal Information Processing Standard (FIPS) and SEC Rule 17a-4, among others.

In addition to these considerations, organizations should focus on delivering self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets. This approach will yield a storage solution for Kubernetes that's both agile and standards-based.

Conclusion

Too often, big data remains inaccessible and organizations that are focused on fully exploiting its business potential require cloud-native flexibilities to run their workloads. As ESG explains, "Trying to address these . . . needs with traditional storage that simply 'bolts-on' cloud-like features, or with cloudlike storage that may be missing some essential enterprise features, will only unnecessarily burden and frustrate IT administrators while slowing down application development initiatives."

Organizations that address these issues via storage platforms that offer the inherent flexibility, security and scale-out properties required in a Kubernetes strategy will be well positioned to deliver on the potential of big data.

Gary Ogasawara is CTO at Cloudian
Share this

Industry News

January 26, 2023

Ubuntu Pro, Canonical’s comprehensive subscription for secure open source and compliance, is now generally available.

January 26, 2023

Mirantis, freeing developers to create their most valuable code, today announced that it has acquired the Santa Clara, California-based Shipa to add automated application discovery, operations, security, and observability to the Lens Kubernetes Platform.

January 25, 2023

SmartBear has integrated the powerful contract testing capabilities of PactFlow with SwaggerHub.

January 25, 2023

Venafi introduced TLS Protect for Kubernetes.

January 25, 2023

Tricentis announced the general availability of Tricentis Test Automation, a cloud-based test automation solution that simplifies test creation, orchestration, and scalable test execution for easier collaboration among QA teams and their business stakeholders and faster, higher-quality, and more durable releases of web-based applications and business processes.

January 24, 2023

Harness announced the acquisition of Propelo.

January 23, 2023

Couchbase announced its Couchbase Capella Database-as-a-Service (DBaaS) offering on Azure.

January 23, 2023

Mendix and Software Improvement Group (SIG) have announced the release of Mendix Quality & Security Management (QSM), a new cybersecurity solution that provides continuous deep-dive insights into security and code quality to immediately address risks and vulnerabilities.

January 23, 2023

Trunk announces the public launch of CI Analytics.

January 23, 2023

Panaya announced a new Partnership Program in response to ongoing growth within its partner network over the past year.

January 23, 2023

Cloudian closed $60 million in new funding, bringing the company’s total funding to $233 million.

January 19, 2023

Progress announced the R1 2023 release of Progress Telerik and Progress Kendo UI.

January 19, 2023

Wallarm announced the early release of the Wallarm API Leak Management solution, an enhanced API security technology designed to help organizations identify and remediate attacks exploiting leaked API keys and secrets, while providing on-going protection against hacks in the event of a leak.

January 19, 2023

ThreatModeler launched Threat Model Marketplace, a cybersecurity asset marketplace offering pre-built, field-tested threat models to be downloaded — free for a limited time — and incorporated into new and ongoing threat modeling initiatives.

January 18, 2023

Software AG has launched new updates to its webMethods platform that will simplify the process by which developers can find, work on and deploy new APIs and integration tools or capabilities.