Leveraging Kubernetes to Drive Big Value From Big Data
October 04, 2021

Gary Ogasawara
Cloudian

The growing adoption of Kubernetes frameworks by VMware, Red Hat and other major infrastructure and cloud players is making it easier to support cloud-native apps for big data. This is an important development, as it enables organizations to unlock additional value from that data.

Big data has significant gravity. Therefore, it's easier and more cost-effective to run analytics or other applications close to the data rather than moving the data to the applications. That's why Kubernetes is key: one of its biggest benefits is extreme agility, and by leveraging Kubernetes, organizations can swiftly move workloads to their big data lakes.

Moreover, Kubernetes makes it easy to deploy the same workloads in different environments, including private clouds, public clouds and the edge. For example, a machine learning algorithm can be developed and deployed in the public cloud, then also deployed at the edge.

Kubernetes has significant potential to accelerate big data deployments. Despite being a relatively young technology, Kubernetes has matured rapidly over the past two years. Thanks to its inherent agility and portability, Kubernetes has seen increasing adoption to support an expanding range of use cases. According to data published late last year, its use among development teams grew from 27% in 2018 to 48% in 2020. In addition, a July 2021 Red Hat survey found that 74% or respondents were using Kubernetes in production environments.

However, for organizations to maximize Kubernetes' value for big data deployments, they need to build a storage strategy that successfully bridges the gap between data repositories and application workloads.

Key Storage Considerations

By adopting a common storage platform across the Kubernetes ecosystem, IT teams can deliver the persistent capabilities that application developers need, make life easier for IT administrators and minimize storage costs. For example, according to analysis from Enterprise Strategy Group (ESG), "maximizing the potential of container-based applications . . . mandates a modern persistent storage foundation — one that can deliver consistent cloud-like access across both on-premises datacenter infrastructure and the public cloud."

This is important if IT professionals are to address key challenges in container-based environments. According to ESG research data, the cost of storage infrastructure is seen as the biggest challenge (cited by 37% of respondents), followed by overall storage performance (36%). Of similar importance are managing container storage environments across a hybrid/multi-cloud environment (35%), ensuring data availability (33%) and backing up/protecting storage for containers (33%).

To address these and other priorities, Kubernetes applications require storage that's agile, scalable, S3-compliant, secure and compatible with today's cloud-native infrastructure technologies and services. On the one hand, this is critical to ensure that Kubernetes users avoid vendor lock-in, while on the other, they also need maximum application portability so developers can write code once and then deploy it where it's needed.

It's particularly important that storage provides S3 compatibility to support these deployments. Modern apps tend to grow rapidly, especially modern apps that support big data use cases. Kubernetes-based, cloud-native environments require highly scalable infrastructure to accommodate this massive growth. S3-based object storage provides limitless scalability because of its horizontal, scale-out architecture. This architecture enables organizations to increase deployments by adding nodes whenever and wherever needed. Because S3-based storage uses a single, global namespace, this scaling can also be done across multiple geographic sites at once.

In addition, S3 compliance is also important for bolstering Kubernetes' portability. As noted earlier, portability is one of the technology's key benefits — organizations look to Kubernetes to help deploy the same apps immediately across different environments, such as on-prem and public clouds. S3-based storage enables seamless integration between private and public cloud deployments, allowing enterprises to quickly move apps and data between the two. This maximizes the agility of Kubernetes apps, making it easier to overcome data gravity and move workloads to big data lakes.

Enterprise-grade security is also a crucial storage consideration for Kubernetes applications used for big data. Modern, cloud-native apps are more complex and have more attack surfaces than traditional apps. As a result, they're harder to secure. Generally, these apps are susceptible to the same vulnerabilities as traditional apps, including ransomware, arguably today's biggest data security threat. To protect Kubernetes apps against ransomware attacks, storage platforms should offer robust encryption capabilities and leverage immutable backup data. By encrypting all sensitive data, including data both in flight and at rest, it becomes impossible for cybercriminals to read or expose that data in any intelligible form. Meanwhile, the use of immutable backup data prevents cybercriminals from being able to alter or delete data and enables quick recovery through an uninfected backup copy.

Broadly speaking, to ensure their storage systems provide comprehensive protection for Kubernetes apps, organizations should look at storage platforms that have earned major security and compliance certifications. These include the Common Criteria for Information Technology Security Evaluation, the Federal Information Processing Standard (FIPS) and SEC Rule 17a-4, among others.

In addition to these considerations, organizations should focus on delivering self-serve storage access using the standard Kubernetes Persistent Volume (PV) and Persistent Volume Claim (PVC) methodology to provision assets. This approach will yield a storage solution for Kubernetes that's both agile and standards-based.

Conclusion

Too often, big data remains inaccessible and organizations that are focused on fully exploiting its business potential require cloud-native flexibilities to run their workloads. As ESG explains, "Trying to address these . . . needs with traditional storage that simply 'bolts-on' cloud-like features, or with cloudlike storage that may be missing some essential enterprise features, will only unnecessarily burden and frustrate IT administrators while slowing down application development initiatives."

Organizations that address these issues via storage platforms that offer the inherent flexibility, security and scale-out properties required in a Kubernetes strategy will be well positioned to deliver on the potential of big data.

Gary Ogasawara is CTO at Cloudian
Share this

Industry News

October 21, 2021

Splunk announced the latest enhancements to the Splunk observability portfolio including advanced product innovations for Splunk Application Performance Monitoring (APM), Splunk Real User Monitoring (RUM), Splunk Synthetic Monitoring, Splunk Log Observer, Splunk Infrastructure Monitoring and Splunk IT Service Intelligence.

October 21, 2021

Platform9 announced a number of new enterprise features that greatly eliminate operational complexities in managing multi-cluster and multicloud Kubernetes deployments, available in the new release, Platform9 5.4 Release.

October 21, 2021

Graylog is announcing Graylog Security.

Designed to overcome legacy Security Information & Event Management (SIEM) challenges, Graylog’s scalable, flexible cybersecurity platform makes security analysts’ jobs easier and faster. With SIEM, Anomaly Detection, and User Entity Behavior Analytics (UEBA) capabilities, Graylog’s security solution will provide security teams with even greater confidence, productivity, and expertise to mitigate risks caused by Insider Threats, credential-based attacks, and other cyber threats.

October 21, 2021

Prodly announced a Series A investment of $10 million led by Leta Capital.

October 20, 2021

SonarSource added over 5,000 customers in the last 12 months, reaching the 15,000 commercial customers milestone in record time.

October 20, 2021

Actian announced the general availability of its newly released DataConnect 12 integration platform, demonstrating a continued focus on ease of use for complex data integration and data quality.

October 20, 2021

Salt Security announced new capabilities in its next-generation Salt Security API Protection Platform to secure GraphQL APIs.

October 20, 2021

vFunction announces the availability of the vFunction Application Transformation Engine and the expanded vFunction Modernization Platform, with new, advanced capabilities that enable enterprises to automatically assess, analyze, and manage the full modernization and migration process from start to finish.

October 20, 2021

Mage raised a $6.3 million seed round led by Gradient Ventures.

October 19, 2021

Couchbase announced its Couchbase Capella hosted Database-as-a-Service (DBaaS) offering on Amazon Web Services (AWS).

October 19, 2021

Checkmarx announced the launch of the Checkmarx Application Security Platform to help CISOs, AppSec teams, and developers address the growing and dynamic security challenges they face.

October 19, 2021

Tasktop announced Affinity Modeling for model-based integration in Tasktop Hub, helping Agile and DevOps software delivery teams reduce time to market and develop software faster.

October 19, 2021

Morpheus Data is continuing released version 5.3.3 targeted at enterprises trying to manage a complex mix of VMware, Kubernetes, and Public Cloud services.

October 19, 2021

Okta announced the availability of Okta Workflows as a standalone offering for all customers.

October 18, 2021

Red Hat announced a series of updates in its portfolio of developer tools and programs aimed at delivering greater productivity, security and scale for developers building applications on Red Hat OpenShift.