DEVOPSdigest

StreamSets announced immediate availability of StreamSets Control Hub, engineered to streamline the development and operational management of many-to-many dataflows.

Available in StreamSets Enterprise Edition, StreamSets Control Hub adds DevOps sensibilities to data movement architectures. It offers centralized collaborative design of dataflow topologies and enables testing, provisioning and elastically scalable execution of dataflows anywhere — on premises, on edge or in the cloud — via the open source StreamSets Data Collector (SDC) and recently announced SDC Edge.

StreamSets Control Hub addresses the challenges caused by the evolution of applications toward complex, time-sensitive data movement, where analytics are performed every step of the way. These dataflow architectures demand the same disciplined approach to design and provisioning that enterprises apply to the applications themselves. Traditional hand-coded and ad-hoc approaches to data pipelines slow delivery of data-driven applications, drive up development and maintenance costs, and make application performance fragile when changes occur to the data.

“As enterprises evolve to adopt continuous deployment and integration (CI/CD) practices, data movement architectures must keep pace, adjusting frequently to support new data sources, compute platforms and analytic steps that occur throughout the dataflow,” said Matt Aslett, research director, Data Platforms and Analytics at 451 Research. “StreamSets is well-positioned to take advantage of interest in continuous data integration as an enabler for real-time analytics by providing a central point for designing shareable and reusable dataflow patterns, and managing continual execution of data pipelines.”

StreamSets Control Hub includes the following new features:

- Hosted dataflow designer for collaborative design of smart pipelines and topologies that can execute anywhere, from edge devices to the data center and across multiple cloud providers

- Automated deployment and scalable execution of dataflows via Kubernetes today, and other container management methods in the future

- A development toolkit that provides a programmatic interface for automated build and scalable execution of pipelines, with support for Java and Python

- Governance integration with enterprise catalogs, starting with Cloudera Navigator and Apache Atlas

- Flexible hosting options, including via cloud.streamsets.com, private cloud, public cloud and on premises
“We have seen our Fortune 500 customers incur substantial effort and pain as they hand-craft solutions that operationalize their StreamSets dataflow architectures, scaling to hundreds of thousands of pipelines in scope,” said Girish Pancha, CEO of StreamSets. “An operational mindset across the lifecycle is required for modern analytic applications to succeed. We built StreamSets Control Hub to provide a focal point for dataflow management so that businesses can accelerate time to value for their data-driven applications.”