Starburst Introduces Python DataFrame Support
September 07, 2023

Starburst extended support for Python with PyStarburst, and announced a new integration with the open source Python library Ibis, built in collaboration with composable data systems builder and Ibis maintainer Voltron Data.

For Starburst and Trino developers and data engineers, this announcement means that they no longer need to offload data to frameworks like PySpark and Snowpark to handle complex transformation workloads. Instead, teams can leverage a single, powerful MPP engine for both their analytical and transformation workloads – reducing the cost and complexity of their stack.

PyStarburst provides a familiar syntax to PySpark and Snowpark for writing and running production-grade ETL pipelines and data transformations, making it easy to not only build new pipelines with PyStarburst but also to migrate existing PySpark and Snowpark pipelines to Starburst without rewriting code.

"Many data engineers prefer writing code over SQL for transformations, and many software engineers are used to building data applications in Python. With PyStarburst, we're giving them the freedom to do so with the increased productivity and performance of Starburst's enterprise-grade Trino," said Martin Traverso, CTO of Starburst.

For developers and data engineers looking to build scalable data applications, the new Ibis integration provides a uniform Python API that can execute queries on more than 18 different engines – including DuckDB, pandas, PostgreSQL, and now Starburst Galaxy. This means you can scale from development on a laptop to production in Galaxy without rewriting a single line of code.

"At Starburst everything is built with openness in mind, and we are interoperable with nearly any data environment, so we're extending that commitment to our programming languages. The partnership with Voltron Data and Ibis was a natural fit," said Harrison Johnson, Head of Technology Partnerships at Starburst.

Together, Ibis and Starburst Galaxy empower users to write portable Python code that executes on Starburst's high-performance data lake analytics engine, operating on data from more than 50 supported sources. Users will now be able to build analytic expressions across multiple data sources with reusable scripts that execute at any scale.

"Python users struggle to bridge the gap between prototypes on their laptops and production apps running on platforms like Starburst Galaxy. Ibis makes it much easier to bridge this gap," said Josh Patterson, CEO of Voltron Data. "With Ibis, you can write Python code once and run it anywhere, with any supported backend execution engine. You can move seamlessly from crunching gigabyte-scale test data on your laptop to crunching petabyte-scale data in production using Starburst Galaxy."

Share this

Industry News

May 23, 2024

Kong announced the general availability of Kong Gateway Open Source (OSS) 3.7.

May 23, 2024

Azul announced the launch of its PartnerConnect training and certification program to empower channel partners to provide advanced Java advisory and delivery services.

May 22, 2024

Mendix announced a partnership with Snowflake to enable the enterprise to activate and drive maximum value from their data through low-code application development.

May 22, 2024

LaunchDarkly set the stage for “shipping at the speed of now” with the unveiling of new features, empowering engineering teams to streamline releases and accelerate the pace of innovation.

May 22, 2024

Tigera launched new features for Calico Enterprise and Calico Cloud, extending the products' Runtime Threat Defense capabilities.

May 22, 2024

Cirata announced the latest version of Cirata Gerrit MultiSite®.

May 21, 2024

Puppet by Perforce announced a significant enhancement to the capabilities of its commercial offering with the addition of new security, compliance, and continuous integration/continuous delivery (CI/CD) capabilities.

May 21, 2024

Red Hat and Nutanix announced an expanded collaboration to use Red Hat Enterprise Linux as an element of Nutanix Cloud Platform.

May 21, 2024

Nutanix announced Nutanix Kubernetes® Platform (NKP) to simplify management of container-based modern applications using Kubernetes.

May 21, 2024

Octopus Deploy announced their GitHub Copilot Extension that increases efficiency and helps developers stay in the flow.

May 20, 2024

Pegasystems introduced Pega GenAI™ Coach, a generative AI-powered mentor for Pega solutions that proactively advises users to help them achieve optimal outcomes.

May 20, 2024

SmartBear introduces SmartBear HaloAI, trusted AI-driven technology deploying across its entire product portfolio.

May 16, 2024

Pegasystems announced the general availability of Pega Infinity ’24.1™.

May 16, 2024

Mend.io and Sysdig unveiled a joint solution to help developers, DevOps, and security teams accelerate secure software delivery from development to deployment.

May 16, 2024

GitLab announced new innovations in GitLab 17 to streamline how organizations build, test, secure, and deploy software.