Apache Spark creator Databricks rolled out a new version of its cloud platform based on Spark that specifically targets data engineering workloads. The company said Wednesday (April 12) its data science platform would enable data engineers to combine SQL, structured streaming, ETL and machine learning workloads running on the cluster-computing framework.
The goal is to accelerate secure deployment of data pipelines in production, the San Francisco-based company said. The data engineering platform also seeks to move Spark deeper into enterprises by delivering what Databricks calls a “unified data analytics platform” that promotes collaboration among data scientists and decision makers. With that in mind, the cloud platform integrates with the company’s data science “workspaces” to streamline the “transition between data engineering and interactive data science workloads.”
Author: George Leopold