The explosion of unstructured and partially structured data has made traditional data lakes harder to manage. Adding to the challenge are “brittle” data pipelines that are time-consuming to create as well as ephemeral.

Or to put it another way, “Pipelines Suck,” asserts autonomous dataflow startup Ascend, which is rolling out a “structured data lake” designed to connect existing data processing engines, business intelligence tools and notebooks on its data management platform. The startup based in Palo Alto, Calif., emerged in July with its dataflow service designed to allow data engineering teams to build and scale Apache Spark-based data pipelines. Ascend claims its service enables pipeline creation with 85 percent less code and reduces the time from prototype to production by 90 percent.

Source: datanami.com
Author: George Leopold