The industry has largely settled on the notion of a data pipeline as a means of encapsulating the engineering work that goes into collecting, transforming, and preparing data for downstream advanced analytics and machine learning workloads. Now the next step forward is to automate that pipeline work, which is a cause that several DataOps vendors are rallying around.

Data engineers are some of the most in-demand people in organizations that are leveraging big data. While data scientists (or machine learning engineers, as many of them are calling themselves nowadays) get most of the glory, it’s the data engineers who do much of the hands-on-keyboard work that makes the magic of data science possible.

Source: datanami.com
Author: Alex Woodie