What is a data pipeline?
Last updated
Was this helpful?
Continuum Knowledge
Last updated
Was this helpful?
A data pipeline is a systematic and automated process for the efficient and reliable movement, transformation, and management of data from one point to another within a computing environment. It plays a crucial role in modern data-driven organizations by enabling the seamless flow of information across various stages of data processing.
A data pipeline consists of a series of data processing steps.
If the data is not currently loaded into the data platform, then it is ingested at the beginning of the pipeline. Then there are a series of steps in which each step delivers an output that is the input to the next step.
This continues until the pipeline is complete. In some cases, independent steps may be run in parallel.
source
processing step or steps
a destination.
In some data pipelines, the destination may be called a sink.
Data pipelines enable the flow of data from an application to a data warehouse, from a data lake to an analytics database, or into a system, for example.
Data pipelines also may have the same source and sink, such that the pipeline is purely about modifying the data set. Any time data is processed between point A and point B (or points B, C, and D), there is a data pipeline between those points.
As organisations look to build applications with small code bases that serve a very specific purpose (these types of applications are called “microservices”), they are moving data between more and more applications, making the efficiency of data pipelines a critical consideration in their planning and development.
Data generated in one source system or application may feed multiple data pipelines, and those pipelines may have multiple other pipelines or applications that are dependent on their outputs.
Consider a single comment on social media. This event could generate data to feed a real-time report counting social media mentions, a sentiment analysis application that outputs a positive, negative, or neutral result, or an application charting each mention on a world map.
Though the data is from the same source in all cases, each of these applications are built on unique data pipelines that must smoothly complete before the end user sees the result.
Common steps in data pipelines include data transformation, augmentation, enrichment, filtering, grouping, aggregating, and the running of algorithms against that data.
As the volume, variety, and velocity of data have dramatically grown in recent years, architects and developers have had to adapt to “big data.”
The term “big data” implies that there is a huge volume to deal with. This volume of data can open opportunities for use cases such as predictive analytics, real-time reporting, and alerting, among many examples.
Like many components of data architecture, data pipelines have evolved to support big data. Big data pipelines are data pipelines built to accommodate one or more of the three traits of big data.
The volume of big data requires that data pipelines must be scalable, as the volume can be variable over time. In practice, there are likely to be many big data events that occur simultaneously or very close together, so the big data pipeline must be able to scale to process significant volumes of data concurrently. The variety of big data requires that big data pipelines be able to recognize and process data in many different formats—structured, unstructured, and semi-structured.
Data pipelines automate the flow of data, reducing manual intervention and minimizing the risk of errors. This enhances overall efficiency in data processing workflows.
With the ability to process data in real-time, data pipelines empower organizations to derive insights quickly and make informed decisions on the fly.
Scalable architectures in data pipelines allow organizations to handle growing volumes of data without compromising performance, ensuring adaptability to changing business needs.
By incorporating data cleansing and transformation steps, data pipelines contribute to maintaining high data quality standards, ensuring that the information being processed is accurate and reliable.
Automation and optimization of data processing workflows result in cost savings by reducing manual labour, minimizing errors, and optimizing resource utilization.
Batch processing involves the execution of data jobs at scheduled intervals. It is well-suited for scenarios where data can be processed in non-real-time, allowing for efficient handling of large datasets.
While Data Pipelines and Extract, Transform, Load (ETL) processes share similarities, there are key differences:
Ensuring the security and privacy of sensitive data throughout the pipeline is crucial to compliance with regulations and protecting organizational assets.
The architecture should be designed to scale horizontally or vertically to accommodate growing data volumes and processing demands.
Building in mechanisms to handle failures and errors gracefully is essential for maintaining the reliability of the pipeline.
Effective metadata management is crucial for tracking the lineage and quality of data as it moves through the pipeline.
While there are use cases such as batch processing with relatively long processing windows, many times a data pipeline feeds mission-critical and time-sensitive operations such as payment processing or fraud detection. In those cases, fast performance and low latency are critical for the business to meet their required service level agreements (SLAs).
Data pipelines may be architected in several different ways.
One common example is a batch-based data pipeline. In that example, you may have an application such as a point-of-sale system that generates a large number of data points that you need to push to a data warehouse and an analytics database. Here is an example of what that would look like:
The Lambda Architecture accounts for both a traditional batch data pipeline and a real-time data streaming pipeline. It also has a serving layer that responds to queries.
A recent abstraction for data pipelines comes from an open source project, Apache Beam.
It provides a programmatic approach to creating data pipelines, with the actual implementation of the pipeline depending on the platform on which the pipeline is deployed. Apache Beam provides a unified model for both batch and streaming data processing, providing a portable and extensible approach especially helpful when considering multi-cloud and hybrid cloud deployments.
The velocity of big data makes it appealing to build pipelines for big data. Then data can be captured and processed in real time so some action can then occur.
ETL has historically been used for batch workloads, especially on a large scale. But a new breed of tools are emerging as part of the pipeline for .
Another example is a streaming data pipeline. In a streaming data pipeline, data from the point of sales system would be processed as it is generated. The engine could feed outputs from the pipeline to data stores, marketing applications, and CRMs, among other applications, as well as back to the point of sale system itself.
A third example of a data pipeline is the , which combines batch and streaming pipelines into one architecture. The Lambda Architecture is popular in big data environments because it enables developers to account for both real-time streaming use cases and historical batch analysis. One key aspect of this architecture is that it encourages storing data in raw format so that you can continually run new data pipelines to correct any code errors in prior pipelines, or to create new data destinations that enable new types of queries.
A more modern variant of the Lambda Architecture is the . This is a much simpler architecture because it uses a single stream processing layer for both real-time and batch processing.
