Your organization’s data is likely buried in many systems and applications that are located across the globe. This distributed raw data is not suitable for analysis; in order to pursue data analysis and reap the rewards of the insights it can provide, your organization should invest in creating a data pipeline.

What is a Data Pipeline?

A data pipeline is a series of processing steps to prepare enterprise data for analysis. It includes various technologies, but the definition does not prescribe any specific technologies, tools or implementation approaches. Data pipeline specifications are driven by the data analytics requirements, governance policies and architecture specifications. A data pipeline can be as simple as loading data from a source system into a data lake or as complex as loading a Medallion data architecture.

Data pipelines historically used extract-transform-load (ETL) processes. These eventually led to extract-load-transform (ELT) processes in the cloud. Today, ETL and ELT patterns are subsets of data pipelines. Data pipelines serve a wider purpose than batch transferring data from a source system to a data warehouse. They handle the full range of structured, semi-structured and unstructured data in both batch and real-time modes. They connect to thousands of data source types, they orchestrate long running task streams, they use large language model (LLM) agents for analysis and transformation and they can interface with real-time event-based data analytics systems for real-time alerts and notifications.

Data Pipeline Functions

Common functions across all these data pipelines are:

• Data pipeline workflow automation (series of processing steps)
• Data ingestion from source systems (enterprise data)
• Data transformation (preparation)
• Data delivery to analytics systems (analysis)
• Data pipeline governance, auditing and monitoring (governance)

Data Pipeline Workflow Automation

Data pipelines are a series of processing steps. Automation and governance procedures for repetitive workstreams ensure that the data is cleansed and transformed consistently. These steps can be governed, organized and executed using any combination of notebooks, schedulers, low-code and no-code tools, custom code and workflow and automation tools.

The complexity of the toolset that is required will depend upon the complexity of the data pipeline. Complex data pipelines require long-running state management and event-based sub-process coordination.

Data Ingestion from Source Systems

A data source can be an application, a device or another database. Data can be pushed from the data source to the data pipeline or pulled from the data source by the data pipeline. It can be ingested in batches or streamed in real-time.

Important considerations include identifying how to securely connect to the source data, determining data source formats, determining how changes in the source data are captured and determining how to manage the changed data.

Data Transformation

Data transformation is an iterative process of operations that verifies, validates, deduplicates, filters, enriches, augments, sorts, groups, aggregates and transforms the source data into an analytics-ready format. The data transformation process is responsible for the quality of all data persisted or delivered to a data analytics system.

There are many toolsets that can be used to implement the data transformation process. To ensure return on investment (ROI), it is important to understand your requirements. Licensing a fully featured tool to load comma-separated files is overkill.

Data Delivery to Analytics Systems

The goal of the data pipeline is to deliver the appropriate data for analysis. The content, format and destination of the data is determined by the type of data analytics that will be performed. The destination may be a data lake, a data warehouse, a data lake house, a data stream or directly to the data analytics application. Like data ingestion, important considerations include identifying how to securely connect to the destination and determining data destination formats.

Data Pipeline Governance, Auditing and Monitoring

Data pipeline governance is responsible for the full lifecycle of the data pipelines. The data engineering team responsible for developing the data pipelines works with the enterprise architecture, data architecture, data science, data governance and information security teams to ensure organizational compliance.

This coordination is important to ensure data quality, maintain data security and privacy, reduce integration complexity, ensure maintainability and meet availability and deliverability requirements. Data catalogs and lineage models link the raw data to its business context and make it analytically meaningful. They also define data quality metrics and identify sensitive data that needs to be masked and access controlled.

In the past, software engineering teams and data teams haven’t collaborated, but data pipelines and AI integrations are quickly becoming critical software components. To successfully work together, data pipelines must follow standard DevSecOps practices.

Data Management Community

Defining, implementing and supporting data pipelines takes a community of data-minded individuals. These individuals span both the business and technology groups and fill many roles. Some of these roles are data architect, data engineer, data scientist, data analyst, data owner, data steward, data custodian and data users.

Data architects are responsible for the overall design pattern that specifies the models, policies, rules and standards governing the collection, arrangement, integration, storage and utilization of data elements within an application. They establish the interrelationships between major data components to ensure effective information storage.

Data engineers are responsible for organizing, managing and analyzing large amounts of data. They are concerned with designing, constructing and maintaining the support systems and infrastructures required to support the organization’s data management goals and requirements that are defined by the data architecture.

Data analysts and data scientists study the organization’s data to derive useful insights for business decision making. They utilize math, computer science and domain expertise. Data analysts and data scientists use the data engineering infrastructure to create ad-hoc pipelines as part of their data collection and preparation processes.

Data owners determine the business definition of the data and grant access it.

Data stewards ensure the quality, accuracy and compliance of the data. They are responsible for data profiling, metadata management and monitoring quality. They authorize and monitor the secure use of data to ensure appropriate access, accuracy, classification, privacy and security.

Data custodians are responsible for the operation and management of the technology and systems that collect, store, process, manage and provide access to company data. They include system administrators.

Data users are authorized individuals who have been granted access to data to perform assigned duties or functions within the company. When individuals become data users, they assume responsibility for the appropriate use, management and application of privacy and security standards for the data they are authorized to use.

We Can Help

Data pipelines are the backbone of any successful data analytics initiative. By automating workflows, ensuring secure ingestion, implementing robust transformations and delivering data ready for analysis, they enable organizations to unlock the full potential of their data assets.

However, building and managing these pipelines can be overwhelming, especially for small teams juggling multiple priorities. That’s where ProBridge comes in. With our expertise in designing scalable, secure and efficient data pipelines, we can help you navigate the complexities and achieve your analytics goals. Reach out to us to explore how we can support your journey toward data analytics excellence.

Bill Fogg is the Assistant Vice President of Enterprise Architecture at ProBridge.