Scrum for Data Engineering: Building Reliable Data Pipelines Iteratively

Data engineering occupies an interesting position in the agile landscape. Unlike application development, where the user experience is visible and testable within a Sprint, data engineering work often produces infrastructure whose value is not immediately apparent — a pipeline that processes data correctly and reliably looks the same to the end user whether it was built with rigorous engineering discipline or held together with ad hoc workarounds. This invisibility creates organisational pressure to cut corners on data quality, documentation, and testing in favour of delivering features quickly. Scrum, properly adapted for data engineering, provides a structure for resisting that pressure while still delivering value in short cycles.

Data quality as acceptance criteria

The most important adaptation for data engineering is defining data quality explicitly in the Definition of Done and in user story acceptance criteria. In application development, acceptance criteria typically describe user behaviour: "when a user clicks the button, the form submits and a confirmation appears." In data engineering, acceptance criteria must also describe data behaviour: what completeness, accuracy, freshness, and lineage requirements must the delivered pipeline or dataset satisfy? Leaving quality implicit — treating "it works" as sufficient — produces a backlog of silent failures where pipelines technically execute but produce data that is subtly wrong, stale, or incomplete.

1. Completeness. What percentage of expected records must be present? What is the acceptable threshold for missing values in key fields? For a sales transaction pipeline, a completeness criterion might require that 100 per cent of transactions from the source system are present in the target dataset, with no null values in the transaction ID, date, or amount fields.

2. Accuracy. Do the values in the target match the values in the source, applying any agreed transformation rules? Accuracy testing requires defined test cases where the expected output is known, and automated reconciliation between source and target counts, sums, and distributions.

3. Freshness. How current does the data need to be? A near-real-time fraud detection pipeline has very different freshness requirements from a monthly financial reporting dataset. Freshness criteria should be explicit in the acceptance criteria and testable through automated data freshness monitoring.

4. Lineage. Can the origin and transformation history of each data element be traced? Data lineage is not only a quality attribute; it is a governance requirement in regulated environments and an essential debugging tool when downstream consumers report unexpected results. The Definition of Done for any pipeline should include documentation of the lineage from source to target, including all transformations applied.

Adapting the backlog for data work

1. Pipeline failures as defects. When a pipeline fails — incorrect output, dropped records, processing errors — that failure should be captured as a defect in the backlog with the same visibility and prioritisation discipline as any other backlog item. Data engineering teams that handle pipeline failures informally ("we just fixed it, no ticket needed") lose the ability to track failure patterns over time, understand root causes, and demonstrate the quality improvement that engineering investment delivers. Defect tracking is not bureaucracy; it is the feedback loop that enables the team to improve.

2. Schema changes as features requiring Product Owner prioritisation. Schema changes in a data warehouse or data lake are not minor technical tasks — they are changes to the contract between producer and consumer teams that can break downstream reports, dashboards, and models. Schema changes should be managed as backlog items with explicit Product Owner prioritisation, stakeholder communication, and version control governance. Treating them as invisible maintenance work is a common source of data trust failures where downstream teams discover a breaking change through a broken report rather than through advance communication.

3. Data contracts between producer and consumer teams. A data contract is a formal agreement between the team that produces a dataset and the teams that consume it. It specifies the schema, the quality guarantees, the SLA for freshness and availability, and the process for managing breaking changes. Data contracts transform implicit dependencies (consumers assume the producer will not change the schema without warning) into explicit commitments (the producer guarantees specific schema stability for a defined period and follows a defined change management process). Backlog items that affect existing contracts require stakeholder review before Sprint commitment.

Writing user stories for data work

The most effective reframe for data engineering user stories is data product thinking: treating each dataset or pipeline as a product consumed by an internal customer for a specific purpose. The question to anchor the user story is: who consumes this data, and what decision or action does it enable? "As a financial analyst, I need a daily reconciled sales-by-region dataset so that I can produce the weekly regional performance report without manual data collection." That framing produces acceptance criteria that reflect the consumer's actual needs — granularity, freshness, accuracy at the field level, coverage — rather than technical implementation details that mean little to the Product Owner or stakeholders. The Definition of Done for a data pipeline typically includes: data quality acceptance criteria met and automated tests passing, lineage documented, schema changes communicated to affected consumers, monitoring alerts configured, and run history available for troubleshooting.

If your data engineering team is adopting Scrum or finding that the standard framework needs adaptation to fit the realities of pipeline and data warehouse work, XNM's program and project delivery practice works with technical teams to design agile operating models that fit the work, not just the framework.