Pipeline, run and data provenance
Support operational oversight, impact analysis and output investigation by linking pipeline definitions, configured jobs and individual runs to execution evidence and qualified lineage assertions.
Bundle → Layer → Finding → Questions Filled
3 bundles · 4 layers · 6 findings · 14 questions
Pipeline definition Separate transformation definitions from configured jobs.
Pipeline and transformations
Describe versioned transformation logic and declared dependencies.
Transformation definition
Identify transformation versions and their declared input and output roles.
- Which versioned transformations does this pipeline definition contain?
- Which inputs and outputs does this transformation definition declare?
Job configuration
Describe a configured execution arrangement for a pipeline.
Job identity and configuration
Identify the job separately from its pipeline and executions.
- Which stable job identifier and pipeline definition version does this job record specify?
- Which configuration version defines this job's parameters, trigger or schedule, execution environment and responsible role?
Pipeline runs Record individual job executions and their output evidence.
Pipeline run
Connect execution state and output provenance to a configured job.
Run record
Retain execution identity, observed results and reproducibility limits.
- Which job identifier and configuration version does this run record reference?
- Which input versions did this run read?
- What reproducibility assessment does this run record provide for its outputs, considering input availability, transformation code, parameters, execution environment and nondeterminism?
- What status, start time, end time when available and external log references does this run record retain?
Manual correction
Preserve manual changes as attributable lineage steps.
- Which output adjustment and responsible person does this manual correction record identify?
- Which separate lineage step represents this manual correction?
Data lineage Record the evidence and limits of dependency assertions.
Lineage assertions
Qualify asserted dependencies by source, granularity and completeness.
Lineage source
Identify how a lineage assertion was established.
- Was this lineage assertion parsed from code, observed at runtime or declared by a person?
- At which dataset or field granularity does this lineage assertion hold?
Unknown inputs
Preserve uncertainty where dependency evidence is incomplete.
- Which input dependencies remain unresolved in this output provenance record?
- What completeness status and missing links does this lineage assessment record?
Classifiers Filled
- Family
- Enterprise profiles
- Category
- Enterprise subject
- Entry kind
- subject
- Domain
- EnterpriseDAT
- Industry
- Cross-industry
- Tags
- EM-DAT-03W2subjectDataPipelinePipelineRunTransformationLineageAssertion
What it is Filled
Transformation definition, job, run, inputs and outputs, lineage at the dataset and field level. High-volume logs remain in external storage.
Why it exists Filled
Support operational oversight, impact analysis and output investigation by linking pipeline definitions, configured jobs and individual runs to execution evidence and qualified lineage assertions.
Distinguishing features Filled
- A pipeline run pins the input versions it read, unlike the pipeline definition it executes.
- Every lineage assertion states its source and level of detail.
- Missing lineage means unknown dependency, not proven independence.
- Distinct from the dataset record, which describes data rather than how it was produced.
What robots and AI may and may not do Filled
Must not
- Treat matching column names as a proven transformation between datasets.
- Present a partial lineage chain as complete.
- Apply manual corrections to outputs without recording them as a step.
- Copy high-volume run logs into the catalogue instead of referencing them.
Only with a human decision
- Approving a manual correction to a published output.
- Accepting lineage gaps for data used in regulatory reporting.
May
- Record runs with input versions, outputs and parameters.
- Assemble lineage chains and mark which links are proven and which are inferred.
- Flag outputs that depend on manual corrections or unknown inputs.
Moral aspects Filled
- False lineage can assign blame or trust to the wrong source.
- Regulated reports require that people can account for where figures came from.
Who is affected
- Data consumers and report readers
- Data engineers
- Auditors and regulators
Owners Filled
Steward
The data owner or head of analytics who answers for data assets and their fitness for use.
Master systems
- Data catalogue
- Data warehouse
- BI platform
- Data orchestrator
Links to other meta-models Filled
neighbor
- EM-DAT-01
references
- WM-DAT-005 - conceptual-candidate
- WM-DAT-006 - conceptual-candidate
What else AI and robots need to interact with it Filled
Identity and identifiers required Filled
- A pipeline definition has an identifier and version.
- A job has a separate stable identifier and versioned configuration referencing a pipeline definition.
- Each run has its own identifier and references the job and configuration version it executed.
Direct properties not applicable Not applicable
Not applicable
Enterprise record contour: physical properties belong to referenced world models.
Recognition required Filled
- A job record describes a configured execution arrangement for a pipeline.
- A run record names its job configuration, input versions, outputs, status, execution times and external log references.
- A run is distinguished from both the configured job and the pipeline definition.
Capabilities and actions required Filled
- Recorded lineage supports tracing outputs to inputs and transformations within the limits of its evidence.
- Output reproducibility depends on available input versions, transformation code, parameters, execution environment and the effects of nondeterminism.
- A reproducibility assessment states the required result fidelity and supporting verification evidence.
Hazards and failure modes required Filled
- Name matching can create a false impression of complete lineage.
- Untracked manual edits can break output provenance and reproducibility.
- Unrecorded external inputs can conceal dependencies.
- Confusing job configuration with run evidence can misrepresent what an execution actually did.
Standards and interfaces optional Filled
- PROV-O from the World Wide Web Consortium supports provenance of runs and data.
- OpenLineage from the LF AI and Data Foundation supports lineage events.
Context of use required Filled
- The record supports data warehouses, analytics platforms and machine learning feature pipelines.
- Financial and regulatory reporting relies on traceable data lineage.
Sources Filled
- W3C DCAT: dataset, distribution, catalog
- W3C PROV and Data Cube: provenance and observation
- SDMX and BI/data catalog practice: indicators, breakdowns, definitions and releases
- W3C Data Catalog Vocabulary (DCAT) Version 3 - Checked live (HTTP 200) on 2026-10-07.
- W3C PROV-DM: The PROV Data Model - Checked live (HTTP 200) on 2026-10-07.
- W3C PROV-O: The PROV Ontology - Checked live (HTTP 200) on 2026-10-07.
- W3C The RDF Data Cube Vocabulary - Checked live (HTTP 200) on 2026-10-07.
Open questions
- How to link lineage without false causality?
- How to record a manual correction?
- How to reproduce a specific output?
- Установить границу и решение reuse/extend/new по действующим спецификациям.
- Подтвердить semantic crosswalk, права и source mastership.
- Выбрать immutable refs; провести проверки fixtures до заявления о публикационной готовности.
Machine files
Provenance
enterprise research programme · published-partial
Built from: enterprise/models/em-dat-03/brief.json, ver-cy/world-models/card-supplements/em-dat-03.json
Still in Russian: suggested owner, blocking decisions, vercy candidates. Translate in research/enterprise/i18n/units.en.json.