Data Lineage

Dataset and distribution node identity

How a lineage node that stands for a dataset or one of its physical distributions is identified, bound to a location-derived name, and resolved to the authoritative dataset record held by WM-DAT-001.

1. Which authoritative master-system identifier identifies the dataset this node stands for, and which system of record issued it? identity
2. What location-derived binding (namespace and name) is recorded for this node, and under which source-type convention was it constructed? interoperability
3. Does this node denote the abstract dataset or one specific distribution or physical materialisation of it? classification
4. How is a node re-resolved when the dataset is renamed, relocated or its namespace convention changes? exception

Field and column node identity

How a field-level lineage node is identified so that column lineage survives renames and schema evolution, given that column lineage facets key on field name while table formats assign immutable numeric field IDs.

1. What stable field identifier does the source system assign to this column, and is it invariant under rename? identity
2. Which field name was observed at capture time, and to which parent dataset node and schema version does that name belong? provenance
3. How are nested, repeated or struct-typed fields addressed within a single field node? composition
4. What happens to existing field-level edges when a column is renamed, retyped or dropped? constraint

Processing activity and agent references

How the activity that performed a derivation and the agent responsible for it are referenced, keeping job and run definition with WM-DAT-005 and agent records with the adopting Dimension's party model.

1. Which processing activity is asserted to have generated the output node, and how is that activity identified? identity
2. Which agent kind bears responsibility for the activity: a person, an organization or a software agent? classification
3. Is the activity node internal to the observed perimeter or an opaque external activity known only by reference? relationship

Declared granularity and coverage perimeter

The explicit declaration of the level at which lineage is asserted and the boundary of what was observed, so that absence of an edge can be distinguished from absence of observation.

1. At which granularity levels is lineage asserted for this subject, and which levels are explicitly not asserted? definition
2. Which systems, zones or hops lie inside the observed perimeter, and which are recorded as opaque or black-box segments? composition
3. Does the absence of an upstream edge for a node mean no derivation exists, or that no observation was made? quality
4. Which manual steps, spreadsheets or user-side workarounds are declared as part of the flow rather than omitted? process

Edge relation typing and identity

Which relation an edge asserts, how the edge is itself identified so it can carry attributes, times and roles, and how directionality and multiplicity are constrained.

1. Which relation does this edge assert between its endpoints, and what is the source of that typing? relationship
2. Is the dependency direct, meaning output values are derived from input values, or indirect, meaning the input influenced the output without contributing values? classification
3. Does this edge carry its own identifier and attributes, or is it an unqualified binary link? identity
4. What role did the input node play in the activity that produced the output? authority

Transformation characterisation and logic evidence

How the transformation behind an edge is described: its typed subtype, whether it obfuscates values, and the reference to the expression or mapping that realises it.

1. Which transformation subtype best characterises this edge, and from which controlled vocabulary is it drawn? classification
2. Does the transformation mask, hash, pseudonymise or otherwise obfuscate the input values? privacy
3. Where is the executable logic that realises this transformation, and at which version was it observed? evidence
4. Which data-preparation operation class does this transformation fall into for documentation obligations? requirement

Structural consistency constraints

The uniqueness, typing, ordering and impossibility constraints a lineage instance must satisfy, and the normalisation rule that makes two differently expressed instances comparable.

1. Which uniqueness constraints apply, so that the same generation or usage is not asserted twice with conflicting attributes? constraint
2. Which event-ordering constraints must hold between generation, usage, start, end and invalidation on this instance? temporal
3. Which node and edge combinations are declared impossible, including type disjointness and reflexive specialization? validation
4. Are cycles permitted in the derivation graph, and how are recursive or self-referencing derivations represented? constraint

Collection, series and partition lineage

How lineage is asserted for aggregates whose members change over time: dataset series, partitions, file sets and logical collections, and how member-level edges roll up to collection-level edges.

1. Is this node a collection, and if so which membership relation defines its members at the asserted time? composition
2. How do member-level derivation edges aggregate into an edge asserted at collection level? relationship
3. How are partition-scoped derivations expressed when only part of an output was regenerated? state
4. When does a new member make a series a new node rather than an update to the existing collection node? lifecycle

Run reference and observation binding

The carried reference to the execution occurrence under which the derivation was observed, including the run identifier, parent run reference and terminal outcome used only to interpret completeness of the assertion.

1. Under which run identifier was this derivation observed, and which model owns that run record? provenance
2. What terminal outcome did the referenced run report, and how does that outcome qualify the completeness of this assertion? state
3. Is this assertion part of a nested run hierarchy, and which parent run does it belong to? composition
4. For a continuous or streaming activity with no terminal state, what bounded observation window does this assertion cover? temporal

Plan, code and configuration provenance

The version-pinned description of the logic and configuration that governed the derivation, sufficient to explain and re-derive it, expressed as references to code, parameters and engine identity.

1. Which plan or program governed the activity, and at which immutable version was it observed? provenance
2. Which externally supplied parameters and which platform-internal settings influenced the derivation? process
3. Which processing engine and resolved dependencies were in effect, and how are they identified? interoperability
4. Is the recorded plan sufficient to re-derive the output, and what is explicitly not captured? quality

Attribution, association and delegation

Who is responsible for the derived entity, who was associated with the activity, and on whose behalf a software agent acted, keeping party records in the identity model.

1. To which agent is the derived entity attributed, and on what basis was that attribution made? ownership
2. Which agents were associated with the activity, and in which roles? authority
3. When a service account or automated agent acted, on whose behalf did it act and under which authorisation? authority
4. Does attribution transfer to downstream derived entities, or must it be re-asserted at each hop? relationship

Lineage stewardship and accuracy attestation

Named accountability for the lineage record itself: who maintains it, who attests that it reflects the real flow, and on what cadence that attestation is refreshed.

1. Which named role is accountable for the accuracy of this lineage record, distinct from the owner of the underlying data? ownership
2. On what cadence, and against which evidence, is the lineage record attested as reflecting the actual flow? process
3. What escalation applies when the accountable role rejects or cannot confirm the recorded lineage? exception

Event time versus observation time

The separation of when the derivation occurred from when the assertion about it was captured and recorded, plus nominal or scheduled time where it differs from both.

1. At what event time did the derivation occur, expressed with seconds and an explicit offset? temporal
2. At what observation or ingestion time was this assertion captured and recorded, and by which producer? provenance
3. What nominal or logical time does the processed data represent, where that differs from wall-clock event time? temporal
4. How are late-arriving or out-of-order assertions ordered and merged against already-recorded ones? process

Assertion validity, supersession and retraction

The lifecycle of the lineage assertion as a record: when it becomes effective, when it is superseded by a corrected assertion, and how a retraction is expressed without erasing history.

1. Over which validity window is this assertion held to be true, and what ends that window? lifecycle
2. Which prior assertion does this one supersede, and what was the reason for the correction? provenance
3. How is a retracted assertion represented so that consumers who already read it can detect the retraction? exception
4. Which state may a lineage record occupy, and which transitions are permitted? state

Version and snapshot pinning

How an edge names the specific state of the input and output datasets it relates, using the version or snapshot identity issued by the owning system, plus an integrity digest where available.

1. Which specific dataset state does each endpoint of this edge refer to, and which identifier scheme expresses it? identity
2. What ordering key allows two states of the same dataset to be compared? temporal
3. What content digest or checksum evidences the integrity of the referenced state? evidence
4. If the endpoint state is not addressable, what weaker binding is recorded and how is that weakness declared? quality

Schema and naming change continuity

Preserving lineage continuity when fields are renamed, retyped or dropped, or when a naming convention changes and location-derived node names would otherwise disconnect.

1. Which schema change occurred, and which fields and downstream edges does it affect? event
2. Which identifier remained stable across the change, and which binding had to be re-established? identity
3. What is the risk that a namespace or naming-convention change silently disconnects existing lineage nodes? validation
4. Which downstream consumers must be informed of the change, and by which route? decision

Capture method and producing agent

The method by which the assertion was obtained (runtime observation, static parsing, declaration, inference or manual entry) and the identity and schema version of the producer that emitted it.

1. By which method was this assertion obtained, and what does that method structurally fail to see? provenance
2. Which producer emitted the assertion, and against which immutable schema version? interoperability
3. Is provenance recorded about the lineage record itself, and where is that meta-provenance held? provenance
4. When two producers assert conflicting lineage for the same endpoints, which one prevails? exception

Confidence grading and supporting evidence

The graded confidence attached to an assertion and the evidence references that justify the grade, including explicit flagging of manual or uncontrolled steps.

1. What confidence grade is assigned to this assertion, and against which published grading scale? measurement
2. Which concrete evidence items support the assertion, and are they retrievable? evidence
3. Is any part of the asserted path dependent on a manual workaround or uncontrolled step? quality
4. What would falsify this assertion, and has that test been run? validation

Lineage instance validation

Structural evaluation of a lineage instance against uniqueness, typing, ordering and impossibility constraints, plus referential checks that every node reference resolves.

1. Which constraint profile and schema version was the instance evaluated against? validation
2. Which violations were found, at which severity, and on which nodes or edges? quality
3. Which node references failed to resolve, leaving dangling or orphan nodes? validation
4. Does a failing validation block acceptance of the assertions, and who decides? decision

Coverage measurement and reconciliation

Quantifying how much of the declared perimeter is actually covered by asserted lineage and reconciling asserted paths against independent evidence of the real flow.

1. What proportion of the declared perimeter has asserted lineage at each granularity level? measurement
2. Which asserted paths were reconciled against independent evidence, and which diverged? evidence
3. Which known gaps remain unclosed, and what is the accepted remediation horizon? quality
4. How is coverage reported to the accountable role and to supervisory or assurance consumers? process

Sensitivity of the lineage record

Classification of a lineage record in its own right, recognising that node names, transformation expressions and masking flags can disclose confidential logic or reveal where personal data flows.

1. What sensitivity classification applies to this lineage record, and is it higher or lower than that of the data it describes? classification
2. Which elements of the record could disclose personal data footprints or confidential business logic if released? privacy
3. Which elements must be redacted or generalised in a lower-trust projection of the record? security
4. Does propagating a masking flag along a derivation path change the classification of downstream nodes? constraint

Disclosure scoping and exceptions

The scope levels at which lineage may be disclosed, the default denial posture, and the named exceptions such as regulatory or incident access, with the evaluation decision left to the access model.

1. At which scope may a requester read lineage: whole bundle, layer, individual finding or a specific artifact? access
2. Which named exceptions permit broader disclosure than the default rule, and for how long? exception
3. Which model evaluates and enforces the disclosure decision, and what does this model contribute to it? authority

Retention, tombstoning and disposition

The retention rule for lineage records, the tombstone representation used when a referenced node or record must be removed, and the propagation duty toward downstream copies, with execution owned elsewhere.

1. What retention period applies to this lineage record, and which obligation sets it? retention
2. When a referenced node must be erased, is the lineage record deleted or tombstoned, and what remains readable? lifecycle
3. Which downstream holders of copies or replications must be informed of an erasure, and how are they identified? relationship
4. Which party actually executes the deletion in the storage substrate, and where is that execution recorded? authority

External obligation binding

Which external obligations require lineage for a given subject, what each requires, and which lineage elements satisfy the requirement, recorded as binding parameters rather than as a reproduction of the obligation.

1. Which external obligations require documented provenance for this subject, and under which jurisdiction? requirement
2. Which specific lineage elements are relied on to satisfy each obligation? evidence
3. Does the obligation require a narrative lineage statement, a machine-readable graph, or both? interoperability
4. Does the obligation require recording the original purpose of collection for personal data upstream of a derivation? privacy

Alignment crosswalk and conformance claims

Element-by-element mapping between this model and external provenance vocabularies, with explicit lossiness notes and evidenced conformance claims rather than assumed compatibility.

1. Which external vocabulary elements does each local element map to, and in which direction is the mapping lossless? interoperability
2. Which conformance claims are made against which standard version, and what evidence supports each claim? validation
3. Where do the aligned standards conflict, and which one governs in this Dimension? exception
4. How is a narrative lineage statement reconciled with the machine-readable graph for the same subject? quality

Namespace and extension governance

Rules for allocating namespaces used in node identity and for defining custom extensions, so that identifiers stay stable and extensions do not collide with governed terms.

1. Who allocates a namespace used in node identity, and what makes an allocation immutable once used? authority
2. What prefix and naming rule prevents a custom extension from colliding with governed terms? constraint
3. How is an extension schema version referenced so that a consumer can resolve exactly the version emitted? interoperability
4. What is the promotion path from a local extension to a governed term, and what changes on promotion? lifecycle

Cross-system node resolution and stitching

How nodes asserted independently by different producers are recognised as the same thing or as related specialisations, and what evidence is required before two graphs are joined.

1. On what evidence are two independently asserted nodes judged to denote the same thing? identity
2. Is the relationship between two nodes identity, alternate presentation, or specialisation of a more general node? relationship
3. How are symlink or mirror identifiers for the same physical dataset recorded without creating duplicate nodes? composition
4. What is done when a stitching decision is later found to be wrong? exception

Exchange payload contract and integrity

The serialization-neutral contract for a unit of lineage handed to another system, including merge semantics for partial or accumulative delivery and the integrity evidence that binds payload to producer.

1. What is the minimum element set a payload must carry to be independently interpretable? requirement
2. Is the payload a complete self-contained snapshot or an accumulative increment to be merged? process
3. What integrity evidence binds the payload to its producer and its declared schema version? security
4. How does a consumer detect that a payload it already merged has since been superseded or retracted? state