{"schema":"https://ver.cy/schemas/card/1.0.0","id":"vr.wm-dat-005","code":"wm-dat-005-data-pipeline","url":"https://ver.cy/models/wm-dat-005-data-pipeline/","name":"Data Pipeline","alternateNames":[],"kind":"world-model","status":"published","version":"0.3.0-research.1","language":"en","classifiers":{"family":"World Models","category":"Information and virtual systems","entryKind":"aggregate","plane":"","domain":["INF.DAT.PIP"],"industry":["Cross-industry"],"navPath":"NAV.INF.DAT.PIP","tags":["data","pipeline","inf.dat.pip"],"facets":{}},"whatItIs":"Owns pipeline identity, version and lifecycle; purpose, domain and ownership; component graph, ports and dependency semantics; dataset and data-contract bindings; logic, artifact, configuration, environment and secret references; trigger, schedule and execution policies; intended lineage, validation, quality, observability, security, privacy, resilience and interoperability projections. Workflow, run, task attempt, dataset, data product, lineage event, deployment, scheduler, software, secret, telemetry, incident, evidence, audit and records masters remain external.","purpose":"Represent one governed reusable data pipeline definition whose topology, interfaces, logic references, policies and controls can be understood and projected without absorbing runtime, dataset or lineage-event masters.","scope":{"in":["Pipeline identity, version, ownership, purpose, topology, components, ports, edges, dependencies, contracts, logic references, parameters, configuration and execution policies","Intended lineage, quality, observability, security, privacy, resilience, lifecycle, releases, runtime bindings, run references, catalog views, audit and projections"],"out":["Owning workflow, run, task attempt, dataset, data product, lineage event, deployment, scheduler, software, credential, telemetry, incident, evidence, audit or records masters","Treating intended topology, a completed run or an emitted lineage event as proof of quality, freshness, publication, consumption, compliance or exactly-once behavior","Autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition"],"boundaries":[{"neighbor":"WM-ACT-003 Process / Workflow","distinction":"The unified parent signal supplies a general workflow boundary but grants no inheritance or ownership. The data pipeline specializes data movement and transformation while preserving its own versioned definition."},{"neighbor":"WM-ACT-053 Data Processing Job / Pipeline Run","distinction":"The run master owns a time-bounded execution, attempts, input and output instances, status and runtime events. The pipeline owns reusable intended structure and policy."},{"neighbor":"WM-DAT-006 Data Lineage","distinction":"The lineage master owns cross-system provenance observations and their evidence. The pipeline stores expected derivations and non-owning lineage references."},{"neighbor":"Dataset, data product, contract, deployment, scheduler, software, secret and telemetry","distinction":"These masters own data semantics, product accountability, interface commitments, runtime state, executable artifacts, credentials and observations. The pipeline binds version-qualified references only."},{"neighbor":"BPMN, CWL, Airflow, OpenLineage, PROV, DCAT, DQV, SHACL, JSON Schema, CloudEvents, OpenTelemetry, Kubernetes and OCI","distinction":"These standards and platforms have distinct scopes. Every mapping is version-pinned, profile-qualified and loss-declaring, never presumed universal or lossless."}]},"distinguishingFeatures":["A reusable versioned definition of data processing steps, not an individual execution.","Declares ports, data contracts and intended lineage before any run happens.","Differs from a general business process or workflow by binding datasets and data contracts.","Static lineage in the definition is intent, while runtime lineage comes from runs."],"structure":{"bundles":[{"id":"pipeline-identity-scope-ownership-and-definition","name":"Pipeline identity, scope, ownership and definition","description":"Groups governed pipeline context for pipeline identity, scope, ownership and definition.","layers":[{"id":"pipeline-root-version-state-and-namespace","name":"Pipeline root, version, state and namespace","description":"Groups governed pipeline context for pipeline root, version, state and namespace.","findings":[{"id":"pipeline-identity-type-version-current-head-and-revision","name":"Pipeline identity, type, version, current head and revision","description":"Records pipeline identity, type, version, current head and revision as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish pipeline identity, type, version, current head and revision?","id":"pipeline-identity-type-version-current-head-and-revision-q01","kind":"identity"},{"text":"Who declared, owns, approves, operates, observes or may rely on pipeline identity, type, version, current head and revision, under which authority and limits?","id":"pipeline-identity-type-version-current-head-and-revision-q02","kind":"event"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify pipeline identity, type, version, current head and revision?","id":"pipeline-identity-type-version-current-head-and-revision-q03","kind":"process"}]},{"id":"namespace-domain-data-product-purpose-criticality-and-classification","name":"Namespace, domain, data product, purpose, criticality and classification","description":"Records namespace, domain, data product, purpose, criticality and classification as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish namespace, domain, data product, purpose, criticality and classification?","id":"namespace-domain-data-product-purpose-criticality-and-classification-q01","kind":"classification"},{"text":"Who declared, owns, approves, operates, observes or may rely on namespace, domain, data product, purpose, criticality and classification, under which authority and limits?","id":"namespace-domain-data-product-purpose-criticality-and-classification-q02","kind":"temporal"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify namespace, domain, data product, purpose, criticality and classification?","id":"namespace-domain-data-product-purpose-criticality-and-classification-q03","kind":"validation"}]}]},{"id":"ownership-stakeholders-and-definition-boundaries","name":"Ownership, stakeholders and definition boundaries","description":"Groups governed pipeline context for ownership, stakeholders and definition boundaries.","findings":[{"id":"owner-steward-author-operator-approver-consumer-and-contact","name":"Owner, steward, author, operator, approver, consumer and contact","description":"Records owner, steward, author, operator, approver, consumer and contact as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish owner, steward, author, operator, approver, consumer and contact?","id":"owner-steward-author-operator-approver-consumer-and-contact-q01","kind":"ownership"},{"text":"Who declared, owns, approves, operates, observes or may rely on owner, steward, author, operator, approver, consumer and contact, under which authority and limits?","id":"owner-steward-author-operator-approver-consumer-and-contact-q02","kind":"composition"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify owner, steward, author, operator, approver, consumer and contact?","id":"owner-steward-author-operator-approver-consumer-and-contact-q03","kind":"privacy"}]},{"id":"workflow-release-deployment-scheduler-run-and-task-boundary","name":"Workflow, release, deployment, scheduler, run and task boundary","description":"Records workflow, release, deployment, scheduler, run and task boundary as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish workflow, release, deployment, scheduler, run and task boundary?","id":"workflow-release-deployment-scheduler-run-and-task-boundary-q01","kind":"relationship"},{"text":"Who declared, owns, approves, operates, observes or may rely on workflow, release, deployment, scheduler, run and task boundary, under which authority and limits?","id":"workflow-release-deployment-scheduler-run-and-task-boundary-q02","kind":"evidence"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify workflow, release, deployment, scheduler, run and task boundary?","id":"workflow-release-deployment-scheduler-run-and-task-boundary-q03","kind":"lifecycle"}]}]}]},{"id":"topology-components-ports-and-data-contracts","name":"Topology, components, ports and data contracts","description":"Groups governed pipeline context for topology, components, ports and data contracts.","layers":[{"id":"nodes-edges-dependencies-and-control-flow","name":"Nodes, edges, dependencies and control flow","description":"Groups governed pipeline context for nodes, edges, dependencies and control flow.","findings":[{"id":"source-transform-sink-gateway-subpipeline-and-component-node","name":"Source, transform, sink, gateway, subpipeline and component node","description":"Records source, transform, sink, gateway, subpipeline and component node as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish source, transform, sink, gateway, subpipeline and component node?","id":"source-transform-sink-gateway-subpipeline-and-component-node-q01","kind":"composition"},{"text":"Who declared, owns, approves, operates, observes or may rely on source, transform, sink, gateway, subpipeline and component node, under which authority and limits?","id":"source-transform-sink-gateway-subpipeline-and-component-node-q02","kind":"ownership"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify source, transform, sink, gateway, subpipeline and component node?","id":"source-transform-sink-gateway-subpipeline-and-component-node-q03","kind":"quality"}]},{"id":"data-control-dependency-condition-order-cycle-fan-in-and-fan-out-edge","name":"Data, control, dependency, condition, order, cycle, fan-in and fan-out edge","description":"Records data, control, dependency, condition, order, cycle, fan-in and fan-out edge as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish data, control, dependency, condition, order, cycle, fan-in and fan-out edge?","id":"data-control-dependency-condition-order-cycle-fan-in-and-fan-out-edge-q01","kind":"relationship"},{"text":"Who declared, owns, approves, operates, observes or may rely on data, control, dependency, condition, order, cycle, fan-in and fan-out edge, under which authority and limits?","id":"data-control-dependency-condition-order-cycle-fan-in-and-fan-out-edge-q02","kind":"measurement"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify data, control, dependency, condition, order, cycle, fan-in and fan-out edge?","id":"data-control-dependency-condition-order-cycle-fan-in-and-fan-out-edge-q03","kind":"security"}]}]},{"id":"ports-datasets-streams-schemas-and-contracts","name":"Ports, datasets, streams, schemas and contracts","description":"Groups governed pipeline context for ports, datasets, streams, schemas and contracts.","findings":[{"id":"input-output-port-dataset-stream-table-topic-object-and-format","name":"Input or output port, dataset, stream, table, topic, object and format","description":"Records input or output port, dataset, stream, table, topic, object and format as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish input or output port, dataset, stream, table, topic, object and format?","id":"input-output-port-dataset-stream-table-topic-object-and-format-q01","kind":"interoperability"},{"text":"Who declared, owns, approves, operates, observes or may rely on input or output port, dataset, stream, table, topic, object and format, under which authority and limits?","id":"input-output-port-dataset-stream-table-topic-object-and-format-q02","kind":"exception"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify input or output port, dataset, stream, table, topic, object and format?","id":"input-output-port-dataset-stream-table-topic-object-and-format-q03","kind":"retention"}]},{"id":"data-contract-key-type-nullability-partition-order-compatibility-and-sla","name":"Data contract, key, type, nullability, partition, order, compatibility and SLA","description":"Records data contract, key, type, nullability, partition, order, compatibility and sla as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish data contract, key, type, nullability, partition, order, compatibility and sla?","id":"data-contract-key-type-nullability-partition-order-compatibility-and-sla-q01","kind":"constraint"},{"text":"Who declared, owns, approves, operates, observes or may rely on data contract, key, type, nullability, partition, order, compatibility and sla, under which authority and limits?","id":"data-contract-key-type-nullability-partition-order-compatibility-and-sla-q02","kind":"provenance"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify data contract, key, type, nullability, partition, order, compatibility and sla?","id":"data-contract-key-type-nullability-partition-order-compatibility-and-sla-q03","kind":"interoperability"}]}]}]},{"id":"logic-configuration-schedule-and-execution-policy","name":"Logic, configuration, schedule and execution policy","description":"Groups governed pipeline context for logic, configuration, schedule and execution policy.","layers":[{"id":"logic-artifacts-parameters-and-runtime-references","name":"Logic artifacts, parameters and runtime references","description":"Groups governed pipeline context for logic artifacts, parameters and runtime references.","findings":[{"id":"code-query-notebook-container-function-artifact-version-and-digest","name":"Code, query, notebook, container, function, artifact, version and digest","description":"Records code, query, notebook, container, function, artifact, version and digest as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish code, query, notebook, container, function, artifact, version and digest?","id":"code-query-notebook-container-function-artifact-version-and-digest-q01","kind":"provenance"},{"text":"Who declared, owns, approves, operates, observes or may rely on code, query, notebook, container, function, artifact, version and digest, under which authority and limits?","id":"code-query-notebook-container-function-artifact-version-and-digest-q02","kind":"process"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify code, query, notebook, container, function, artifact, version and digest?","id":"code-query-notebook-container-function-artifact-version-and-digest-q03","kind":"decision"}]},{"id":"parameter-configuration-dependency-environment-secret-and-credential-reference","name":"Parameter, configuration, dependency, environment, secret and credential reference","description":"Records parameter, configuration, dependency, environment, secret and credential reference as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish parameter, configuration, dependency, environment, secret and credential reference?","id":"parameter-configuration-dependency-environment-secret-and-credential-reference-q01","kind":"security"},{"text":"Who declared, owns, approves, operates, observes or may rely on parameter, configuration, dependency, environment, secret and credential reference, under which authority and limits?","id":"parameter-configuration-dependency-environment-secret-and-credential-reference-q02","kind":"validation"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify parameter, configuration, dependency, environment, secret and credential reference?","id":"parameter-configuration-dependency-environment-secret-and-credential-reference-q03","kind":"state"}]}]},{"id":"triggers-schedules-and-execution-semantics","name":"Triggers, schedules and execution semantics","description":"Groups governed pipeline context for triggers, schedules and execution semantics.","findings":[{"id":"manual-time-event-dependency-trigger-schedule-timezone-and-data-interval","name":"Manual, time, event or dependency trigger, schedule, timezone and data interval","description":"Records manual, time, event or dependency trigger, schedule, timezone and data interval as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish manual, time, event or dependency trigger, schedule, timezone and data interval?","id":"manual-time-event-dependency-trigger-schedule-timezone-and-data-interval-q01","kind":"temporal"},{"text":"Who declared, owns, approves, operates, observes or may rely on manual, time, event or dependency trigger, schedule, timezone and data interval, under which authority and limits?","id":"manual-time-event-dependency-trigger-schedule-timezone-and-data-interval-q02","kind":"privacy"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify manual, time, event or dependency trigger, schedule, timezone and data interval?","id":"manual-time-event-dependency-trigger-schedule-timezone-and-data-interval-q03","kind":"identity"}]},{"id":"retry-timeout-concurrency-idempotency-checkpoint-backfill-replay-and-catchup","name":"Retry, timeout, concurrency, idempotency, checkpoint, backfill, replay and catchup","description":"Records retry, timeout, concurrency, idempotency, checkpoint, backfill, replay and catchup as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish retry, timeout, concurrency, idempotency, checkpoint, backfill, replay and catchup?","id":"retry-timeout-concurrency-idempotency-checkpoint-backfill-replay-and-catchup-q01","kind":"process"},{"text":"Who declared, owns, approves, operates, observes or may rely on retry, timeout, concurrency, idempotency, checkpoint, backfill, replay and catchup, under which authority and limits?","id":"retry-timeout-concurrency-idempotency-checkpoint-backfill-replay-and-catchup-q02","kind":"lifecycle"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify retry, timeout, concurrency, idempotency, checkpoint, backfill, replay and catchup?","id":"retry-timeout-concurrency-idempotency-checkpoint-backfill-replay-and-catchup-q03","kind":"classification"}]}]}]},{"id":"lineage-quality-observability-and-data-products","name":"Lineage, quality, observability and data products","description":"Groups governed pipeline context for lineage, quality, observability and data products.","layers":[{"id":"intended-lineage-provenance-and-runtime-observation","name":"Intended lineage, provenance and runtime observation","description":"Groups governed pipeline context for intended lineage, provenance and runtime observation.","findings":[{"id":"expected-input-output-field-mapping-transform-semantics-and-derivation","name":"Expected input, output, field mapping, transform semantics and derivation","description":"Records expected input, output, field mapping, transform semantics and derivation as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish expected input, output, field mapping, transform semantics and derivation?","id":"expected-input-output-field-mapping-transform-semantics-and-derivation-q01","kind":"provenance"},{"text":"Who declared, owns, approves, operates, observes or may rely on expected input, output, field mapping, transform semantics and derivation, under which authority and limits?","id":"expected-input-output-field-mapping-transform-semantics-and-derivation-q02","kind":"quality"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify expected input, output, field mapping, transform semantics and derivation?","id":"expected-input-output-field-mapping-transform-semantics-and-derivation-q03","kind":"relationship"}]},{"id":"runtime-lineage-event-source-version-confidence-gap-and-current-view","name":"Runtime lineage event, source, version, confidence, gap and current view","description":"Records runtime lineage event, source, version, confidence, gap and current view as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish runtime lineage event, source, version, confidence, gap and current view?","id":"runtime-lineage-event-source-version-confidence-gap-and-current-view-q01","kind":"evidence"},{"text":"Who declared, owns, approves, operates, observes or may rely on runtime lineage event, source, version, confidence, gap and current view, under which authority and limits?","id":"runtime-lineage-event-source-version-confidence-gap-and-current-view-q02","kind":"security"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify runtime lineage event, source, version, confidence, gap and current view?","id":"runtime-lineage-event-source-version-confidence-gap-and-current-view-q03","kind":"authority"}]}]},{"id":"quality-rules-slos-telemetry-and-cost","name":"Quality rules, SLOs, telemetry and cost","description":"Groups governed pipeline context for quality rules, slos, telemetry and cost.","findings":[{"id":"validation-rule-dimension-threshold-quarantine-acceptance-and-exception","name":"Validation rule, dimension, threshold, quarantine, acceptance and exception","description":"Records validation rule, dimension, threshold, quarantine, acceptance and exception as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish validation rule, dimension, threshold, quarantine, acceptance and exception?","id":"validation-rule-dimension-threshold-quarantine-acceptance-and-exception-q01","kind":"quality"},{"text":"Who declared, owns, approves, operates, observes or may rely on validation rule, dimension, threshold, quarantine, acceptance and exception, under which authority and limits?","id":"validation-rule-dimension-threshold-quarantine-acceptance-and-exception-q02","kind":"retention"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify validation rule, dimension, threshold, quarantine, acceptance and exception?","id":"validation-rule-dimension-threshold-quarantine-acceptance-and-exception-q03","kind":"requirement"}]},{"id":"log-metric-trace-freshness-latency-volume-error-cost-and-slo","name":"Log, metric, trace, freshness, latency, volume, error, cost and SLO","description":"Records log, metric, trace, freshness, latency, volume, error, cost and slo as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish log, metric, trace, freshness, latency, volume, error, cost and slo?","id":"log-metric-trace-freshness-latency-volume-error-cost-and-slo-q01","kind":"measurement"},{"text":"Who declared, owns, approves, operates, observes or may rely on log, metric, trace, freshness, latency, volume, error, cost and slo, under which authority and limits?","id":"log-metric-trace-freshness-latency-volume-error-cost-and-slo-q02","kind":"interoperability"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify log, metric, trace, freshness, latency, volume, error, cost and slo?","id":"log-metric-trace-freshness-latency-volume-error-cost-and-slo-q03","kind":"constraint"}]}]}]},{"id":"security-privacy-resilience-and-compliance","name":"Security, privacy, resilience and compliance","description":"Groups governed pipeline context for security, privacy, resilience and compliance.","layers":[{"id":"identity-access-secrets-zones-and-protection","name":"Identity, access, secrets, zones and protection","description":"Groups governed pipeline context for identity, access, secrets, zones and protection.","findings":[{"id":"service-identity-permission-data-zone-encryption-network-and-least-privilege","name":"Service identity, permission, data zone, encryption, network and least privilege","description":"Records service identity, permission, data zone, encryption, network and least privilege as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish service identity, permission, data zone, encryption, network and least privilege?","id":"service-identity-permission-data-zone-encryption-network-and-least-privilege-q01","kind":"security"},{"text":"Who declared, owns, approves, operates, observes or may rely on service identity, permission, data zone, encryption, network and least privilege, under which authority and limits?","id":"service-identity-permission-data-zone-encryption-network-and-least-privilege-q02","kind":"decision"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify service identity, permission, data zone, encryption, network and least privilege?","id":"service-identity-permission-data-zone-encryption-network-and-least-privilege-q03","kind":"event"}]},{"id":"secret-credential-key-reference-rotation-separation-and-exception","name":"Secret, credential, key reference, rotation, separation and exception","description":"Records secret, credential, key reference, rotation, separation and exception as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish secret, credential, key reference, rotation, separation and exception?","id":"secret-credential-key-reference-rotation-separation-and-exception-q01","kind":"security"},{"text":"Who declared, owns, approves, operates, observes or may rely on secret, credential, key reference, rotation, separation and exception, under which authority and limits?","id":"secret-credential-key-reference-rotation-separation-and-exception-q02","kind":"state"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify secret, credential, key reference, rotation, separation and exception?","id":"secret-credential-key-reference-rotation-separation-and-exception-q03","kind":"temporal"}]}]},{"id":"failure-recovery-rights-retention-and-obligations","name":"Failure, recovery, rights, retention and obligations","description":"Groups governed pipeline context for failure, recovery, rights, retention and obligations.","findings":[{"id":"failure-mode-detection-rollback-recovery-continuity-disaster-and-residual-risk","name":"Failure mode, detection, rollback, recovery, continuity, disaster and residual risk","description":"Records failure mode, detection, rollback, recovery, continuity, disaster and residual risk as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish failure mode, detection, rollback, recovery, continuity, disaster and residual risk?","id":"failure-mode-detection-rollback-recovery-continuity-disaster-and-residual-risk-q01","kind":"requirement"},{"text":"Who declared, owns, approves, operates, observes or may rely on failure mode, detection, rollback, recovery, continuity, disaster and residual risk, under which authority and limits?","id":"failure-mode-detection-rollback-recovery-continuity-disaster-and-residual-risk-q02","kind":"identity"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify failure mode, detection, rollback, recovery, continuity, disaster and residual risk?","id":"failure-mode-detection-rollback-recovery-continuity-disaster-and-residual-risk-q03","kind":"composition"}]},{"id":"purpose-rights-minimization-residency-retention-deletion-consent-and-legal-hold","name":"Purpose, rights, minimization, residency, retention, deletion, consent and legal hold","description":"Records purpose, rights, minimization, residency, retention, deletion, consent and legal hold as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish purpose, rights, minimization, residency, retention, deletion, consent and legal hold?","id":"purpose-rights-minimization-residency-retention-deletion-consent-and-legal-hold-q01","kind":"privacy"},{"text":"Who declared, owns, approves, operates, observes or may rely on purpose, rights, minimization, residency, retention, deletion, consent and legal hold, under which authority and limits?","id":"purpose-rights-minimization-residency-retention-deletion-consent-and-legal-hold-q02","kind":"classification"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify purpose, rights, minimization, residency, retention, deletion, consent and legal hold?","id":"purpose-rights-minimization-residency-retention-deletion-consent-and-legal-hold-q03","kind":"evidence"}]}]}]},{"id":"lifecycle-governance-runs-and-projections","name":"Lifecycle, governance, runs and projections","description":"Groups governed pipeline context for lifecycle, governance, runs and projections.","layers":[{"id":"release-change-deployment-and-run-linkages","name":"Release, change, deployment and run linkages","description":"Groups governed pipeline context for release, change, deployment and run linkages.","findings":[{"id":"draft-reviewed-approved-published-deployed-active-paused-deprecated-and-retired","name":"Draft, reviewed, approved, published, deployed, active, paused, deprecated and retired","description":"Records draft, reviewed, approved, published, deployed, active, paused, deprecated and retired as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish draft, reviewed, approved, published, deployed, active, paused, deprecated and retired?","id":"draft-reviewed-approved-published-deployed-active-paused-deprecated-and-retired-q01","kind":"lifecycle"},{"text":"Who declared, owns, approves, operates, observes or may rely on draft, reviewed, approved, published, deployed, active, paused, deprecated and retired, under which authority and limits?","id":"draft-reviewed-approved-published-deployed-active-paused-deprecated-and-retired-q02","kind":"relationship"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify draft, reviewed, approved, published, deployed, active, paused, deprecated and retired?","id":"draft-reviewed-approved-published-deployed-active-paused-deprecated-and-retired-q03","kind":"ownership"}]},{"id":"change-impact-compatibility-migration-rollback-deployment-run-incident-and-remediation","name":"Change impact, compatibility, migration, rollback, deployment, run, incident and remediation","description":"Records change impact, compatibility, migration, rollback, deployment, run, incident and remediation as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish change impact, compatibility, migration, rollback, deployment, run, incident and remediation?","id":"change-impact-compatibility-migration-rollback-deployment-run-incident-and-remediation-q01","kind":"lifecycle"},{"text":"Who declared, owns, approves, operates, observes or may rely on change impact, compatibility, migration, rollback, deployment, run, incident and remediation, under which authority and limits?","id":"change-impact-compatibility-migration-rollback-deployment-run-incident-and-remediation-q02","kind":"authority"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify change impact, compatibility, migration, rollback, deployment, run, incident and remediation?","id":"change-impact-compatibility-migration-rollback-deployment-run-incident-and-remediation-q03","kind":"measurement"}]}]},{"id":"catalog-projection-audit-and-records","name":"Catalog, projection, audit and records","description":"Groups governed pipeline context for catalog, projection, audit and records.","findings":[{"id":"catalog-dataset-data-product-owner-status-and-discoverability","name":"Catalog, dataset, data product, owner, status and discoverability","description":"Records catalog, dataset, data product, owner, status and discoverability as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish catalog, dataset, data product, owner, status and discoverability?","id":"catalog-dataset-data-product-owner-status-and-discoverability-q01","kind":"classification"},{"text":"Who declared, owns, approves, operates, observes or may rely on catalog, dataset, data product, owner, status and discoverability, under which authority and limits?","id":"catalog-dataset-data-product-owner-status-and-discoverability-q02","kind":"requirement"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify catalog, dataset, data product, owner, status and discoverability?","id":"catalog-dataset-data-product-owner-status-and-discoverability-q03","kind":"exception"}]},{"id":"bpmn-cwl-airflow-openlineage-prov-dcat-dqv-shacl-cloudevents-and-otel-projection","name":"BPMN, CWL, Airflow, OpenLineage, PROV, DCAT, DQV, SHACL, CloudEvents and OTel projection","description":"Records bpmn, cwl, airflow, openlineage, prov, dcat, dqv, shacl, cloudevents and otel projection as governed pipeline-definition context while deployment, scheduler, run, dataset, lineage-event, secret, telemetry and records masters remain external.","questions":[{"text":"What stable identity, pipeline version, typed value and explicit unknown establish bpmn, cwl, airflow, openlineage, prov, dcat, dqv, shacl, cloudevents and otel projection?","id":"bpmn-cwl-airflow-openlineage-prov-dcat-dqv-shacl-cloudevents-and-otel-projection-q01","kind":"interoperability"},{"text":"Who declared, owns, approves, operates, observes or may rely on bpmn, cwl, airflow, openlineage, prov, dcat, dqv, shacl, cloudevents and otel projection, under which authority and limits?","id":"bpmn-cwl-airflow-openlineage-prov-dcat-dqv-shacl-cloudevents-and-otel-projection-q02","kind":"constraint"},{"text":"Which effective, release, deployment, scheduled, event, recorded, ingested and knowledge times, evidence and uncertainty qualify bpmn, cwl, airflow, openlineage, prov, dcat, dqv, shacl, cloudevents and otel projection?","id":"bpmn-cwl-airflow-openlineage-prov-dcat-dqv-shacl-cloudevents-and-otel-projection-q03","kind":"provenance"}]}]}]}]},"agentConduct":{"may":["Register a pipeline definition with topology and version.","Validate the graph, contracts and controls before release.","Describe the pipeline and its dependencies for impact analysis.","Propose changes as new versions for review."],"mustNot":["Deploy, schedule, execute, backfill or replay the pipeline without delegation.","Rotate credentials or widen data access.","Change a released version in place.","Publish or delete outputs.","Claim exactly-once or deterministic behaviour without scoped evidence."],"requiresHuman":["Releasing or retiring a pipeline version.","Approving schedule or trigger changes for production.","Approving new access to personal or restricted data."]},"ethics":{"considerations":["Pipelines move personal data between systems, so purpose and access controls must travel with the definition.","Hidden transformations can introduce bias or errors into decisions about people.","Wasteful schedules consume compute and energy without value."],"affectedParties":["Data subjects","Downstream consumers of outputs","Data platform operators"]},"owners":{"steward":"Dimension owner, namespace authority, data governance mandate and accountable data product owner","roles":[{"name":"Data product owner","responsibilities":["Own purpose, outcome, criticality, consumers, acceptance and accountable use of the pipeline."]},{"name":"Data steward","responsibilities":["Own dataset meaning, contracts, quality, lineage, classification, rights and retention profiles."]},{"name":"Pipeline author and data engineer","responsibilities":["Maintain topology, transform logic references, parameters, dependencies, tests and versioned change evidence."]},{"name":"Platform operator and SRE","responsibilities":["Maintain deployment, scheduler, runtime, observability, capacity, recovery and incident linkages outside the definition."]},{"name":"Security and privacy reviewer","responsibilities":["Review identity, secrets, access, zones, encryption, purpose, minimization, residency and deletion controls."]},{"name":"Independent reviewer and release authority","responsibilities":["Review topology, contracts, change impact, validation, exceptions, compatibility and release decisions within mandate."]},{"name":"Records and interoperability steward","responsibilities":["Own audit, retention, disposition, source pins, mappings and declared projection loss."]}],"masterSystems":[]},"relations":[{"target":"WM-ACT-003 Process / Workflow","type":"references","note":"Represent the unfrozen parent boundary without inheritance, mutation, execution or cascade authority."},{"target":"WM-ACT-053 Data Processing Job / Pipeline Run","type":"references","note":"Resolve time-bounded executions while preserving separate definition, deployment, run and task identities."},{"target":"WM-DAT-006 Data Lineage","type":"references","note":"Resolve authoritative runtime provenance without absorbing the lineage master."},{"target":"Dataset, Data Product, Data Contract, Deployment, Scheduler, Software, Secret, Telemetry, Incident, Evidence, Audit and Records models","type":"references","note":"Resolve authoritative semantics, state, artifacts, controls and evidence without changing their ownership."},{"target":"BPMN 2.0.2, CWL 1.2.1, Airflow 3.3.0, OpenLineage 1.53.0, PROV-O, DCAT 3, DQV, SHACL, JSON Schema 2020-12, CloudEvents 1.0.2, OpenTelemetry 1.44.0, Kubernetes and OCI 1.1.1","type":"aligned","note":"Project version-pinned workflow, runtime, lineage, catalog, quality, validation, event, telemetry and packaging views with declared loss."},{"target":"WM-ACT-003 Process / Workflow","type":"neighbor","note":"The unified parent signal supplies a general workflow boundary but grants no inheritance or ownership. The data pipeline specializes data movement and transformation while preserving its own versioned definition."},{"target":"WM-ACT-053 Data Processing Job / Pipeline Run","type":"neighbor","note":"The run master owns a time-bounded execution, attempts, input and output instances, status and runtime events. The pipeline owns reusable intended structure and policy."},{"target":"WM-DAT-006 Data Lineage","type":"neighbor","note":"The lineage master owns cross-system provenance observations and their evidence. The pipeline stores expected derivations and non-owning lineage references."},{"target":"Dataset, data product, contract, deployment, scheduler, software, secret and telemetry","type":"neighbor","note":"These masters own data semantics, product accountability, interface commitments, runtime state, executable artifacts, credentials and observations. The pipeline binds version-qualified references only."},{"target":"BPMN, CWL, Airflow, OpenLineage, PROV, DCAT, DQV, SHACL, JSON Schema, CloudEvents, OpenTelemetry, Kubernetes and OCI","type":"neighbor","note":"These standards and platforms have distinct scopes. Every mapping is version-pinned, profile-qualified and loss-declaring, never presumed universal or lossless."},{"target":"WM-ACT-003","type":"parent"},{"target":"WM-ACT-053","type":"contains"}],"interaction":{"identity":{"applicability":"required","items":["Authoritative master-system identifier for each pipeline, version, component, contract, policy, release, binding or projection, qualified by issuer, namespace and record kind.","Governed globally resolvable pipeline IRI.","Dimension UUID or ULID when neither preceding identifier exists."]},"properties":{"applicability":"not-applicable","items":[]},"recognition":{"applicability":"optional","items":["A pipeline definition names inputs, outputs, steps, dependencies, triggers and a version.","Often confused with a pipeline run, a scheduler job entry, a dataset or a code repository."]},"capabilities":{"applicability":"required","items":["Register a pipeline definition: Governed operation to register a pipeline definition without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Compose or import pipeline topology: Governed operation to compose or import pipeline topology without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Bind ports and data contracts: Governed operation to bind ports and data contracts without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Bind logic, configuration and runtime references: Governed operation to bind logic, configuration and runtime references without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Define triggers, schedule and execution policy: Governed operation to define triggers, schedule and execution policy without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Validate graph, contracts and controls before release: Governed operation to validate graph, contracts and controls before release without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Release, deprecate or retire a pipeline definition: Governed operation to release, deprecate or retire a pipeline definition without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Bind a deployment and scheduler registration: Governed operation to bind a deployment and scheduler registration without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Instantiate or reference a pipeline run: Governed operation to instantiate or reference a pipeline run without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition.","Query, project, correct, retain and audit: Governed operation to query, project, correct, retain and audit without autonomous deployment, execution, scheduling, backfill, replay, credential rotation, access widening, data publication, output deletion, incident closure or records disposition."]},"hazards":{"applicability":"required","items":["Breaking change released without contract checks.","Unauthorized data flows across boundaries.","Cascading failures in dependent pipelines."]},"interfaces":{"applicability":"required","items":["Common Workflow Language (CWL).","OpenLineage job facets.","W3C PROV-O and DCAT 3.","W3C SHACL and JSON Schema for data contracts.","CNCF CloudEvents and OpenTelemetry.","OCI image specification."]},"context":{"applicability":"required","items":["Privacy, residency, retention, deletion, consent, access and evidence duties depend on jurisdiction and organization.","CWL, BPMN, Airflow, OpenLineage, Kubernetes and OCI are versioned technical profiles and not assumed universal, binding or lossless.","W3C, JSON Schema, CloudEvents, OpenTelemetry, NIST and RFC sources are versioned profiles whose adoption and stability must be declared."]}},"sources":[{"title":"OpenLineage Object Model","url":"https://openlineage.io/docs/spec/object-model/","note":"OpenLineage"},{"title":"DAGs","url":"https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html","note":"Apache Software Foundation"},{"title":"Common Workflow Language Workflow Description","url":"https://www.commonwl.org/v1.2/Workflow.html","note":"Common Workflow Language Project"},{"title":"Business Process Model and Notation","url":"https://www.omg.org/spec/BPMN/2.0.2/","note":"Object Management Group"},{"title":"PROV-O The PROV Ontology","url":"https://www.w3.org/TR/prov-o/","note":"World Wide Web Consortium"},{"title":"Data Catalog Vocabulary DCAT Version 3","url":"https://www.w3.org/TR/vocab-dcat-3/","note":"World Wide Web Consortium"},{"title":"Data on the Web Best Practices Data Quality Vocabulary","url":"https://www.w3.org/TR/vocab-dqv/","note":"World Wide Web Consortium"},{"title":"Shapes Constraint Language SHACL","url":"https://www.w3.org/TR/shacl/","note":"World Wide Web Consortium"},{"title":"JSON Schema Draft 2020-12","url":"https://json-schema.org/draft/2020-12","note":"JSON Schema Project"},{"title":"CloudEvents Specification","url":"https://github.com/cloudevents/spec","note":"Cloud Native Computing Foundation"},{"title":"OpenTelemetry Semantic Conventions","url":"https://opentelemetry.io/docs/specs/semconv/","note":"OpenTelemetry"},{"title":"Jobs","url":"https://kubernetes.io/docs/concepts/workloads/controllers/job/","note":"Kubernetes"},{"title":"OCI Image Format Specification","url":"https://github.com/opencontainers/image-spec","note":"Open Container Initiative"},{"title":"Secure Software Development Framework Version 1.1","url":"https://csrc.nist.gov/pubs/sp/800/218/final","note":"National Institute of Standards and Technology"},{"title":"Time Ontology in OWL","url":"https://www.w3.org/TR/owl-time/","note":"World Wide Web Consortium"},{"title":"Date and Time on the Internet Timestamps","url":"https://www.rfc-editor.org/info/rfc3339/","note":"Internet Engineering Task Force"}],"openQuestions":["Approve or reject the WM-ACT-003, WM-ACT-053 and WM-DAT-006 relationships and register Dataset, Data Product, Data Contract, Deployment, Scheduler, Software, Secret, Telemetry, Incident, Evidence, Audit and Records edges.","Create orchestrator, runtime, data-platform, database, streaming, transaction, schema-evolution, quality, security, privacy and jurisdiction profiles.","Test idempotency, checkpoint, retry, replay, backfill, ordering and exactly-once claims against concrete engine and storage combinations.","Validate organization-specific access, credential, data-zone, residency, minimization, consent, retention, deletion, incident and recovery policies.","Conformance-test version-pinned BPMN, CWL, Airflow, OpenLineage, PROV, DCAT, DQV, SHACL, JSON Schema, CloudEvents, OpenTelemetry, Kubernetes and OCI mappings and obtain independent external review before canonical promotion.","Claude and Grok each timed out on one bounded attempt; no independent external result was admitted.","The unified parent WM-ACT-003 and candidate relations to WM-ACT-053 and WM-DAT-006 are not approved frozen composition edges.","Engine-specific task types, expression languages, scheduler semantics, exactly-once guarantees and database transaction behavior require separate profiles.","Jurisdiction, organization, data sensitivity, residency, privacy, retention and rights requirements require adopting-Dimension profiles."],"resources":{"spec":"/models/wm-dat-005-data-pipeline/spec.yaml","agents":"/models/wm-dat-005-data-pipeline/AGENTS.md","source":"https://github.com/ver-cy/world-models/tree/feat/mega-model-registry/publications/wm-dat-005-data-pipeline"},"provenance":{"origin":"world-models research","builtFrom":["models/wm-dat-005-data-pipeline/spec.yaml","ver-cy/world-models/card-supplements/wm-dat-005-data-pipeline.json"],"providers":["Codex"],"researchStatus":"reviewable-draft","generatedAt":"2026-09-06T12:38:16Z","builder":"tools/build_cards.py@1.0.0"},"completeness":{"sections":{"classifiers":"filled","whatItIs":"filled","purpose":"filled","distinguishingFeatures":"filled","structure":"filled","agentConduct":"filled","ethics":"filled","owners":"filled","relations":"filled","interaction.identity":"filled","interaction.properties":"not-applicable","interaction.recognition":"filled","interaction.capabilities":"filled","interaction.hazards":"filled","interaction.interfaces":"filled","interaction.context":"filled","sources":"filled"},"notes":{"interaction.properties":"Institutional or informational subject: no invented physical properties.","_supplement":"Sections authored in card supplement 1.0.0 by Claude (Opus 5.5) (2026-10-05, unreviewed). Written from the published specification and established practice in the field; no new sources were read. Unreviewed."},"score":1.0}}