Ablation Study

Study identity and accountable scope

One persistent study record binds its purpose and accountable role across protocol revisions. A run, report or reusable experimental recipe has a different identity.

1. Which authoritative study identifier and revision distinguish this study from a rerun or replication? identity
2. Which accountable study role owns the record and which evidence custodian maintains its references? ownership
3. What precise factor contribution is being investigated on which target population? definition
4. Is this a planned, exploratory or confirmatory study and how was that classification determined? classification

Frozen comparison intent

A frozen protocol revision states intended contrasts, metrics and decision rules. Amendments remain visible and post-result choices cannot be relabelled as prespecified.

1. When were hypotheses and analysis rules frozen relative to the first outcome inspection? temporal
2. Which transitions move this study from design through execution, analysis and review to closure or withdrawal? lifecycle
3. Which protocol amendments occurred and which comparisons must be repeated or reclassified? event
4. Who may freeze or amend the protocol and what review is required for an outcome-informed change? authority

With-factor and without-factor arms

Each contrast names a reference arm and an ablated arm. Removal, masking or replacement semantics are explicit; training-time changes and inference-time disabling answer different questions. Incidental changes limit attribution.

1. Which component or configuration factor is removed, masked or replaced in each arm? composition
2. Does the intervention require retraining or only alter inference and what remains fixed? process
3. How is actual intervention compliance checked against intended arm configurations? validation
4. What happens when removing the factor makes an arm non-executable or changes its interface? exception

Confounders and interaction limits

The local design records nuisance variables, budget policy and blocks. A contrast at one background setting estimates only that conditional comparison; multiple removals do not automatically identify independent contributions.

1. Which nuisance factors and execution budgets are controlled, blocked or left uncontrolled? constraint
2. How are run order and repeat allocation determined within declared blocks? process
3. Which other factor settings condition the contrast and which interactions are untested? relationship
4. Could unequal tuning or resource allowances explain the observed difference? quality

Common evaluation cases and labels

Both arms reference the same frozen evaluation case set, split and label revision for a contrast. Input transformation may be the declared intervention while case identity remains stable. Membership drift and training overlap are recorded as threats.

1. Which dataset revision, split and ordered or keyed case membership are common to both arms? identity
2. Which label revision and preprocessing lineage support each evaluation case? provenance
3. What checks identify leakage, duplicate cases or exposure during tuning and training? quality
4. How are case omissions or label corrections handled without silently changing the paired cohort? exception

One scorer definition per contrast

A contrast binds one scorer version, rubric, metric direction and aggregation rule across arms. A stochastic scorer needs its own repeat policy. Changing a scorer creates a new scoring revision for both arms.

1. Which metric, units, preferred direction and aggregation define the comparison score? measurement
2. Which harness, scorer code or rubric revision and configuration are shared by both arms? identity
3. How are scorer randomness, blinding and order effects handled? constraint
4. What evidence shows both arms were scored with the pinned scoring contract? validation

Execution realization and repeats

Execution remains externally mastered. The study indexes arm assignments, run attempts, environment snapshots and repetition units, preserving differences between a retry, an independent repeat and a reanalysis.

1. Which external training, inference and evaluation runs realize each arm and repeat? relationship
2. Which environment, configuration, seed and dependency revisions were actually used? provenance
3. When did each attempt start and end and which protocol revision governed it? temporal
4. Which attempts are retries, independent repeats or rescoring of existing outputs? classification

Complete outcomes and failures

Case and repeat keys join the two arms. Failures and absent results remain explicit; complete-case analysis must disclose exclusions and its population. A successful retry never overwrites a failed attempt.

1. Which per-case or aggregate observations and missingness indicators back each arm score? evidence
2. Do pair keys join one-to-one and how are duplicates, unmatched cases and nonfinite scores rejected? validation
3. Which failures, timeouts and exclusions occurred and how do they affect the denominator? exception
4. How are original attempts preserved when a retry or rescore produces a new observation? retention

Signed comparison and uncertainty

The proposed convention is delta = score_with_factor - score_without_factor. Metric direction controls interpretation. Uncertainty identifies its sampling unit and method; dataset-level test guidance is not automatically applicable to item-level observations.

1. What signed difference and practical importance threshold are reported for each contrast? measurement
2. Which variability sources and sampling units support the uncertainty estimate? quality
3. Which interval or test procedure was selected and why are its assumptions appropriate? process
4. How are undefined relative changes, insufficient repeats and non-estimable intervals represented? exception

Bounded conclusions and multiplicity

The study separates observed differences from explanatory claims and deployment decisions. Multiple contrasts, metrics or selected slices require a disclosed analysis family and selection history; a null result alone does not establish equivalence.

1. What conclusion is supported within this corpus, scorer and intervention setting? decision
2. Which hypothesis family and stopping or selection rules bound multiplicity and selective reporting? constraint
3. Which sensitivity checks assess exclusion, scoring or seed choices and where do conclusions change? quality
4. What additional evidence is needed before claiming equivalence, causal isolation or wider generalization? requirement

Evidence-backed review disposition

A reviewer assesses the record and its limitations independently of execution status. Review can reject or qualify a conclusion. A local release records availability and reproducibility limits without certifying safety or external acceptance.

1. Which qualified reviewer evaluated the contrast and what conflicts of interest were declared? authority
2. What review disposition and unresolved issues apply to each reported claim? state
3. Which evidence export profile and pinned mapping preserve links and disclose losses? interoperability
4. Which released artifacts permit recomputation and which dependencies remain unavailable? evidence

Controlled evidence custody

The proposed study policy separates public conclusions from restricted inputs and outputs. Disclosure, retention and disposal use local authority and rights checks. Dangerous evaluations are represented only by policy-level scope and oversight metadata.

1. Who may view each bundle, layer, finding and artifact under the evidence access policy? access
2. Which personal or sensitive content requires minimization or redaction before release? privacy
3. Which retention schedule, preservation holds and disposal rules govern each evidence class? retention
4. What authorization and audit trail govern access changes or restricted evaluation metadata? security