AI Model Evaluation

Evaluation subject binding

The immutable binding between the evaluation and the model or system version, weights, configuration and inference settings under test.

1. Which authoritative identifier and version designate the exact model or system under evaluation? identity
2. Is the evaluated unit the bare model, a guarded endpoint, or the full deployed system including scaffolding and tools? classification
3. Which inference configuration, decoding parameters, system prompt, tooling and safety filters form part of the evaluated subject? composition
4. How is a hosted or continuously updated endpoint pinned so results remain attributable to a fixed subject? temporal

Intended purpose and claim scope

The intended purpose, deployment context and the specific claim the evaluation supports, with explicit out-of-scope uses and generalisability limits.

1. What precise claim about capability, safety or quality is this evaluation designed to support or refute? definition
2. Which documented intended purpose, deployment context and affected populations bound the validity of the claim? relationship
3. Which uses, populations, languages or environments are explicitly outside the evaluated claim? constraint
4. What documented limits on generalisability apply outside the tested distribution? quality

Evaluation mandate and trigger

The legal, contractual, policy or research basis obliging the evaluation, the event that triggered this instance, and the decision that consumes it.

1. Under which legal, regulatory, contractual or internal-policy basis is this evaluation required or commissioned? authority
2. Which event triggered this evaluation instance — release gate, capability threshold, incident, drift alarm or scheduled review? event
3. Which decision or gate consumes the result, and who is accountable for that decision? decision

Assessor identity, competence and independence

Who designed, executed, scored and reviewed the evaluation, their competence evidence, degree of independence from the development team, and declared conflicts.

1. Which named organisations and roles designed, executed, scored and reviewed this evaluation? ownership
2. What degree of independence separates the evaluators from the development team, and how is it evidenced? classification
3. What evidence establishes that assessors and human raters are competent for the domain and risk being measured? evidence
4. Which conflicts of interest were declared and how were they managed? constraint

Evaluation method class and baseline

Classification of the evaluation method — statistical, formal, empirical, human-subject or adversarial — with justification, declared blind spots and comparison baselines.

1. Which method class is applied, and why is it appropriate to the property being assessed? classification
2. Which properties can this method class not evidence, and what complementary method covers that gap? constraint
3. Against which baseline, reference implementation or comparator system are results interpreted? measurement
4. How is the method shown to reflect the state of the art at the time of execution? requirement

Task and dataset binding with contamination control

Version-pinned reference to evaluation tasks, datasets and splits, the ground-truth definition, and controls against train/test contamination and held-out set erosion.

1. Which dataset version, snapshot or split identifiers, and which checksums, are bound to this evaluation? relationship
2. How is ground truth or the reference answer defined, produced and quality-assured for each item? definition
3. What controls establish that the evaluation items were not present in training or tuning data? validation
4. How is a held-out or canary split protected from repeated exposure that would invalidate future results? access

Scenario, task and adaptation protocol

A HELM scenario is a task plus domain plus language. Inspect Task composes dataset, solver or agent, and scorer, optionally with a sandbox and tools. Standardized few-shot or other adaptation is required so comparisons attribute performance to the subject. UK AISI autonomous-system submissions must be Inspect tasks and should expose a binary success scorer.

1. What task, domain, language or modality, and user-facing use case define this scenario, and which HELM or Inspect eval identifier does it correspond to? identity
2. What adaptation protocol (zero-shot, few-shot, chain-of-thought, agent loop, tool use) is frozen, including prompt templates and in-context example policy? process
3. If the task is agentic, which tools, MCP servers and sandbox (Docker, Kubernetes or other) isolate untrusted model actions, and what approval gates exist for tool calls? security
4. Does the scenario depend on a geographic, legal or language locale, and if so which locale constraints apply to items and raters? spatial

Metric specification

Formal specification of each metric: construct measured, computation, aggregation, reporting precision and the scorer implementation and version producing it.

1. What is the exact computational definition, unit and aggregation rule for each reported metric? definition
2. Which construct does the metric claim to measure, and what evidence links the metric to that construct? quality
3. Which scorer implementation, version and configuration produced the metric values? provenance
4. What rounding, significant-figure and tie-breaking conventions govern reported values? constraint

Disaggregation factors and subgroup analysis

Factors along which results must be broken down, the sufficiency rules for subgroup reporting, and the disparity comparisons performed.

1. Which factors must results be disaggregated by, and how was that factor list justified? composition
2. What minimum sample size per subgroup is required before a disaggregated result is reportable? measurement
3. Which disparity comparison is applied across subgroups, and what gap is treated as a finding? validation
4. How are protected attributes obtained or inferred for disaggregation, and under which lawful basis? privacy

Acceptance criteria and thresholds

Pass/fail criteria, probabilistic thresholds, risk tiers and safety margins fixed and recorded before results are observed, with change control for later alteration.

1. Which acceptance thresholds and probabilistic criteria were fixed before results were observed, and where is that pre-registration recorded? requirement
2. Who set each threshold, on what basis, and with what safety margin? authority
3. How is a change to a threshold after results are known authorised, justified and disclosed? lifecycle
4. What consequence follows automatically from breaching each threshold? decision

Human evaluation protocol

Recruitment, representativeness, instructions, blinding, agreement measurement and participant protection for human-rated evaluation.

1. How were raters recruited and how does the rater population relate to the affected population? composition
2. Which inter-rater agreement statistic is computed and what value is required for the result to be usable? measurement
3. Which human-subject protections, consent arrangements and ethics approvals govern the evaluation? privacy
4. How are raters blinded to system identity and condition to prevent expectancy effects? process

Adversarial and red-team protocol

Threat model, attacker capability and knowledge assumptions, attack-class coverage, elicitation effort and stopping rules for adversarial testing.

1. Which threat model, attacker goals, capabilities and knowledge assumptions define this adversarial test? security
2. Which attack classes — evasion, poisoning, privacy, prompt injection, misaligned output — are in and out of scope? classification
3. How much elicitation effort, tooling and time was expended, and how is that effort evidenced as sufficient? measurement
4. What stopping rule ends the exercise, and how are unresolved attack avenues recorded? constraint

Run record, configuration and resource accounting

Identity of a single run, its complete configuration, timing, and the compute, token, latency and energy quantities attributable to it.

1. Which identifier uniquely designates this run, and how does it relate to the campaign and to repeated epochs? identity
2. Which configuration values — seeds, sampling parameters, batch, concurrency, epochs, prompt template — are captured with the run? composition
3. What are the run start and end times, and how are they distinguished from result publication and record ingestion times? temporal
4. What compute, token, latency, monetary and energy quantities are recorded for the run? measurement

Environment capture and replication

Software and hardware environment, dependency versions, bounded non-determinism, replication tolerance and declared irreproducible components.

1. Which code revision, dependency versions, container image and hardware description are recorded for the run? provenance
2. Which sources of non-determinism are permitted, and how are they bounded or seeded? constraint
3. What exact procedure and inputs allow an independent party to replicate the run, and what tolerance defines success? process
4. Which parts of the run are irreproducible, such as a retired endpoint or a one-off rater panel, and how is that recorded? exception

Result record, aggregation and item-level evidence

Reported metric values keyed by subject, task, split, metric and factor, the traceable aggregation path from per-item scores, and the retained per-item inputs, outputs, traces and scores.

1. How is a single reported result value uniquely identified across subject, task, split, metric and factor? identity
2. What is the traceable path from per-item scores through reducers to each headline value? composition
3. Which fields are retained per evaluated item, and which are required to re-score without re-running inference? evidence
4. When a result is recomputed or corrected, how are the prior value, the scorer version and the reason retained? state

Uncertainty and statistical validity

Sample size adequacy, variance, interval estimates, significance rules and multiple-comparison control governing what the numbers may be said to show.

1. Which uncertainty estimate accompanies each headline value, and by what procedure was it computed? measurement
2. How was the number of items and repetitions determined to be adequate for the claimed resolution? validation
3. Under which rule is a difference between two systems or two versions declared meaningful rather than noise? constraint
4. How are multiple comparisons across metrics, factors and checkpoints controlled? quality

Scoring and judging

Inspect scoring maps output to a Score against a Target, with built-in match, multiple-choice, math, model-graded and perplexity scorers, custom scorers, multiple scorers and epoch reductions. Model grading uses another model as judge. Human raters and CJE-style calibration of judges against oracles are first-class. ISO/IEC TR 29119-11 highlights the oracle problem when expected results are hard to specify. Refusals may be logged separately from task success.

1. Which scorer or judge configuration produced each sample score, including grader model, rubric, and whether scoring was live or deferred and later re-scored? process
2. For each sample, what were the input, output, target, score value, explanation and refusal or scanner flags? measurement
3. If a model or crowd judge was used, what agreement with gold oracles, inter-rater reliability and bias controls support treating those judgments as evidence? quality
4. Do sample inputs, outputs or rater notes contain personal data or prohibited content, and what minimization or redaction applies? privacy

Evidence package and integrity

Bundling, manifesting, digesting and sealing of evaluation evidence so it can be transferred to auditors or authorities without loss of integrity.

1. Which items constitute the sealed evidence package for this evaluation, and how is completeness asserted? composition
2. Which digest, signature or timestamping mechanism protects the package, and who holds the signing authority? security
3. How is a later alteration of any packaged artefact detected and reported? validation
4. In what neutral form is the package transferred to an auditor or authority without loss of meaning? interoperability

Provenance and lineage graph

Graph linking datasets, subject artefacts, runs, scorers, results, findings and decisions to responsible agents and times, using entity/activity/agent semantics.

1. Which generation, derivation, usage and attribution edges must be recorded between evaluation entities? relationship
2. Which agent — person, organisation or software — is recorded as responsible for each activity and artefact? ownership
3. How are activity start and end times recorded and reconciled with the time the record reached the evaluation store? temporal
4. How is lineage preserved when datasets, models or scorers are governed in external systems? interoperability

Evaluation finding record

An atomic, separately citable finding derived from results: what was observed, its severity and likelihood, the evidence behind it, and its state.

1. How is a single finding identified and cited independently of the report that first presented it? identity
2. Which specific results, items and traces substantiate the finding? evidence
3. Which severity and likelihood scale classifies the finding, and who assigns it? classification
4. What states can a finding occupy — open, accepted, mitigated, disputed, withdrawn — and what moves it between them? state

Validity threats and declared limitations

Threats to construct, internal and external validity, benchmark saturation or gaming, disclosure of inconclusive results, and periodic review of the measurement approach itself.

1. Which threats to construct, internal and external validity are declared for this evaluation? quality
2. How is benchmark saturation, gaming or optimisation against the test set assessed and disclosed? validation
3. How are inconclusive or negative results recorded so they cannot be silently dropped? evidence
4. How and when is the effectiveness of the measurement approach itself reviewed? process

Approval decision, conditions and waivers

The authorised approve, approve-with-conditions, reject or defer decision, its signatories, the evidence version reviewed, the authorised scope, and any time-bounded waiver with compensating controls.

1. Who holds the authority to approve or reject on the basis of this evaluation, and under which delegation? authority
2. What decision was taken, on which sealed evidence version, and with what conditions attached? decision
3. Which deployment scope, markets, populations and time window does the authorisation cover? constraint
4. On what documented basis may release proceed when an acceptance criterion is not met, and when does that waiver expire? exception

Evaluation lifecycle and re-evaluation triggers

States an evaluation instance passes through, the conditions requiring repetition or invalidation, and the cadence of periodic re-evaluation and in-production measurement.

1. Which states does an evaluation instance pass through from planned to archived, and which transitions are irreversible? lifecycle
2. Which changes to model, data, threat landscape, scaffolding or deployment context oblige re-evaluation? event
3. At what cadence is periodic re-evaluation and in-production measurement required? temporal
4. Under what conditions is a completed evaluation declared invalid, and how are dependent approvals affected? state

Evaluation report and audience tiering

Report instances derived from the evaluation, tiered by audience, with rules preventing public claims from exceeding internal evidence, and versioning for corrections and retractions.

1. Which audience tiers exist and what content is withheld or summarised in each? classification
2. How is it ensured that public claims never exceed what the full internal evidence supports? validation
3. How are report versions, corrections and retractions identified and communicated? lifecycle
4. Which report content is mandated by regulation or code, and by what deadline? requirement

External and third-party evaluation

Arrangements for independent evaluators: access tier, scope, safe harbour, publication rights, exemption basis and handling of dissent.

1. What level of access — black-box, grey-box, weights, fine-tuning or internal tooling — is granted to external evaluators? access
2. Which contractual terms bound scope, timing, safe harbour and publication rights? constraint
3. On what basis may an external evaluation be omitted, and how is that basis evidenced? exception
4. How is disagreement between internal and external assessments recorded and resolved? process

Access control and hazardous detail handling

Default access rules per record scope, redaction of uplift-relevant or exploit-enabling detail, handling of personal data in traces, and access auditing.

1. What is the default access rule for each record scope, and which roles hold read or write rights? access
2. Which uplift-relevant or exploit-enabling details must be redacted, held separately, or shared only with authorities? security
3. How are personal data in prompts, traces and rater records minimised, lawfully based and protected? privacy
4. Which access events are logged, and for how long are those logs themselves retained? evidence

Retention, deletion and legal hold

Retention periods and legal bases per record class, resolution of deletion-versus-evidence conflicts, legal-hold handling, and the tombstone that keeps prior claims auditable after deletion.

1. What minimum and maximum retention period applies to each evaluation record class, and on what legal basis? retention
2. How are conflicts resolved between data-minimisation deletion duties and evidence-retention duties? exception
3. How is a legal hold applied, recorded and released across evaluation records? process
4. What summary or digest must survive after primary records are deleted so that prior claims remain auditable? requirement

Standards alignment, conformance limits and exchange interoperability

Which external framework clauses each record claims to address, how alignment is kept distinct from certified conformance, which identifier schemes are used, and when results from different organisations or rounds may be compared.

1. Which external framework clauses does each evaluation record claim to address, and what evidence supports the claim? interoperability
2. How are alignment statements prevented from being read as certified conformance? constraint
3. Which identifier schemes and namespaces are used for subjects, datasets, metrics and reports across organisations? identity
4. Under what conditions may results from two organisations or two rounds be compared without misleading readers? measurement
5. Where two frameworks impose conflicting requirements, how is the conflict recorded and resolved? decision