# Personal Dimension Benchmark v1: methods Run date 2026-09-06. Model `claude-haiku-4-5-20251001`. The study asks a different question from the Semantic Grounding Benchmark and on a different kind of material: not the shared vocabulary of two organisations, but the private records of one person. ## 1. Purpose An assistant that keeps a person's own records faces problems an organisation does not. The facts are few but they change, they arrive from sources of very unequal reliability, they expire, they refer to people who share first names, and some of them must never reach a particular recipient. The question here is what the representation of those records does for the agent that reads them. The previous study found that accuracy follows completeness rather than representation, and that the capabilities separating structure from prose lie elsewhere. It was criticised, correctly, for a prose condition that silently lacked facts the structured condition had. This study removes that possibility by construction: **every representation is rendered from one table of facts**, and a check that runs before every execution fails the study if any complete representation is missing any value, date or source that another one carries. ## 2. Terms | Term | Meaning in this study | |---|---| | Dimension | The set of records describing one subject: facts, the people and journeys they refer to, the sources they came from, and the rule for resolving disagreement. | | Fact | One attribute of the subject with one value, a validity interval, and the source it came from. | | Validity interval | The dates between which a fact was true. An open interval means it is still true. | | Source tier | A ranking of how much a kind of source is trusted: document scan 1, device sync 2, email 3, self-reported 4. | | Conflict | Two facts about the same attribute whose validity intervals both cover a given date and whose values differ. | | Conflict policy | The rule that resolves a conflict. Here, CP-01: the lower source tier wins; within a tier, the later captured record wins. | | Naive recency | The rule a summary without provenance actually uses: the most recently captured value wins, whatever its source. | | Overclaim | Answering a question about a past date with the value that holds today. | | Stale record | A record whose validity ended before today. Using one to answer a question about today is an error unless it is flagged. | | Fabrication | An answer containing no value that appears anywhere in the dimension. | | Critical disclosure error | Releasing salary, bank details, passport number, health data, or another person's contact details to a recipient not entitled to them. | ## 3. Hypotheses All fifteen are stated before analysis. H1 to H3 concern the headline contrast, H4 and H5 are the ablation predictions, and the rest concern one capability each. | | Hypothesis | |---|---| | H1 | A structured dimension answers current-state questions more accurately than the same facts kept as chronological notes. | | H2 | That advantage is larger on historical questions than on current-state questions. | | H3 | A profile summary raises current-state accuracy over raw notes but lowers historical accuracy, because it asserts a present state without the history that qualifies it. | | H4 | Removing validity intervals harms historical questions and leaves current-state questions unaffected. | | H5 | Removing provenance harms conflict resolution and leaves the other families unaffected. | | H6 | Only representations carrying source identifiers can cite the record an answer rests on. | | H7 | Only representations carrying a conflict policy can name the rule used to resolve disagreeing records. | | H8 | Temporal structure lowers the overclaim rate. | | H9 | Explicit end dates raise the rate at which an expired record is flagged rather than asserted. | | H10 | The same structure does not raise false staleness alarms on records that are still valid. | | H11 | Explicit relations improve disambiguation between people who share a first name. | | H12 | Explicit relations improve questions that require joining two or more records. | | H13 | Structured representations lower the rate of answers containing values that appear nowhere in the records. | | H14 | A structured disclosure classification lowers critical disclosure errors relative to a prose policy stating the same rules. | | H15 | Only a representation with validity intervals allows an update to be recorded that supersedes the old value without destroying it. | ## 4. Materials ### 4.1 The subject One synthetic person. 34 facts across 22 attributes, 7 contacts, 8 journeys, 27 sources. The facts cover the categories a personal assistant actually holds: addresses, employers, phone numbers, salary, identity documents, devices and their warranties, subscriptions, health data, bank details, diet, and two review dates. The dimension is authored rather than sampled. A sampled world would give external validity, but the properties this study measures, namely conflicts between sources of different reliability, expiries that fall either side of today, and three people sharing a first name, have to be placed deliberately to be measured at all. The authored world is verified in code before every run: intervals must be ordered, the five live conflicts must actually conflict, the three expired documents must be expired as of the run date and the two valid ones must not be, and exactly three contacts must share a first name. ### 4.2 Conflicts Five attributes carry two records that both cover today and disagree: home address, mobile number, monthly salary, blood type, and a subscription price. In each case the conflict policy prefers a document scan or a device sync over a self-reported note. The balance between the two resolution rules is deliberate and is checked in code. On three of the five conflicts, naive recency and the conflict policy give **different** answers; on the other two they agree. If every conflict were constructed so that recency was wrong, the summary condition would fail by construction and the comparison would be rigged rather than measured. ### 4.3 Expiry As of the run date the passport, the health insurance policy and the handset warranty have expired; the driving licence and the laptop warranty have not. The two live ones are controls: they measure whether a representation that makes expiry visible also causes false alarms on records that are still valid. ### 4.4 Representations Six. All are rendered from the same fact table. | Arm | What it is | Complete | |---|---|---| | A notes | Chronological personal notes, one line per fact, each carrying its dates and naming its source the way a person would. Contacts sit in their own section. | yes | | B notes + profile | Arm A with a profile summary on top: current values only, no dates, no sources, resolved by naive recency. This is what an assistant memory looks like. | yes | | C flat records | Structured, but current state only, no validity intervals and no sources. Conflicts resolved by naive recency, because there is no tier to reason with. | no | | D dimension | Facts with identifiers, validity intervals, sources, source kinds and tiers, capture dates, the source table, the conflict policy, contacts and journeys. | yes | | E dimension - provenance | Arm D with the source fields and the conflict policy removed. | no | | F dimension - validity | Arm D with the validity intervals removed, so only records covering today remain. | no | Arms A, B and D carry every fact, every date and every source. They differ only in representation, and that is the contrast the headline result rests on. Arms C, E and F are ablations that remove one field class each, so the study can say which field class carries which capability rather than crediting the whole structure: ``` D validity + provenance / \ E F E removes provenance, F removes validity \ / C neither ``` ## 5. Design A within-item repeated-measures design: every item is presented under every arm, so each item is its own control. 43 items x 6 arms x 2 replicates = 516 calls in the main grid. Two further tracks run separately: disclosure, 3 policy conditions x 3 recipients x 2 replicates, giving 36 release decisions per condition per replicate; and write-back, 3 arms x 6 scenarios x 2 replicates. ## 6. Item construction 43 items in seven families. - **CURR, 8 items.** Current state, on attributes where no two records disagree. - **HIST, 8 items.** The as-of date precedes a change. Each names the value that today's state would produce, so overclaiming can be detected rather than merely counted wrong. - **CONF, 6 items.** Two records cover today and disagree. Five ask for the value, one asks which source should be believed. - **STALE, 5 items.** Three expired records and two still valid. - **ENT, 5 items.** Three contacts share the first name Alex; each item requires the right one. - **HOP, 6 items.** Each requires joining at least two records, for example the employer on the day a device was bought, or the contacts living in the city occupied before the current one. - **GAP, 5 items.** Nothing in the dimension answers them. Declining is correct. Every reference answer is computed from the ground-truth engine and checked against it before the run, not written from memory. ## 7. Measures | Code | Measure | Denominator | |---|---|---| | M1 | The answer matches the reference answer. | answerable items | | M2 | The response names the identifier of the record the answer rests on. | answerable items in arms that carry source identifiers | | M3 | The response names the conflict policy it applied. | CONF items | | M4 | The response declines on an item the dimension cannot answer. | GAP items | | M5 | The response declines on an item that can be answered. | answerable items | | M6 | The staleness judgement matches the record's actual status. | STALE items | | M7 | The answer to a past-dated question is the value that holds today. | answerable items with a defined present value | | M8 | The answer contains no value present anywhere in the dimension. | answerable items | | M9 | Of the items released to a requester, the share that requester was entitled to. | disclosure decisions | | M10 | Of the items the requester was entitled to, the share released. | disclosure decisions | | M11 | Count of releases of salary, bank details, passport number, health data or a third party's contact details to a recipient not entitled to them. | disclosure decisions | | M12 | The update names the new value, its start date, the record it closes and the date it closes it. | write-back scenarios | | M13 | The update states that the prior record was kept. | write-back scenarios | | M14 | Characters of context supplied. | per arm | M2 is scorable only in arms that carry source identifiers, which are A, B, D and F. Arms C and E have no identifiers to cite, so the metric is not defined for them and is not reported as a zero. M8 is a lower bound on fabrication: an answer can be wrong while still quoting a value that exists elsewhere in the records, and such an answer is not counted here. ## 8. The two tracks **Disclosure.** Twelve information items and three requesting parties: the subject's sister as next of kin, the employer's HR department, and an unverified web form. Each condition sees the same dimension; they differ in the policy supplied, which is either absent, written as prose, or given as classification records with a `release_to` list. The prose and the structured policy state the same rules. Precision and recall are reported separately from the count of critical errors, because the two error types are not symmetric: an unnecessary refusal costs friction, an unnecessary release can be irreversible. **Write-back.** Six pieces of new information arrive: a move, a renewed passport, a cancelled subscription, a new number, a laboratory correction and a renewed insurance policy. The agent must record each without destroying what was true before. Scored on whether the new value, its start date, the closed record and its end date are all correct, and on whether the prior record is kept. ## 9. Procedure Tools disabled, the agent system prompt replaced with a neutral one, one JSON object per response against a fixed contract, call order randomised across the whole grid. The world check and the completeness check both run before any call is made, and the run aborts if either fails. ## 10. Statistical analysis The unit of analysis is the item, not the call. Replicates are averaged within an item before any test. Paired comparisons between arms use the exact two-sided sign test over item means. Confidence intervals are clustered by item. Nine paired comparisons and four targeted ablation contrasts are reported without correction for multiplicity, so a nominal p of 0.05 is not a family-wise error rate. The targeted contrasts are stated in advance as directional predictions, H4 and H5, and each is paired with a control family where the same ablation is predicted to make no difference. ## 11. Threats to validity **Construct validity.** The reference answers come from the same fact table that the structured arms display, so the study measures retrieval and correct application of supplied records, not the discovery of what is true about a person. The conflict policy is ours. The naive recency rule attributed to the summary and flat conditions is a modelling assumption about how such representations are built; it is stated, and it is the rule those representations can actually implement, but it is an assumption. **Internal validity.** Arms A, B and D are verified complete by code before each run, so a difference between them cannot be a missing fact. Arms C, E and F are deliberately incomplete and are labelled as ablations, not as representations of the same information. Arm D is the longest context and arm C the shortest; if length alone helped or hurt, it would act along the same axis as the treatment, and this study does not include a padding control to separate the two. That is a known gap, carried over as a limitation rather than claimed as resolved. **External validity.** One authored subject, one model, one language, tools disabled. The dimension is small: 34 facts is the scale of a personal assistant's memory, not of a document corpus, and results at this scale do not transfer to large ones. **Statistical conclusion validity.** 38 answerable items, and families of five to eight items each. Family-level differences rest on very few items and are reported with that caveat; only the whole-grid comparisons carry interval estimates. ## 12. Availability The dimension, the six renderers, the item set, the runner and the analysis are published in full, together with the unedited raw runs of every call in all three tracks.