# Collaborative Memory Benchmark v1: methods

Run date 2026-09-07. Model `claude-haiku-4-5-20251001`. This study compares six ways of
representing one organisation's shared memory, three of which are reimplementations of
shapes that shipping products use.

## 1. Purpose

The two earlier studies in this series compared a structured representation against
documentation, and found that on average accuracy a complete, well-kept document is hard
to beat. Neither addressed the case this study is about: several teams writing into one
memory, disagreeing with each other, restating each other's numbers, and holding records
that must not cross a team boundary.

That case is not hypothetical and it is not unserved. Shared memory for agents is an
active product category, and a vendor-neutral memory format is an active standards
question. The question worth measuring is therefore not whether structured memory is a
new idea. It is narrower: **when a memory is written by several teams, does it matter
whether ownership, the conflict rule and the release list live inside the record or in a
governance layer beside it?**

## 2. Position in the landscape

This benchmark was designed after a survey of what exists, and the survey is part of the
result. Shared memory for agents is served by several products, a vendor-neutral memory
format is being worked on in at least four places including a W3C community group, and
the governance problem is named openly by the vendors themselves: one memory vendor's own
research lists privacy, consent, identity resolution and staleness detection as open
problems its tools were not designed to solve, and the common industry answer is a
separate governed glossary or catalogue beside the memory.

No claim of primacy is made or tested here. The comparison is between representations,
not between companies, and the strongest competitor in the design is the architecture the
industry actually recommends: a temporal graph with a governance catalogue next to it.

## 3. What is and is not compared

Three arms are reimplementations of documented product shapes. **No product was executed.**
Nothing was run against a vendor's service, no account was used, and retrieval, ranking,
indexing, ingestion quality and storage are all out of scope. What is compared is the shape
of the record an agent is given to read, which is one component of a memory system and not
the system.

A vendor could reasonably object that their product would do better than the shape of
their record suggests, because their retrieval selects what the agent sees. That objection
is correct and is the main limitation of this design.

| Arm | Shape | Modelled on |
|---|---|---|
| A shared wiki | team pages, one line per record, each carrying its dates, its writer and its source, plus a page on who owns what and a page on how disagreements are settled | how organisations actually keep this today |
| B vector memory with timestamps | extracted memory strings, each with a write date and a writer, no validity intervals, no ownership, no policy, no release list | a vector-first memory store that keeps timestamps but supports no as-of query, the documented behaviour of Mem0 |
| C bi-temporal knowledge graph | entity nodes, fact edges carrying valid-at and invalid-at and the episode they were extracted from | the documented shape of Zep and its Graphiti engine |
| D graph plus a governance catalogue | arm C with a separate document listing concept owners, accountable roles, authoritative sources, the rule for competing entries and the access rules | the architecture data-catalogue vendors position beside a memory graph, such as Atlan or Collibra |
| E dimension | one record set where each record carries value, version, validity interval, writing team, source, concept owner, accountable role and release list, with the conflict policy stated once | Vercy |
| F dimension minus governance | arm E with owner, role, policy and release list removed | ablation |

Arms A, D and E carry everything. B, C and F are ablations that drop a field class.

## 4. The fidelity check

Arms C and F remove the same field classes by two different routes: C by reimplementing a
competitor's shape, F by deleting fields from arm E. If the reimplementation in C is fair,
the two should score alike despite their different surface syntax. If C scored much worse
than F, that would indicate the competitor had been rendered as a straw man, and the study
would say so.

## 5. Materials

One fictional organisation, six teams, 30 records, 8 business concepts, 28 documents.
Every representation is rendered from that one record table, and a completeness check
fails the run if any complete arm is missing a value, a date, a writer, a source, an
ownership statement or a release target that another arm carries. That check found the
wiki paraphrasing definitions rather than carrying them while the instrument was being
built, and the wiki renderer was rebuilt from the fields as a result.

The organisational analogue of a source tier is ownership. Six concepts carry a rival
definition written by a team that does not own them, three concepts have a superseded
earlier version, and five records are restricted to a named list of teams.

## 6. Design

Within-item repeated measures: every item is presented under every arm. 35 items x 6 arms
x 2 replicates = 420 calls in the main grid, plus a scope track of 5 arms x 6 requesting
teams x 2 replicates over 30 release decisions each, and a restatement track of 3 arms x 5
scenarios x 2 replicates.

## 7. Item families

- **AUTH, 6.** Whose record is authoritative, and who owns the concept.
- **CONF, 6.** Two teams' records both cover today and disagree. Which stands.
- **HIST, 5.** What was in force on a date before another team revised it.
- **STALE, 5.** Whether a team's own entry has gone stale against the owner's. Three
  stale, two still current as controls.
- **ROUTE, 5.** Who must be asked to change something.
- **PROV, 4.** Which document holds the record in force.
- **GAP, 4.** Not recorded anywhere. Declining is correct.

## 8. Measures

| Code | Measure |
|---|---|
| M1 | The answer matches the reference answer. |
| M2 | The accountable team is named correctly. |
| M3 | The conflict rule applied is named. |
| M4 | The record or document relied on is cited. Citing the record is accepted as readily as citing the document it came from. |
| M5 | Correct abstention on unrecorded items; declining an answerable item is counted separately. |
| M6 | The superseded judgement matches the record's actual status. |
| M7 | The answer reports the figure written by a team that does not own the concept. |
| M8 | The answer contains a value that appears nowhere in the memory. |
| M9 | Release precision across the scope track. |
| M10 | Release recall across the scope track. |
| M11 | Critical over-shares: salary bands, litigation or acquisition talks released to a team not entitled to them. |
| M12 | A restatement records the new value with the right effective date and closes the right record. |
| M13 | The restatement preserves the earlier record and the other team's entry. |
| M14 | Characters of context supplied. |

## 9. Procedure

Tools disabled, the agent system prompt replaced with a neutral one, one JSON object per
response against a fixed contract, call order randomised across the grid. The world check
and the completeness check both run before any call is made.

## 10. Statistical analysis

The unit of analysis is the item. Replicates are averaged within an item before any test.
Paired comparisons use the exact two-sided sign test over item means; intervals are
clustered by item. Nine paired comparisons and four ablation contrasts are reported
without correction for multiplicity, so a nominal p of 0.05 is not a family-wise error
rate. The ablation contrasts are directional predictions stated in advance and each is
paired with a control family where the removed fields are predicted to make no difference.

## 11. Defects found and fixed

Two, both found by reading raw responses rather than summaries.

**Before the run.** The provenance measure originally accepted only the document
identifier. In arms that also carry record identifiers the model cited the record, which
identifies the source just as unambiguously, so the measure would have penalised the arms
with more identifiers. This was the same defect found in the previous study; here it was
fixed before the first call.

**During the run.** Three of the five historical items asked for a version number. Only
some representations carry an explicit version counter, and a bi-temporal graph carries
validity intervals without one, so the family was measuring which arms have a version
field rather than which arms can say what was in force on a date. The three items were
reworded to ask for the content in force, a match rule was added that requires the
period's wording and rejects the later wording, and the family was re-run under all six
arms. Both the original and the corrected runs are published; the analysis uses the
corrected one and says so in code.

## 12. Threats to validity

**Construct validity.** Reference answers come from the same record table the arms
display, so this measures retrieval and correct application of supplied records, not the
discovery of what is true in an organisation. The conflict rule is ours. The three
competitor shapes are reimplementations from public documentation and may not match a
current release.

**Internal validity.** Arms A, D and E are verified complete before each run, so a
difference between them cannot be a missing record. Arm E is the longest context and arm B
the shortest, and there is no padding control separating length from content.

**External validity.** One authored organisation, 30 records, one model, tools disabled.
Retrieval is excluded, which is the component several of the modelled products consider
their main contribution.

**Statistical conclusion validity.** 31 answerable items in families of four to six.
Family-level differences rest on very few items and are reported with that caveat.

## 13. Availability

The organisation, the six renderers, the item set, the runner, the analysis and every raw
run are published in full, including the superseded first run of the historical family.