Vercy · reproducible study
Tools did not close the gap. They widened it.
- Objective
- Every study in this series disabled tools and said so as its main limitation: with a shell and a data file the arithmetic component of the error changes, and the balance between representations might shift. This tests that.
- Design
- The published semantic grounding benchmark, re-run unchanged except that the five tables move out of the prompt onto disk as CSV files and the agent is given Bash, Read, Glob and Grep in a fresh working directory per call. Same six context conditions, same 40 items, same reference answers, same scorer. 240 calls, one replicate per item, median 55 seconds each.
- Principal result
- Tools removed the arithmetic error only where the meaning was known. The four conditions carrying complete definitions rose to between 97.1% and 100%. The two without them did not move: schema alone went from 26.5% to 23.5%, incomplete documentation from 29.4% to 32.4%.
- The uncomfortable part
- Where definitions were missing, tools made the agent more confidently wrong. The rate of answering with exactly the number a superseded rule produces rose from 16.2% to 29.4% on schema alone and from 35.3% to 47.1% on incomplete documentation, and appropriate abstention fell from 83.3% to 50.0% and 66.7%.
- Conclusion
- The limitation is resolved in the opposite direction to the usual expectation. Tools are not a substitute for knowing what the words mean; they raise the cost of not knowing. The gap between having definitions and not having them grew from 50 points without tools to 74 points with them.
What changed and what deliberately did not
Changed: the five tables move out of the prompt and onto disk, and the agent gets a shell in a fresh working directory per call. That is what a real deployment looks like and it is the whole point. Unchanged: the six context conditions, the 40 items, the reference answers, the output contract and the scorer are the ones the original study published, and the context still arrives in the prompt, because the context is the treatment. Moving it to a file as well would change two things at once.
Accuracy, before and after
| Condition | Definitions | Without tools | With tools | Change |
|---|---|---|---|---|
| A schema only | absent or incomplete | 26.5% | 23.5% | -2.9 |
| B documentation, incomplete | absent or incomplete | 29.4% | 32.4% | +2.9 |
| E prose, complete | complete | 76.5% | 100.0% | +23.5 |
| C versioned records | complete | 76.5% | 97.1% | +20.6 |
| F records + inert padding | complete | 85.3% | 100.0% | +14.7 |
| D records + federation rules | complete | 77.9% | 100.0% | +22.1 |
The split is clean. Every condition that knows what the words mean goes to the ceiling; neither condition that does not know moves at all. A shell computes what you ask it to compute, and the question of what to compute is exactly what a definition answers.
The two error types
Accuracy is the least interesting column here. The two that matter are how often the agent returned exactly the number a superseded or rival rule would produce, and how often it correctly declined a question the data cannot answer.
| Condition | Superseded-rule answer, no tools | with tools | Correct abstention, no tools | with tools |
|---|---|---|---|---|
| A schema only | 16.2% | 29.4% | 83.3% | 50.0% |
| B documentation, incomplete | 35.3% | 47.1% | 83.3% | 66.7% |
| E prose, complete | 2.9% | 0.0% | 66.7% | 50.0% |
| C versioned records | 1.5% | 0.0% | 58.3% | 66.7% |
| F records + inert padding | 0.0% | 0.0% | 66.7% | 50.0% |
| D records + federation rules | 1.5% | 0.0% | 66.7% | 83.3% |
Both move the wrong way where definitions are missing. Given a calculator and no definition, the agent computes something, and what it computes is more often the number a superseded rule would give. It also stops saying that it cannot answer: appropriate abstention on schema alone fell by a third. Where definitions are present the superseded-rule rate goes to zero in every condition.
By question family, with tools
| Condition | computation | as-of a past date | counterparty-facing | unanswerable |
|---|---|---|---|---|
| A schema only | 41.7% | 20.0% | 8.3% | 50.0% |
| B documentation, incomplete | 41.7% | 50.0% | 8.3% | 66.7% |
| E prose, complete | 100.0% | 100.0% | 100.0% | 50.0% |
| C versioned records | 100.0% | 100.0% | 91.7% | 66.7% |
| F records + inert padding | 100.0% | 100.0% | 100.0% | 50.0% |
| D records + federation rules | 100.0% | 100.0% | 100.0% | 83.3% |
The definition-bearing conditions are perfect or near perfect on all three answerable families. The unanswerable family is where tools hurt: the agent has a shell, so it produces a number rather than declining.
The comparisons the earlier study rested on
| Comparison | p without tools | From, with tools | To, with tools | p with tools |
|---|---|---|---|---|
| A schema only against C versioned records | 0.00055 | 23.5% | 97.1% | 0.0 |
| B documentation, incomplete against C versioned records | 0.00151 | 32.4% | 97.1% | 0.0 |
| E prose, complete against C versioned records | 1.0 | 100.0% | 97.1% | 1.0 |
| C versioned records against D records + federation rules | 1.0 | 97.1% | 100.0% | 1.0 |
| A schema only against E prose, complete | 0.00151 | 23.5% | 100.0% | 0.0 |
The finding that carried the original study, that supplying definitions at all is what raises accuracy, survives and strengthens: 25 items better and none worse. The finding that went against us also survives: complete prose and versioned records remain indistinguishable, now at 100% and 97.1%, with the single differing item favouring prose.
Conclusion
Tools resolve the limitation the earlier studies declared, and they resolve it against the common assumption. They do not compensate for missing semantics: they make the failure quieter and more confident, because an agent that can compute stops saying it cannot answer. The practical reading is that giving an agent database access without giving it the definitions is worse than it looks, and looks better than it is.
What this does not show
One replicate per item rather than two, so there is no within-item averaging and the estimates are noisier than the original study's. One model, one synthetic world, and a shell with local CSV files rather than a real database with a query planner and permissions. Tool use also cost about five times the wall-clock: a median of 55 seconds per call against roughly 10 without. The context conditions still arrive in the prompt; a deployment that also put the definitions on disk might behave differently, and that is the next thing to test.