Benchmark Claim

A reported score, plus the four things the number cannot say: what moved, what was already saturated, what has no metric, and on what data.

Why it matters in a memory layer

Retrieval benchmarks are comparable because the corpus and the judgements are shared, so a difference between two papers is a difference between two systems. Memory benchmarks share almost nothing — their conversations, question types, notions of a correct answer, and crucially the division of labour between the memory layer and the model reading its output. A number can improve because a larger reader compensated for worse retrieval.

Apply the reading to your own claim first. The most defensible one this course can make is "1.000 on four metrics across six profiles", every word true, and one of the four is informative — the other three were already saturated before the level began.

honest is checkable rather than a judgement about the claimant: does the claim carry its moved, flat, absent and corpus qualifiers? And the qualifier most often dropped is the last one, which here reads one corpus, one persona, 24 turns.

Connections

Taught in: Reading Benchmark Claims Critically

Used in: Hardening Pass

Do not confuse with: Saturated Metric