Evaluating Memory Systems¶
How does it survive time, scale, multiplicity, and scrutiny?
Module 7 of 9 in Time, Scale, Multiplicity, Trust · 6 lessons · about 4.7 hours
70. Why Memory Eval Is Hard · ~45 min
Count what can be scored at all, before scoring anything.
71. Component Metrics · ~50 min
Score the stages -- and debug the metric before believing it about the system.
72. Build Your Own Harness · ~50 min
One battery across every profile -- and name what it cannot see.
73. Regression Testing a Stateful System · ~45 min
Half the tests pin a measured number -- and a frozen corpus is what makes that affordable.
74. LLM as Judge, and Its Failure Modes · ~45 min
One judgement of three model calls -- and arbitration is never one of them.
75. Reading Benchmark Claims Critically · ~45 min
Four questions for any score -- applied to your own claim first.