Count Only What You Can Prove

Every tool in the stack produces numbers: findings, costs, scores, counts. The easy way to make those numbers look good is to count what is convenient. The useful way is to count only what you can prove, and to say so plainly when you cannot.

This cycle, three products drew that line in three different places.

TheAuditor: clean means accounted for

A security analysis that reports zero findings is making a strong claim: it looked, and nothing was there. That claim only holds if the analysis can show that it actually looked at everything it was supposed to.

TheAuditor now holds itself to that. A step that found nothing reports “empty” only when everything it depends on was verified. If something upstream was not, the result is a gap, not an empty answer, and the run is reported as incomplete rather than clean. The details are in a run that cannot account for every fact does not read as clean.

Arbiter: a ticket’s cost comes from proven work

A work list is easy to fill with estimates. Arbiter’s Tickets do something stricter: a ticket records a session’s time, tokens, cost, edited files and commits only when Arbiter can prove that session worked on that ticket.

Estimates follow the same rule. They come from your own closed tickets and are never invented, so the number on a new ticket is a statement about how similar work actually went. More in Tickets: a work list joined to what your sessions actually did.

Curator: the measure that matters is the one you feel

Memory systems are usually judged by benchmarks that ask whether the right note comes back on request. That measures retrieval. Curator is now held to a plainer test: how often you have to tell your agents the same thing again.

Benchmarks still run, as regression checks, but they no longer stand in for the result. The reasoning is in the real test of a memory is how often you repeat yourself.

The same reason BenchProctor exists

BenchProctor was built on this idea from the start. A scanner’s detection rate is a claim until it is scored against labeled cases where the right answer is known. Its published corpus is split exactly in half, 1,469,209 vulnerable cases and 1,469,209 safe ones, so a tool’s false alarms are measured as carefully as its detections.

Counting only what you can prove costs something. Some numbers get smaller, some get slower to produce, and a few turn into a plain “we do not know yet.” But a number you cannot stand behind is worse than no number at all, because someone, or some agent, will build on it.