Seven harness changes were measured over one week and six of them moved nothing. That is a strange result to keep collecting without asking the obvious question: what is actually failing?

Two measurements answer it. Seventy-five runs — fifteen benchmark cases, five passes — classified by what the run left behind. And eight hundred and thirty-six model turns from the recorded traces, classified by what the model was sent.

A third of runs fail

Two thirds of the failures are wrong code

Sixty-seven per cent of failures are a file that changed and a job that did not get done. That matters more than the headline rate, because it says what kind of fix could help.

A refusal catches a draft that is recognisably wrong: a name that says nothing, a Find: line copied from memory, a module written twice. It cannot catch a draft that is plausible and wrong, because nothing deterministic can tell those apart. Only running the code can, and the harness already does — the suite runs, and a red suite is one of the verdicts above.

So the remaining two thirds are not waiting for another rule.

Nothing claims success having done nothing

Of the twenty-four failures, eight ended by saying done. Every one of them had written a file:

env-flag      writes=2  AssertionError: 0
venv-python   writes=1  venv_python not found in any module
initials      writes=2  AssertionError: A L

Nought of eight wrote nothing. That was the commonest and worst shape a week ago — a run reporting success with the file untouched, at two of nine failures — and it is gone at seventy-five runs. The rule that closed it asks a run to quote a line from the file it says was already correct, and to prove that a name it was asked to add is actually there.

That is the one measured gain of the week, and it is worth being precise about what it gained: not a higher pass rate, but no lies about it.

Where the failures sit

Platform work is still the hardest, at a little over half.

A quarter of every run is the harness saying no

Eight hundred and thirty-six turns over seventy runs — about twelve turns a run.

What it is told, most often:

58  run the tests before finishing
36  read the file before patching it
32  that is the wrong file
11  the draft did not parse
10  too late to ask, files are changed
10  you already ran that skill

The second line is worth a note. When that rule was added, the write-up called it “narrower than it sounds”, reasoning that a task naming a file gets that file located and so the rule would rarely fire. The reasoning about the mechanism was right and the guess about the frequency was wrong: it is the second most common thing the harness says, because models reach for a second file far more often than expected.

What this changes

The seven measurements that moved nothing now have an explanation rather than a shrug. Two thirds of what still fails is the model writing plausible, wrong code, and no deterministic rule separates plausible and wrong from plausible and right. The harness has taken the failures it can take.

That is an argument for the weights, and it is the first one on this site that comes from measuring the failures rather than from hoping. It also sets the bar: a fine-tune has to move the sixty-seven per cent, not the honesty, because the honesty is already fixed.

Reproducing this

python scripts/measure/bench.py --repeat 5
python scripts/weights/collect_traces.py --passes 4

The first prints a row per run with what it wrote and why it failed. The second gathers the turns; both sets of numbers here come from reading those two files and nothing else.