The instrument was broken ended with a question it could not answer: a hosted 32B scored 1 of 10 on tier 3, which is not a number a 32B produces, and the reason looked like a parser fault rather than a fact about the model.

It was a parser fault. With the fence stripped, the same model on the same cases scores 9 of 10.

The measurement

Tier 3, ten runs, same two cases, same prompts. The only thing that changed between the two columns is whether the harness takes the markdown fence off a draft before writing it to a file.

Per case, after: slugify 5 of 5, wordcount 4 of 5. The single failure is an ordinary one — word_count not found in any module — not a broken file. Median run 15.3 seconds.

Both columns were measured after the missing answerer was fixed, so the two faults do not overlap and the jump belongs to the fence alone.

The local models do not move

This is the part worth keeping. The same fix, measured on the two local weights the 32B was being compared against:

Identical, which is the answer the theory predicts.

That is the control, and it is meant to show nothing. Local weights do not wrap their code in a fence — zero of twenty recorded turns contain one — so stripping a fence they never write cannot change what they score. The number that moved is the number that was being corrupted.

The control is worth reading for a second reason. The same model on the same code scored 14 of 20 in the note before this one and 10 of 20 here. Nothing changed between them but the night. Tier 3 is two cases, and two cases cannot resolve anything smaller than the effect measured above — which is why the fence result is believable and why a future comparison between two models should run all fifteen.

What kind of bug this was

Neither fault was a bug in the model, and neither was visible while only local models were measured:

• The benchmark supplied nobody to answer a question, so a run that stopped to ask scored a failure. llama3.1:8b asks once in twenty runs and barely noticed. qwen2.5-coder:7b asks eleven times in twenty, and nearly doubled once somebody answered.
• The parser fed markdown to Python. Local weights happen not to fence their code. Every hosted chat model does.

Both have the same shape: the harness had quietly specialised to the one model it runs itself. Each fault was invisible from inside that choice and appeared the moment something else was plugged in. That is the same class as a rule that only misfires on somebody else’s repository, and it is worth looking for more of them before the next comparison, not after.

What it cost, and what it bought

Two days of model comparisons produced no fact about any model. Every per-tier number published before 6 September is unsafe, and the two models, one wall note carries a warning saying so.

Against that: the fine-tune evaluation was going to run on this instrument. It would have punished the trained model for asking, fed it markdown, and reported both as capability. A wrong answer to should we train is worth more than two days.

What is still not known

Tier 3 is now saturated for the 32B, so it says nothing about the wall. The wall is at tier 6 — env-flag, venv-python, read-env-file, retry — where two 7–8B models stopped at the same place. Whether size clears that is the open question, and it is now askable for the first time.

Reproducing this

HF_TOKEN=… python scripts/measure/bench.py --tier 3 --engine openai \
  --model Qwen/Qwen2.5-Coder-32B-Instruct --repeat 5
python scripts/measure/bench.py --tier 3 --model llama3.1:8b --repeat 10
python scripts/measure/bench.py --tier 3 --model qwen2.5-coder:7b --repeat 10

The fix is unfenced() in src/harness/act/parse.py (#283), applied to Find, Replace and Append. Two of its behaviours exist because mutation testing killed the first draft: anything after the closing fence is dropped, because a model that signs off with “That should do it.” otherwise puts that sentence in the file; and the first closing fence ends the code, so a reply showing a second example block does not swallow the prose between them.