The question was whether a bigger model breaks the wall that two 7–8B models hit at the same place. Answering it needed comparisons, and the comparisons found faults in the thing doing the comparing.

Both faults were invisible while only local models were measured. Both would have made a fine-tune evaluation wrong.

A 14B does not run here

qwen2.5-coder:14b is 9 GB and pulled. It timed out on every case, three of three, which confirms the earlier record rather than overturning it: 9 GB of weights plus a growing key-value cache does not fit in 18 GB beside everything else. The local ceiling is 7–8B, and the two best candidates for that size have both been measured.

The benchmark punished asking a question

A run that stops to ask needs somebody to answer. The benchmark supplied nobody, so the question ended the run and scored a failure. How often a model asks turns out to vary enormously:

So the benchmark was measuring willingness to act without asking, and reporting it as capability.

The answer now given is deliberately empty: use the most likely reading, say which you chose, and continue. Answering properly would hand over the thing each case checks. A question costs a turn, which is what it costs a person who is not watching closely.

The second model nearly doubled. The gap between those two was published here as a difference in capability and was mostly this.

The benchmark fed markdown to the Python parser

With that fixed, the hosted 32B still scored one of ten, which is not a number a 32B produces. Watching one run turn by turn showed why. It writes drafts like this:

Action: patch
Path: src/orders.py
Append:
```python
def slugify(text: str) -> str:
    return text.lower()

The fence goes through unchanged, so what reaches the file is:

’python' 'def slugify(text: str) -> str:' ' return text.lower()' '’ -> SyntaxError: invalid syntax

By its third turn the model was reporting an unterminated string literal
in a test file — reading back the wreckage of its own earlier write.
Nine of ten runs then spent the whole step budget and produced nothing
that would load.

Local models happen not to fence their code. Every hosted chat model
does. So the harness has a fault that only appears the moment it is
pointed at a model it does not run itself, which is the same class of
thing as a rule that only fires on somebody else's repository.

## What this costs

Every model comparison published before today is unsafe. The local one
was distorted by the missing answerer; the remote one measured a parser
fault. Whether size breaks the wall is still unanswered, and the credits
spent bought the discovery of the second fault rather than the answer.

That is the better outcome of the two available. An instrument that
punishes asking and mangles fenced code would have made the fine-tune
evaluation wrong in the same two ways, and that evaluation was going to
be the basis for weeks of work.

## What has to happen before the question can be asked again

Strip the fence where a draft is parsed. Then re-run the three models on
the same cases, and only then read the result as being about model size.

## Reproducing this

```bash
python scripts/measure/bench.py --tier 3 --repeat 10
python scripts/measure/bench.py --tier 3 --model qwen2.5-coder:7b --repeat 10
HF_TOKEN=… python scripts/measure/bench.py --tier 3 --engine openai \
  --model Qwen/Qwen2.5-Coder-32B-Instruct --repeat 5

Each row now records how often the run asked, so the effect above can be seen rather than inferred.