Which model

Keep llama3.1:8b. A 7B trained on code is a little better at operations work and a little worse at the jobs this tool is for: write a test, add a small function, fix a bug. A 30B does not finish a single task on this laptop.

Related: model lanes · fine-tune or harness · two models, one wall · Hub models.

How this was measured

scripts/measure/bench.py runs tasks in six tiers and then runs the code it wrote. A case counts as working when the function does the job — not when a file appears.

Tiers 1, 2, 4 and 5 are the jobs this project is built for: write a test, add a small component, fix a bug.

The result

Measured on one laptop, 29 August 2026, through Ollama.

The 30B result is not a score. Every case ended in a timeout, so the model never produced a usable turn at all.

Same-night daily jobs, 5 September 2026

scripts/measure/eval_daily.py. Write a test, add clamp, fix a sum. Three repeats. Twelve steps.

The 7B coder stopped twice to ask where clamp should go. The logic-bug 3 / 3 on the 8B and 7B coder is a compiler return 0 bind, not the model writing the sum. A two-case gap is noise.

Write-tests 3 / 3 on the extra tags does not call the model. Clamp is the first generate. A cold load plus one reply burned the 180s Ollama cap. A second pass warmed each tag first (keep_alive 30 minutes) and finished. OpenCoder and StarCoder2: warmup got 0 bytes in 300s. SWE-agent-LM was in memory and still timed out on clamp. CodeLlama’s warmup returned, then clamp still hit 180s. DeepSeek got one clamp through, then the same cap. A one-word generate (Reply with the single word ok.) finished on the 8B (3.9 s), 7B coder (11.5 s), DeepSeek (3.8 s), and SWE-agent-LM (19.6 s). StarCoder2, CodeLlama, and OpenCoder hit 180s. The same four then finished the daily clamp text (no helper prompt) in 22–40 s. The real first helper chat is about 1,700 tokens and finished in 12–39 s on those four. A clean cold first turn (unload, then that chat) was 17 s on the 8B, 54 s on DeepSeek, and 38 s on SWE-agent-LM. keep_alive 0 did not evict the 7B coder. ollama stop did: DeepSeek first clamp passed (steps), second hit 180s. SWE-agent-LM from empty VRAM still hit 180s on the first clamp generate (the tag was loaded after). A follow-up ok generate while it was still listed hit 60s. After /api/ps was empty again, the same prompt finished in 6.8 s. The real first helper chat then finished in 14.5 s; daily first clamp on that loaded tag hit 180s. Two copies of the Agent body (no keep_alive) then hit 180s; one copy with keep_alive finished in 115.7 s. A remasure of two keep_alive POSTs from empty VRAM: first 180s, second 44.8 s. A concurrent bench.py --tier 3 on the 8B was holding /api/chat during those walls. That 8B bench ended; the same rerun then loaded qwen2.5-coder:7b. An idle local remasure (empty VRAM, no client on 11434): first Agent chat 180s, then SWE listed; second 2.15 s. A new bench.py --tier 6 then held the 8B and started the 7B coder. A later idle first chat with a 300s client cap still timed out and /api/ps stayed empty. After that a new bench.py --tier 3 on the 8B held /api/chat again. Once that ended, an idle one-word generate finished in 7.42 s (load 5.09 s). A later helper-sized Agent chat while the 8B was listed finished in 36.69 s. A second helper chat while SWE was still listed finished in 74.83 s (load 0.02 s, 1,700 eval tokens). Later an 8B client held /api/chat; empty helper was not remasured. A second local client then held the 8B again. After that an empty-VRAM helper chat finished in 43.45 s. One daily clamp job then ran 12 steps in 464 s with no def clamp (8B listed at start). An empty-VRAM remasure of the same job timed out on the first generate at 180 s; after that /api/ps listed SWE. A later one-word SWE load finished in 4.93 s, then the same daily job with SWE listed timed out on the first generate at 180 s; after that /api/ps was empty. A later Agent-body warm at num_ctx 8192 finished in 29.76 s, then the same daily job timed out on the first generate at 180 s; after that /api/ps still listed SWE. A reply is not a daily score. Do not switch. See Hub models.

One run is not a score

After this comparison was published the same nine cases were run six times against unchanged code:

9/9   6/9   8/9   7/9   8/9   7/9

Five of the nine pass every time, and three of those five finish without calling the model. Across the whole fifteen-case bench, ten of fifteen changed verdict between identical runs and the totals ranged from 7 to

1. Each model above was measured once, which is enough to show that the 30B never finished and not enough to separate llama3.1:8b from qwen2.5-coder:7b. Treat a gap smaller than about four cases as noise.

What that means

Do not switch. The 7B coder trades two of the four jobs you do most for one extra operations task. That is a bad trade whichever way the variance falls.

Do not read too much into single runs. The same model on the same task gives a different answer between runs. One case failed on spacing, "A L" where "AL" was wanted; another took nineteen minutes. Where this page gives a number, it is one run, and the direction matters more than the digit.

The finding underneath

Across the six tiers, the cases that pass reliably are the ones the harness finishes without asking the model at all:

cover-discount   yes   steps=0   0.2s
cover-shout      yes   steps=0   0.2s
fix-nameerror    yes   steps=0   0.1s

A misspelled name next to the right one, a missing import for a well-known module, a test that needs adding to a file that already has one — these are compiler jobs. Done deterministically, they cannot be got wrong, and they do not vary between runs.

Everything that still fails is the model reasoning badly rather than formatting badly: a flag reader that does not treat "0" as false, a file reader that returns None, a retry that never calls what it was given.

That is the case against reaching for training first. Fine-tuning on tool traces teaches a model to emit the protocol. The protocol is not where these runs fail.

Do not

• Do not quote a tier score as a capability. It is one run of a model that varies.
• Do not pull the 30B expecting the numbers to improve. It times out.
• Do not train on synthetic traces to fix reasoning errors. Record real ones first and look at what they actually contain.
• Do not switch to opencoder:8b or swe-agent-lm:7b until a daily table exists. They are not ollama pull tags; import them with scripts/weights/import_hf_ollama.py. See Hub models.