0.5B sample-and-run

Question. On the same 18 held-out scripts, do four independent drafts at temperature 0.7 beat one greedy draft? Does the step-100 LoRA help, or only the loop (draw again, run, one traceback repair)?

Answer. The loop wins. The LoRA loses. Four drafts plus one repair scored 9 / 18. The later loop (prepend datetime, say when stdout is wrong, one 8B hint) scored 12 / 18. Zero of those twelve were a hint-repair. Greedy LoRA is 0 / 54.

Cite this note: Cite. Related: 0.5B exact-stdout eval · Fine-tune or harness · What to improve · Experiments.

How this was scored

5 September 2026. One laptop. MLX Qwen2.5-Coder-0.5B-Instruct-4bit. Same 18 prompts as the Ollama greedy note. None of those prompts are in the 45 train pairs. A run counts only when the extracted script exits 0 and stdout matches (trailing newline ignored).

Three MLX runs the same day:

Temperature is how randomly the next token is picked. It is not skill. 0 is almost the same script every time. 0.7 is the Qwen chat default; use it only when every draft is executed and junk is thrown away. Four drafts at temperature 0 are four copies of one script.

The earlier Ollama greedy run (no LoRA) is a different engine: 7 / 54 base, 12 / 54 with one repair. Rates are directional, not a paired A/B.

The greedy MLX run had a later loop tweak: if the last script NameErrors on sys or re, prepend the import and rerun once, and reject a repair whose first line is a traceback. That prepend did not fire. Greedy drafts never NameErrored on sys or re. weekday was still datetime. Treat the first two MLX grids as the same model, not as a harness A/B. The third run added datetime prepend and the 8B hint.

The two grids

Four drafts, temperature 0.7, one trial. First exact-stdout win counts.

Greedy, temperature 0, three repeats. Every repeat printed the same reason for the same task.

The 6/54 and 9/54 totals matching the four-draft 6/18 and 9/18 is a coincidence of counts, not the same score.

What sampling found

Four drafts found a different set, not a superset of greedy.

Greedy base locked clamp and fib (first draft, all three repeats). Four drafts on base found clamp, median, hhmmss, unique-order, wrap, relpath — and missed fib. fib only passed after a repair at temperature 0.7.

Wins that were not the first draw, base: clamp on draw 2, median on 3, unique-order on 3, wrap on 4, relpath on 2. hhmmss landed on draw 1.

base + repair added fizzbuzz on draw 4, rotate on 2, sum-even on 2 (those three were new draws, not a traceback fix) and fib as a true repair. Only one of the +3 from 6 to 9 is self-debug. The rest is “draw again.”

Greedy repair shipped weekday 3/3: first draft NameError: datetime, second draft printed Saturday. That task never passed on the four-draft grid. At temperature 0 the 0.5B can add one import. At 0.7 it did not.

LoRA at temperature 0 is a brick. weekday printed what day is this? plus 2026. Repair added it's not a leap year. Clamp, fizzbuzz, and slugify printed nothing. iso-date invented datetime.fromiso8601. Repair recovered zero tasks. At 0.7 the adapter lucked into hhmmss and palindrome, then four more with repair (count-ext, anagram, fizzbuzz, unique-order). It still lost clamp, median, wrap, relpath, rotate, fib, and sum-even to the untuned base.

Five tasks never passed on the first four-draft grid: weekday, slugify, csv-col, indent4, iso-date. Greedy later cleared weekday only. The later loop cleared weekday on the first draw (prepend datetime) and also count-ext, palindrome, and csv-col. wrap flipped off. n = 1; treat single-task flips as noise except weekday.

What failed

Dominant on base, four drafts: used sys.argv and never imported sys; called main() without defining it; printed extra words (Number of files ending with '.md' in .: 2 instead of 2); parsed 2026-09-05T17:27:00 with a space format.

Dominant on LoRA, four drafts: invented APIs (list.distinct(), import slugify), def count(sys.argv) as a SyntaxError, empty stdout with exit 0. Temperature 0.7 plus the style adapter widened the broken set, not the useful one.

Dominant on repair: the 0.5B sometimes pasted the traceback into the next file. A true repair that fixed a NameError was rare (fib on base, palindrome on LoRA, weekday on greedy).

Dominant on greedy base: extra words (count-ext), fizzbuzz stopped at 15, slugify echoed the raw string, rotate c d b, palindrome Yes, sum-even 21, indent4 left two-space padding, csv-col FileNotFoundError: '1', median NameError: main, iso-date int('05T17:27:00'). Three copies of each.

The 8B hint cell

Same 18, four drafts, temperature 0.7, one trial, base + repair. After a miss, llama3.1:8b writes one line from the traceback or from “exited 0 but stdout is wrong,” then the 0.5B rewrites once. The loop also prepends from datetime import datetime when that NameError appears.

12 / 18. Zero of the twelve are a repair pass. Every win was a clean first draft on some sample.

The +3 versus 9 / 18 is weekday (mechanical prepend) plus three sample flips in and wrap out. n = 1. The 8B note did not turn a failing last draft into a pass. That matches Asking a bigger model: the stuck point is worth a remote call only when the local model is already on the right file. Here the 0.5B was not.

Still down after the hint: slugify, indent4, anagram, wrap, iso-date, relpath.

Estimate of next steps

Ordered by expected unique-task lift on these 18, per hour of work. No new 0.5B train pairs. Daily run stays an 8B through the same guard.

The 8B-hint estimate was +3 to +6 unique if the leftover class was extra words. The leftover class is still extra words, ISO, and slugify. The 0.5B does not take the note. Stop.

What the 0.5B still will not do: daily Action: parse, a ≥1 KB fix, or slugify / indent4 / iso-date. Everyday-ready is still an 8B on live parse and a real fix.

Decision

1. Freeze the step-100 adapters. Do not add pairs to chase slugify or iso-date.
2. For 0.5B stdout smoke: untuned base, four drafts, one repair, plus the datetime prepend. The LoRA is style only.
3. Keep generate → run → one repair. Prefer a new draw over a second greedy repair. Do not wait on an 8B hint to make the 0.5B debug itself.
4. Stop spending hours on this 18-script board. Daily work stays an 8B. The 8B daily-jobs table on Experiments is 8 / 9.