0.5B exact-stdout eval
Question. If the 0.5B writes a short script and we run it, how often does stdout match? Does one traceback repair help?
Answer. 7 / 54 base. 12 / 54 after one repair. The 0.5B is still not daily work. Repair is worth keeping. More style pairs are not.
Cite this note: Cite. Related: 0.5B sample-and-run · 0.5B vibe review · Fine-tune or harness · Experiments.
How this was scored
5 September 2026. One laptop. Ollama qwen2.5-coder:0.5b (untuned base,
not the LoRA). Eighteen prompts that are not in the 45 train pairs.
Each task three times. A run counts only when the extracted script exits
0 and stdout matches the expected line (trailing newline ignored).
The unit tests for the checkers all passed: every reference script scores, junk output fails. The failures below are the model, not the harness tests.
LoRA variants were not run on Ollama. The same-day MLX pair (four drafts, then greedy, with and without the step-100 LoRA) is 0.5B sample-and-run. Engines differ, so the rates are directional, not a paired A/B.
The score
| Variant | Passed | Rate | Typical miss |
|---|---|---|---|
| base | 7 / 54 | 13% | crash, or extra words |
| base + one traceback repair | 12 / 54 | 22% | still extra words |
Repair lifted the total. It did not make the model reliable. Verdicts still flip on the same prompt: fizzbuzz 2/3, hhmmss 2/3.
What failed
Two classes, almost even on the base runs:
| Class | Base runs | What it looked like |
|---|---|---|
| Nonzero exit | 24 / 54 | Often NameError: sys is not defined — the script used sys.argv and never imported sys. Also TypeError on %, IndexError on argv |
| Wrong stdout | 23 / 54 | The number was right, the line was not. Clamped value: 10 instead of 10. The median is: 3 instead of 3 |
| Pass | 7 / 54 | fizzbuzz, hhmmss, and a few others, not every repeat |
Repair then failed 42 of 54: 22 wrong stdout, 20 still crashing.
The 0.5B can write a FizzBuzz. It does not print only what it was asked
to print, and it often forgets the import that its own argv read needs.
Per task
Passes out of three repeats.
| Task | Base | + repair |
|---|---|---|
| fizzbuzz | 2 | 3 |
| hhmmss | 2 | 2 |
| fib | 1 | 2 |
| unique-order | 0 | 2 |
| rotate | 1 | 1 |
| wrap | 1 | 1 |
| median | 0 | 1 |
| weekday | 0 | 0 |
| count-ext | 0 | 0 |
| clamp | 0 | 0 |
| slugify | 0 | 0 |
| palindrome | 0 | 0 |
| sum-even | 0 | 0 |
| csv-col | 0 | 0 |
| indent4 | 0 | 0 |
| anagram | 0 | 0 |
| iso-date | 0 | 0 |
| relpath | 0 | 0 |
Eleven of eighteen tasks never passed, even with a repair.
Decision
Freeze the 0.5B adapters. Keep generate → run → one repair in the loop. Daily work stays an 8B. Do not add more handwritten style pairs to chase these eighteen scripts. The MLX follow-up put a number on that: greedy LoRA scored 0 / 54. Four untuned drafts plus one repair scored 9 / 18. The later loop scored 12 / 18 with zero hint-repairs.