Fine-tune or harness
Question. python-vibe’s everyday brain is an untuned 8B. The published adapter is a 0.5B style prior. Should we fine-tune again to make the loop behave like a bigger coding agent?
Answer. Not yet. Fine-tune after the harness already refuses a lie, and only on traces recorded inside this harness. Thirty seed rows and another 0.5B run will not buy agency. That is what this laptop measured, and it is what the 2026 small-model agent papers say when they separate the harness effect from the weight effect.
Related: small models, classic development · what to improve · hub models · everyday laptop.
What is on disk
| Artifact | Rows / steps | Role |
|---|---|---|
data/python-vibe/train.jsonl |
35 chat pairs | 0.5B QLoRA style prior |
data/python-vibe/valid.jsonl + test.jsonl |
5 + 5 | Same style, held-out wording |
Hub YauhenBichel/python-vibe-0.5b |
step 100 of 400 | Public download. Val ~0.91 then overfit |
data/agent-loop/train.jsonl |
30 Action traces | Seed for a 7B-class LoRA |
configs/python-vibe-8b.yaml |
200 iters, not run | train.py --everyday. No adapters on disk |
eval/action_prompts.jsonl |
12 prompts | Live parse. Not training data |
A LoRA on 35 short pairs teaches tone. A LoRA on 30 Action traces
teaches the first line of the protocol. Neither teaches “patch the
leftover name, then write a test that calls it, then refuse done.”
That last sentence is a harness job.
What this laptop already measured
29 Aug 2026, one machine, Ollama llama3.1:8b unless noted.
| Measurement | Result | What a new LoRA would change |
|---|---|---|
| 0.5B held-out vibe (weekday, count-md, jsonl, docstring) | 0 / 4 | Already tried. Base 0.5B failed the same class |
| 0.5B sample-and-run (18 scripts, MLX, 5 Sep 2026) | four drafts 6 / 18 base, 9 / 18 with repair; later loop 12 / 18 with 0 hint-repairs; greedy LoRA 0 / 54 | The adapter lost to the untuned base. An 8B note did not make the 0.5B debug itself. Freeze step 100 |
0.5B parsed Action: that day |
0 / 2 | Misses the protocol. More 0.5B steps overfit |
| 8B first Action on three scoped tasks | 3 / 3 | Start is often right without a LoRA |
| 8B live Action parse | 2 / 3 early; 7 / 10 later the same day | The prompt and the file opened first, not the weights |
scripts/run/demo.py on demo/orders |
Agent said done; independent check 1 / 4 file jobs | Finish is a lie. Oracles, not SFT |
| 7B-class everyday LoRA | Config only | Nothing to compare until traces exist |
| 30B coder on disk | Timeout at 180s | Bigger local weight is not the everyday path |
The failure that looks like “the model is too small” is usually “the
loop accepted done.” Live 8B left subtotl in src/orders.py after a
green suite that never called total_with_tax. It wrote def test_
into the implementation file. It renamed calc and left a NameError.
Those are compiler-shaped bugs. They are now refuses
(refuse_undefined_draft, refuse_test_in_impl, refuse_done_oracle).
A fine-tune that still emits Action: done after those mistakes would
lose to the current harness.
What the papers measure
Two 2026 results separate harness from fine-tune. They are not this repo. They are the comparison.
Specialized small-model subagents (Ranjan, 2026-06-11, slm-agents white paper). A 2×3 grid: {large, small, small fine-tuned} × {generic harness, custom harness}. File-navigation on held-out real repos. No paid API.
| Cell | What they saw |
|---|---|
| Large + generic (C1) | 72B succeeds 95.5% |
| Small + generic (C2) | Naive 3B swap: 48.0% success, worse cost-per-success than 72B |
| Small + custom harness (C3) | Harness alone can underperform the naive small model |
| Small fine-tuned + generic (C4) | Fine-tune alone can hurt. 1.5B scored 1.0% |
| Small fine-tuned + custom (C5) | 3B succeeds 94.5% (indistinguishable from 72B), 84.8% lower cost-per-success |
Only the pair wins. The 1.5B fine-tune scored 1.0% in the generic
harness and 93.5% in the co-designed one. A custom harness around a
model that cannot follow it, or a LoRA that speaks Action: with no
write limit, both lose. The 8B everyday brain is already capable enough that
harness adaptation (oracles, pinned Path:) is the Better-Harnesses
regime — not the 1.5B “C3 underperforms C2” regime.
Better harnesses, smaller models (arXiv:2607.08938). Seven routine agent tasks, three SLM families. An adapted harness (instructions, tools, anti-loop checks) improved 16 of 21 task–model pairs. Seven pairs closed the SLM–frontier gap. Best reported recovery: 89.7% of frontier performance at 4% of the cost. Adaptation worked when the workflow repeated and the base model was already capable enough.
SWE-smith / SWE-agent training. They fine-tune on resolved expert trajectories generated inside the same scaffold they evaluate. They do not train on thirty handwritten “Action: grep” lines and call the model a software engineer.
None of those papers argue for another 0.5B QLoRA on 35 stdlib pairs.
Map their grid onto this repo
| Their cell | python-vibe today | What “win” looks like |
|---|---|---|
| C1 large + generic | A hosted IDE agent | Out of scope. Different product |
| C2 small + generic | 0.5B or 8B with no write limit | Already measured: 0.5B 0/4 vibe; 8B says done with subtotl |
| C3 small + custom harness | Everyday path now. 8B is capable enough; oracles close finish | Finish the known demo fails without a new weight |
| C4 small fine-tuned + generic | train.py --everyday on 30 rows, then drop the write limit |
Do not do this |
| C5 small fine-tuned + custom | 7B/8B LoRA on ~2k verifier-clean --record turns |
Later. Same Action schema the harness already parses |
C3 is the work that moved this week: opening the right file before the
model’s first turn, pinning the Path: inside a skill, refusing a test
that does not set up its inputs, checking for names that are used but
never defined, and refusing paths that only work on one platform.
C5 is a real later lever — only after C3 stops accepting a green
suite that never called the bug.
The Ranjan C4 cell is the warning for python-vibe-8b.yaml: a LoRA that
speaks Action: in a plain chat window, with no file opened for it and
no checks around it,
can be worse than the untuned 8B you already run.
Decision
| Proposal | Do it? | Why |
|---|---|---|
| More 0.5B train steps | No | Overfit after step 100. Held-out vibe 0/4 |
Train python-vibe-8b on the 30 seed rows |
No | Memorizes the protocol line. Does not finish a change |
Raise --steps instead of oracles |
No | Burns tokens. Live add-feature already hit the budget |
| Add bash / browser as 8B tools | No | The write limit is the product. Papers that use bash use a container and a frontier model |
Keep refusing done until the compiler is quiet |
Yes | Matches C3. Matches the demo misses |
--record only turns the oracles already accept |
Yes | Future C5 data. Gitignored data/agent-loop/extra.jsonl |
| 7B-class LoRA after ~2k clean traces | Later | C5. Distill from a hosted agent or from a passing 8B loop in this schema |
Fine-tuning is required once the harness is the bottleneck — when the first Action is wrong on a task where the harness had already opened the right file, after the oracles are quiet, at a rate an 8B cannot close. That is not where this laptop is. The bottleneck is still finish.
How to fine-tune later without wasting a week
- Keep the Action schema stable. A LoRA trained on today’s
Action: patch/Find:/Append:is worthless if next month the protocol becomes JSON tool calls. - Record only turns that passed the oracles:
ast.parse, no new undefined names, tests not in the impl file, unittest that calls the change, design scan clean when the task asked for structure. - Redact hostnames and home paths. Do not commit
extra.jsonl. - Distill from a larger model in this harness (same opening step, same
refuses), not from a generic chat transcript. A 14B–70B that timed out
on the laptop is reached with
--engine openaiorOLLAMA_HOST; the write limit does not move. See cloud weights. - Evaluate with
scripts/measure/eval_everyday.py --liveandscripts/run/demo.py. Everyday-ready still means: beat the untuned 8B on parse and pass the independent file checks, including the planted NameError that the existing suite misses. - If a 7B LoRA loses to untuned 8B + current harness on those checks, delete the adapter. The papers already showed C4 can score 1%.
That is the investigation: behavior improvement is still harness work. Fine-tuning is the second half of a co-designed pair, not a shortcut around classic development.
Save money: next steps
The cheap path is local 8B and this write limit. A hosted IDE agent costs a usage pool. You only save that money when python-vibe finishes the job so you do not reopen the paid tool to clean up.
Use python-vibe when the tree is small and the job is one of: a typed question, add a function and a test, fix a NameError, rename, a pathlib / venv helper, a script, an HTTP client, a tally, an algorithm. Those are free once they finish.
Keep the paid tool for a large tree, extra servers, a browser, another language, or a hundred-file review. Those are not this product.
Order of work, cheapest first:
- This week. A unique NameError typo and a typed rename are
mechanical. The harness writes them, runs the suite, and ends
without a model when tests are already green. Live-retest
write-tests(still a model job) and any task that is not a unique typo. A passing suite that never called the change is still not a finish. - This month. Use it on your own small Python trees.
--recordonly turns the oracles already accept. Each finished job is money you did not spend. - Not this month. Do not train more 0.5B. Do not train
python-vibe-8bon 30 rows. A bad LoRA is a week of electricity that still sends you back to the paid tool. - Later. If the first Action is still wrong after the oracles are quiet, then a 7B LoRA on ~2k clean traces (C5). That is how the papers got 72B-class success at a fraction of the cost — not by skipping the write limit.