py-harness

ask, test, fix, add

GitHub

Results

Every score on this site comes from one laptop. Open one note. Each note is one question, what I typed, and what happened.

Not ready for daily work. That bar means beating a plain 8B both at picking the next step and at fixing a real bug the helper cannot do itself. It does not, yet.

Public pages do not name other editors or chat products.

9 / 98B daily jobs, evening of 5 Sep
7 / 97B coder, same jobs
8 / 158B first-step reading
0 / 54tiny 0.5B LoRA, greedy

Read these first

The scores

Experiments — paper form: abstract, methods, every measured run. 0.5B is 500 million weights.

Which model — keep llama3.1:8b. A 7B coder is close, not better.

Hub models — which Hugging Face weights fit 18 GB, and how to import two that Ollama does not ship.

How to use the tool

What you type — the four jobs on demo/orders.

First-run four — those jobs failed, then the helper finished them.

Cite — APA and BibTeX for this software.

References — papers the design sits on.

Which model

Note In one sentence
Which model Evening daily: 8B 9/9, 7B coder 7/9. 8192-warmed SWE daily clamp: first generate 180s. Keep the 8B.
The instrument was broken A day comparing models found two faults in the benchmark instead. Every model number before this is unsafe.
The fence was the whole story The hosted 32B was never the problem. 1 of 10 became 9 of 10 once four backticks stopped reaching the Python parser.
The wall two local models share Tier six: 18 of 20 hosted 32B, 8 and 11 for the two local models. They are level; the gap to the larger one is not.
What the totals were hiding 9 of 20 against 10 of 20 looked like noise. It was two opposite effects, one of them a wrong function written with no model. Now 19 of 20.
A day of repairs Nine fixes, 51 of 75 to 64 of 75. Almost none of it was the model. Two moved nothing and are kept anyway.
Where py-harness stands The status board: what is fixed, what is open, and what to do next, ranked by harm.
Two models, one wall The same 75 jobs. 51 vs 50. They fail in opposite ways.
Model lanes Which local weight for a question, a write, or a ship. Default stays 8B.
Hub models 8192-warmed SWE daily clamp: first generate 180s. Do not switch.
Cloud weights A larger model on a rented GPU. The helper stays on this machine.
Everyday laptop Why the 0.5B adapter is a demo, not daily work.
Bench record The machine, what fits in 18 GB, and the runs behind the numbers.

What we measured

Note In one sentence
Experiments Paper form. 0.5B means 500 million weights, not the daily 8B.
0.5B exact stdout 18 scripts, three times each. 7 / 54, then 12 / 54 after one repair.
0.5B sample-and-run Four drafts found 9 / 18. A later loop 12 / 18. The adapter at greedy temperature: 0 / 54.
0.5B vibe review Held-out short scripts, and a 100-file walk that was not a review.
First-run four 0 / 4 by hand, then 4 / 4 once the helper did the compiler jobs.
Same jobs, same evening Eleven demo tasks. Laptop 8B vs a hosted IDE agent.
Local loop vs hosted agents Every shipped path against a hosted IDE agent.

Where it fails

Note In one sentence
Where the failures are A third of runs fail. Most of those wrote the wrong code.
When a run says done and means nothing Five in ten claimed success having written nothing. After two fixes, none did.
What the helper cannot fix Cases where the helper knew the answer and the model still missed.
Asking a bigger model A run that stops to ask has already failed, three times in three.
Small steps, measured Splitting one hard job into easy ones bought nothing.
What to improve Gaps a helper can close, and gaps it cannot.

How it is built

Note In one sentence
Fine-tune or harness Do not train on 35 pairs or 30 seed traces. Later, about 2k clean turns.
Small models, classic development How an 8B finishes careful work: checks, not hope.
Harness comparison What transfers from other published helpers. No free shell.
Everyday skills Each skill is one copy-paste step, written for an 8B.
Skills The twenty-four kit skills and when each one loads.
Platform engineering Small files that must work on every OS.
Architecture Layers from the bottom up. A cycle fails the merge gate.