Results
Every score on this site comes from one laptop. Open one note. Each note is one question, what I typed, and what happened.
Not ready for daily work. That bar means beating a plain 8B both at picking the next step and at fixing a real bug the helper cannot do itself. It does not, yet.
Public pages do not name other editors or chat products.
9 / 98B daily jobs, evening of 5 Sep
7 / 97B coder, same jobs
8 / 158B first-step reading
0 / 54tiny 0.5B LoRA, greedy
Read these first
The scores
Experiments — every measured run, with what I typed.
Which model — keep llama3.1:8b. A 7B coder is close, not better.
Hub models — which Hugging Face weights fit 18 GB, and how to import two that Ollama does not ship.
How to use the tool
What you type — the four jobs on demo/orders.
First-run four — those jobs failed, then the helper finished them.
Cite — APA and BibTeX.
Which model
| Note | In one sentence |
|---|---|
| Which model | Evening daily: 8B 9/9, 7B coder 7/9. SWE first helper chat 14.5s; daily first clamp still 180s. Keep the 8B. |
| The instrument was broken | A day comparing models found two faults in the benchmark instead. Every model number before this is unsafe. |
| Two models, one wall | The same 75 jobs. 51 vs 50. They fail in opposite ways. |
| Model lanes | Which local weight for a question, a write, or a ship. Default stays 8B. |
| Hub models | SWE first helper chat 14.5s; daily first clamp still 180s. Do not switch. |
| Cloud weights | A larger model on a rented GPU. The helper stays on this machine. |
| Everyday laptop | Why the 0.5B adapter is a demo, not daily work. |
| Bench record | The machine, what fits in 18 GB, and the runs behind the numbers. |
What we measured
| Note | In one sentence |
|---|---|
| Experiments | The long table: example, score, how to replay. |
| 0.5B exact stdout | 18 scripts, three times each. 7 / 54, then 12 / 54 after one repair. |
| 0.5B sample-and-run | Four drafts found 9 / 18. A later loop 12 / 18. The adapter at greedy temperature: 0 / 54. |
| 0.5B vibe review | Held-out short scripts, and a 100-file walk that was not a review. |
| First-run four | 0 / 4 by hand, then 4 / 4 once the helper did the compiler jobs. |
| Same jobs, same evening | Eleven demo tasks. Laptop 8B vs a hosted IDE agent. |
| Local loop vs hosted agents | Every shipped path against a hosted IDE agent. |
Where it fails
| Note | In one sentence |
|---|---|
| Where the failures are | A third of runs fail. Most of those wrote the wrong code. |
| When a run says done and means nothing | Five in ten claimed success having written nothing. After two fixes, none did. |
| What the helper cannot fix | Cases where the helper knew the answer and the model still missed. |
| Asking a bigger model | A run that stops to ask has already failed, three times in three. |
| Small steps, measured | Splitting one hard job into easy ones bought nothing. |
| What to improve | Gaps a helper can close, and gaps it cannot. |
How it is built
| Note | In one sentence |
|---|---|
| Fine-tune or harness | Do not train on 35 pairs or 30 seed traces. Later, about 2k clean turns. |
| Small models, classic development | How an 8B finishes careful work: checks, not hope. |
| Harness comparison | What transfers from other published helpers. No free shell. |
| Everyday skills | Each skill is one copy-paste step, written for an 8B. |
| Skills | The twenty-four kit skills and when each one loads. |
| Platform engineering | Small files that must work on every OS. |
| Architecture | Layers from the bottom up. A cycle fails the merge gate. |