py-harness

ask, test, fix, add

GitHub

References

To cite this software or a laptop score, use Cite. This page is the other direction: the publications the design and the measurements sit on.

None of these papers is a hidden dependency. The helper is original code. The models below are the weights the commands actually call. The rest is related work: same problem, different ruler.

Collected 6 Sep 2026 from the papers, not from a second-hand summary.

What the tool runs

Daily ask / run uses Ollama llama3.1:8b. The 0.5B style prior and the exact-stdout eval use Qwen2.5-Coder. The adapter method is LoRA.

Grattafiori, A., et al. (2024). The Llama 3 herd of models. https://arxiv.org/abs/2407.21783

Hui, B., Yang, J., Cui, Z., Yang, J., et al. (2024). Qwen2.5-Coder technical report. https://arxiv.org/abs/2409.12186

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. ICLR. https://arxiv.org/abs/2106.09685

The 0.5B files on the Hub are a style prior. They are not a daily agent. Scores: Which model · 0.5B exact stdout.

The helper

The loop is one Action: block, then tools. That is a tight read-act loop, not a free shell.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. ICLR. https://arxiv.org/abs/2210.03629

Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. NeurIPS. https://arxiv.org/abs/2405.15793

Previous best retrieval-only: 3.8%. SWE-agent: 12.5%. Same model: retrieval 1.3% → agent 12.5%. py-harness’s first-run four jobs were 0 / 4 then 4 / 4 after the helper. Same shape, smaller tree. First-run four.

Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., & Ji, H. (2024). Executable code actions elicit better LLM agents. ICML. https://arxiv.org/abs/2402.01030

That paper lets the model emit Python as the action. This project does the opposite: named actions, a write limit, no general shell.

Later benches treat the helper as a variable, not a footnote.

Yao, Y., Tan, X., Liu, C.-H., Li, Y., Wang, Z., Yu, W., Tan, Z., Tian, Y., Zhao, G., Sun, L., Zhang, X., & Yang, T. (2026). Harness-Bench: Measuring harness effects across models in realistic agent workflows. https://arxiv.org/abs/2605.27922

Zheng, M., Han, K., Li, B., Xu, H., Tian, Y., He, W., Zhou, H., Guo, J., Hu, H., Ma, L., et al. (2026). Claw-SWE-Bench: A benchmark for evaluating OpenClaw-style agent harnesses on coding tasks. Same backbone: thin adapter 19.1% Pass@1, full adapter 73.4%. Helper sweep 27.4 points, model sweep 29.4. https://arxiv.org/abs/2606.12344

Repair and oracles

run writes, then runs the tests, then may send one traceback back. A green suite that never called the bug is not done.

Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. NeurIPS. https://arxiv.org/abs/2303.11366

Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023). Is your code generated really correct? Rigorous evaluation of large language models for code generation. NeurIPS (EvalPlus / HumanEval+). https://arxiv.org/abs/2305.01210

McAndrews, C. J. (2026). Feedback over form: Why execution feedback matters more than pipeline topology in 1–3B code generation. Laptop study: generate, run, refine. Fixes NameError and SyntaxError. Rarely fixes logic errors. https://arxiv.org/abs/2604.21950

Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? CCS. https://arxiv.org/abs/2211.03622

The field’s ruler

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? ICLR (oral). https://arxiv.org/abs/2310.06770

This project does not report a SWE-bench score. The public numbers are four jobs on demo/orders and a 4,580-file write rate of 1 / 12. SWE-bench is the field’s ruler. It is the wrong ruler for a one-folder laptop helper.

Related later benches (not run here):

Small models

Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., Lin, Y. C., & Molchanov, P. (2025). Small language models are the future of agentic AI. https://arxiv.org/abs/2506.02153

Agrees at 7–8B. Does not claim a 0.5B style adapter is an agent. Laptop split: 8B 9 / 9 daily; 0.5B vibe 0 / 4; greedy LoRA 0 / 54.

Kon, P. T. J., Pradeep, A., Chen, A., Ellis, A. P., Hunt, W., Wang, Z., Yang, J., & Thompson, S. (2026). SWE-Protégé: Learning to selectively collaborate with an expert. A 7B coder plus SWE-agent plus rare expert calls: 42.4% Pass@1 on SWE-bench Verified. https://arxiv.org/abs/2602.22124

That is a different job: train a 7B to ask a larger model. Daily py-harness stays on one local 8B.

Lee, W., Cho, J., & Choi, J. (2026). MapCoder-Lite: Distilling multi-agent coding into a single small LLM. Findings of EACL. Agent-wise LoRA on a 7B, not a 0.5B. https://aclanthology.org/2026.findings-eacl.346/ https://arxiv.org/abs/2509.17489

Sharma, R., & Mehta, M. (2025). Small language models for agentic systems: A survey of architectures, capabilities, and deployment trade offs. https://arxiv.org/abs/2510.03847

BibTeX

@article{grattafiori2024llama3,
  title = {The Llama 3 Herd of Models},
  author = {Grattafiori, Aaron and others},
  year = {2024},
  eprint = {2407.21783},
  archivePrefix = {arXiv}
}

@article{hui2024qwen25coder,
  title = {Qwen2.5-Coder Technical Report},
  author = {Hui, Binyuan and Yang, Jian and Cui, Zeyu and others},
  year = {2024},
  eprint = {2409.12186},
  archivePrefix = {arXiv}
}

@inproceedings{hu2022lora,
  title = {{LoRA}: Low-Rank Adaptation of Large Language Models},
  author = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  booktitle = {ICLR},
  year = {2022},
  eprint = {2106.09685},
  archivePrefix = {arXiv}
}

@inproceedings{yao2023react,
  title = {{ReAct}: Synergizing Reasoning and Acting in Language Models},
  author = {Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan},
  booktitle = {ICLR},
  year = {2023},
  eprint = {2210.03629},
  archivePrefix = {arXiv}
}

@inproceedings{yang2024sweagent,
  title = {{SWE}-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
  author = {Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir},
  booktitle = {NeurIPS},
  year = {2024},
  eprint = {2405.15793},
  archivePrefix = {arXiv}
}

@inproceedings{jimenez2024swebench,
  title = {{SWE}-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author = {Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
  booktitle = {ICLR},
  year = {2024},
  eprint = {2310.06770},
  archivePrefix = {arXiv}
}

@inproceedings{liu2023evalplus,
  title = {Is Your Code Generated Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
  author = {Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming},
  booktitle = {NeurIPS},
  year = {2023},
  eprint = {2305.01210},
  archivePrefix = {arXiv},
  note = {EvalPlus / HumanEval+}
}

@inproceedings{shinn2023reflexion,
  title = {Reflexion: Language Agents with Verbal Reinforcement Learning},
  author = {Shinn, Noah and Cassano, Federico and Labash, Beck and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu},
  booktitle = {NeurIPS},
  year = {2023},
  eprint = {2303.11366},
  archivePrefix = {arXiv}
}

@inproceedings{perry2023insecure,
  title = {Do Users Write More Insecure Code with {AI} Assistants?},
  author = {Perry, Neil and Srivastava, Megha and Kumar, Deepak and Boneh, Dan},
  booktitle = {CCS},
  year = {2023},
  eprint = {2211.03622},
  archivePrefix = {arXiv}
}

@misc{belcak2025slm,
  title = {Small Language Models are the Future of Agentic {AI}},
  author = {Belcak, Peter and Heinrich, Greg and Diao, Shizhe and Fu, Yonggan and Dong, Xin and Muralidharan, Saurav and Lin, Yingyan Celine and Molchanov, Pavlo},
  year = {2025},
  eprint = {2506.02153},
  archivePrefix = {arXiv}
}

@inproceedings{wang2024codeact,
  title = {Executable Code Actions Elicit Better {LLM} Agents},
  author = {Wang, Xingyao and Chen, Yangyi and Yuan, Lifan and Zhang, Yizhe and Li, Yunzhu and Peng, Hao and Ji, Heng},
  booktitle = {ICML},
  year = {2024},
  eprint = {2402.01030},
  archivePrefix = {arXiv}
}

@article{mcandrews2026feedback,
  title = {Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1--3{B} Code Generation},
  author = {McAndrews, Charles Junichi},
  year = {2026},
  eprint = {2604.21950},
  archivePrefix = {arXiv}
}

@inproceedings{lee2026mapcoderlite,
  title = {{MapCoder}-Lite: Distilling Multi-Agent Coding into a Single Small {LLM}},
  author = {Lee, Woongkyu and Cho, Junhee and Choi, Jungwook},
  booktitle = {Findings of EACL},
  year = {2026},
  url = {https://aclanthology.org/2026.findings-eacl.346/},
  eprint = {2509.17489},
  archivePrefix = {arXiv}
}

@article{yao2026harnessbench,
  title = {Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows},
  author = {Yao, Yilun and Tan, Xinyu and Liu, Chao-Hsuan and others},
  year = {2026},
  eprint = {2605.27922},
  archivePrefix = {arXiv}
}

How to cite py-harness itself: Cite. The measured scores: Results.