OpenMOSS 公開 ABC-Bench 評測工具與完整數據集
OpenMOSS/ABC-Bench
OpenMOSS 於 2026/01/20 公開 ABC-Bench 評測工具及完整數據集,用於評估智能體在真實後端開發中的編碼能力。基準包含 224 項任務,取自 127 個 MIT 授權倉庫,涵蓋 8 種語言和 19 個框架;其中 92 項要求自主配置環境並啟動容器化服務。
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
📄 Paper | 🌐 Blog | 🤗 Dataset
ABC-Bench is a benchmark for Agentic Backend Coding. It evaluates whether code agents can explore real repositories, edit code, configure environments, deploy containerized services, and pass external end-to-end API tests (HTTP-based integration tests) across realistic backend stacks.
📰 News
- 2026/01/20: Evaluation harness and the full ABC-Bench dataset released on Hugging Face.
- 2026/01/19: Blog post released, detailing benchmark construction and baseline results.
- 2026/01/16: Public preprint released: ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development.
- End-to-End Lifecycle: repository exploration → code editing/implementation → environment setup → containerized deployment → external end-to-end API verification.
- Real-World Diversity: 224 tasks curated from 127 MIT-licensed repositories, spanning 8 languages and 19 frameworks.
- Environment-Aware Tasks: 92 tasks require autonomous environment configuration and containerized service startup.
- Automated Construction: built via ABC-Pipeline with minimal manual intervention, enabling scalable task creation and future expansions.
- Challenging Baselines: even state-of-the-art models remain far from fully reliable.
📊 Benchmark Composition
⚖️ Evaluation Protocol
💾 Dataset Access
Download the full benchmark (tasks, build assets, verification suites) on Hugging Face:
👉 🤗 OpenMOSS-Team/ABC-Bench
After downloading, set --dataset-path to the local dataset root directory.
⚡ Quickstart
1. Prerequisites
- Docker
- Python ≥ 3.10
2. Install Terminal-Bench CLI
pip install terminal-bench
Verify:
tb --help
3. Run Evaluation
Replace <DATASET_PATH> with your local dataset root directory downloaded from Hugging Face.
tb run \ --dataset-path <DATASET_PATH> \ --agent openhands \ --model openai/GPT-5 \ --n-attempts 3 \ --global-agent-timeout-sec 3600 \ --global-test-timeout-sec 1800 \ --n-concurrent 30 \ --run-id demo
🧠 SFT Models
We provide two models that have been Supervised Fine-Tuned (SFT) specifically for agentic backend coding tasks:
👉 🤗 OpenMOSS-Team/Qwen3-8B-ABC
👉 🤗 OpenMOSS-Team/Qwen3-32B-ABC
🤝 Contributing
Pull requests and issues are welcome. For substantial changes (new scripts, new baselines, major doc updates), please open an issue first.
📝 Citation
@misc{yang2026abcbenchbenchmarkingagenticbackend, title={ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development}, author={Jie Yang and Honglin Guo and Li Ji and Jiazheng Zhou and Rui Zheng and Zhikai Lei and Shuo Zhang and Zhiheng Xi and Shichun Liu and Yuxin Wang and Bo Wang and Yining Zheng and Tao Gui and Xipeng Qiu}, year={2026}, eprint={2601.11077}, archivePrefix={arXiv}, primaryClass={cs.SE}, url={https://arxiv.org/abs/2601.11077}, }
🙏 Acknowledgements
ABC-Bench is built from MIT-licensed open-source repositories. We thank the maintainers and contributors whose work makes realistic evaluation possible.
來源:OpenMOSS / 復旦NLP / 上海創智 / MOSI · github.com


