長時程基準測試是否真的夠深入?DEPTH 如何衡量及合成終端任務深度
Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
DEPTH 以命令序列之間的依賴深度而非回合數衡量終端任務,並據此合成更具挑戰性的基準測試。研究審核了六個現有數據集的 734 項任務,發現 94% 已解決的合成任務可在三回合內完成;DEPTH 合成的 105 項任務在深度和工作量上均超越受審核的數據集,六個現有模型沒有一個通過其中一半。以這些任務的執行記錄微調 Qwen3-8B,可提升其在兩項終端基準測試上的表現。
Published on Nov 1
Authors:
,
,
,
,
,
,
,
Abstract
DEPTH generates complex terminal tasks by measuring sequential dependency depth rather than turn count, producing harder benchmarks that improve agent fine-tuning.
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however, that the turn count largely reflects the habits of the model, not only the demands of the task, and that the labels barely track how often models pass. We describe a terminal task by its work, the number of operations it requires, and its depth, the number of turns that must happen in sequence because the commands of each turn can only be chosen after reading the output of the previous one. Estimating depth as the smallest turn cap, stated to the agent, under which the task passes, we audit 734 tasks from six existing datasets. Synthesis has mostly added work, not depth: 94% of the solved synthetic tasks pass within three turns, yet depth, unlike the turn count, predicts difficulty beyond work. We propose DEPTH, in which a generation agent writes each task one segment at a time in a real software environment, carrying out each segment before writing it and starting the next from the state it leaves. Our 105 tasks exceed every audited dataset in depth and work, and none of six current models passes half of them. Fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.01927
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.01927 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.01927 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.01927 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co