Termination Lag:編碼智能體解題後仍持續執行
Termination Lag: Coding Agents Solve Early but Stop Late
研究追蹤 5 個 LLM、2 種智能體框架及 3 個 SWE-bench 數據集,發現編碼智能體常在產生正確補丁後仍繼續執行。38–82% 的成功執行在首次原始碼修改時已得到正確補丁;核心設定下,成功執行中位數有 35–60% 的步驟是在正確補丁產生後執行。這些額外步驟主要用於驗證和探索,1–8% 的個案會把已正確的解答改壞;研究認為,改善停止判斷或可減少不必要計算。
What if coding agents already solved the problem but don't know when to stop?
We studied 5 LLMs, 2 agent frameworks, and 3 SWE-bench datasets, tracking and evaluating intermediate code patches throughout agent execution.
🔍 Key findings:
- 38–82% of runs that reach a correct patch get it right on their first source-code edit.
- A median of 35–60% of steps in successful runs occur AFTER the correct patch is produced in our core settings.
- Agents spend most of these extra steps verifying and exploring.
- In 1–8% of cases, agents even break an already-correct solution and finish with an incorrect patch.
These findings suggest an important opportunity: making coding agents recognize when to stop could significantly reduce unnecessary computation.
💬 How should coding agents decide when a solution is good enough to submit?
Github:
https://github.com/musacim/termination-lag
HF:
https://huggingface.co/datasets/musacim/termination-lag-trajectories
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co