跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 8 天前AI 評分57

SWE-CoWork:以組織情境基準評測編碼智能體

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

AI 導讀

SWE-CoWork 基準在模擬軟件組織中評測編碼智能體,涵蓋 101 項任務、101 個真實程式碼庫、11 種語言和 12 個應用領域。評測 11 個模型時,兩個領先模型雖通過約 63% 的失敗轉通過(F2P)測試,完整完成的任務卻少於 8%。向 Seed Evolving 和 GPT-6 Astra 預先提供完整需求,令 F2P 通過率提高約 16 個百分點,但完整任務成功率沒有改變。

正文

Published on Nov 15

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

A benchmark of realistic software engineering assignments reveals that current coding agents struggle to track requirements from clarification through implementation, verification, and handoff, completing few tasks despite partial test success.

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

View arXiv page View PDF Project page GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2611.20218

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.20218 in a model README.md to link it from this page.

Datasets citing this paper 1

Tokenwave-AI/swe-cowork

Viewer •

Updated about 3 hours ago

•

101

•

7

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.20218 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co