跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 28 天前AI 評分49

Co-Design Bench:面向智能體模型的多模態協作編碼評測

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

AI 導讀

Co-Design Bench 以真實用戶修改現有應用程式的請求,評測智能體模型理解多模態輸入並完成協作編碼的能力。基準包含 273 項任務及 2,331 項驗收準則,涵蓋多模態編輯和程式碼庫編輯。10 個模型端點中,gpt-6-astra 平均每項任務通過 69.9% 的準則,但僅 26.4% 的任務完全達標;Qwen3.8-27B 整體完全達標率為 64.4%。

正文

Can a coding agent turn a real user's multimodal request into a complete, working application revision?

We're sharing Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, a benchmark built around real users' requests to revise existing applications. Each task starts with a runnable application and asks the agent to understand the user's intent, make the necessary changes, and satisfy every acceptance criterion.

Real requests, multimodal inputs, and connected changes.

Co-Design Bench contains 273 tasks and 2,331 expert-authored acceptance criteria, curated from 37,463 deduplicated queries across 26,314 real user coding sessions. It covers two complementary tracks:

  • Multimodal Editing (100 tasks): localized revisions grounded in annotated screenshots and optional image, audio, video, or document inputs.
  • Repository Editing (173 tasks): coordinated changes across components, routes, state, data flow, and content.

Requests can involve reproducing distinct text animations from a video, correcting a dashboard from a receipt and updating its dependent totals, or coordinating layout and interaction changes across an application. Tasks span Grounded Understanding, UI Transformation, and Functional Implementation, often combining several capabilities within one request.

Evaluate the running application, with evidence for each requirement.

Our reference-free judge-then-refute protocol assesses hidden acceptance criteria using source code and browser evidence. A separate verifier challenges positive judgments for stubs, hardcoded outputs, unreachable functionality, and requirement mismatches. Evidence audits require screenshots for visual claims and recorded browser actions for interaction claims. No gold implementation or target screenshot is required.

Partial requirement satisfaction and complete delivery tell different stories.

Across 10 model endpoints under a shared agentic harness, the strongest endpoint, gpt-6-astra, passes an average of 69.9% of criteria per task, yet fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository requests spanning three or four edit types than on those spanning one or two.

Qwen3.8-27B reaches 64.4% overall strict completion, ranking fourth overall, and 64.5% on Repository Editing, ranking second on that track in this evaluation. It also leads the text-only Multimodal Editing subset at 77.9%. Rankings vary with the editing setting; the small repository-track score differences do not establish a general model advantage.

The ME-96 subset comprises 96 Multimodal Editing tasks and 521 acceptance criteria, a distinct scope from the full 273-task paper evaluation.

Explore the real cases, demonstrations, evaluation tools, and results:

Project & demos · Paper · Code & evaluation · Dataset

We welcome researchers and builders working on coding agents, multimodal grounding, and reliable application editing to try the benchmark and share their findings!

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co