跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 7 天前AI 評分38

VisualArtifactSpec 1.0:按任務規格評估視覺作品

VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications

AI 導讀

VisualArtifactSpec 1.0 是一項評測基準,涵蓋 211 項創作及修改任務,涉及簡報、靜態圖像、動畫與影片,以及網頁介面。它分別評估任務完成度和作品表現,後者包括內容、組織、視覺效果與交付;研究團隊以 9 種模型與測試框架配置評測全部任務,總分介乎 42.12 至 70.52。作品表現佔最高與最低配置分數差距的 73.4%。

正文

Published on Nov 16

Authors:

,

,

,

,

,

Abstract

Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.

View arXiv page View PDF Project page GitHub 6 Add to collection

Get this paper in your agent:

hf papers read 2611.21382

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.21382 in a model README.md to link it from this page.

Datasets citing this paper 1

Accio-Lab/VisualArtifactSpec

Viewer •

Updated 1 day ago

•

211

•

136

•

1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.21382 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co