跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分47

Hybrid Computer Use:為通用智能體整合程式化、語義與視覺介面

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

AI 導讀

Hybrid Computer Use(HCU)整合程式化、基於無障礙功能的語義及視覺互動通道,讓通用智能體按需要選擇和組合不同方式處理電腦任務。在 OSWorld 2.0 上,HCU 的 Partial 和 Binary 評分均為受比較系統中最高;所有受評估模型使用 HCU 後都有所提升,並在效率及任務成功率上持續勝過比較基線。

正文

Published on Oct 30

Authors:

,

,

,

,

,

,

Abstract

Hybrid Computer Use unifies programmatic, accessibility-based, and visual interaction channels to improve agent efficiency and success on complex desktop tasks.

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.

View arXiv page View PDF Project page GitHub 10 Add to collection

Get this paper in your agent:

hf papers read 2610.41432

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.41432 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.41432 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.41432 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co