跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-23AI 評分47

TraceTrial-Bench:評測 AI 智能體 harness 的可靠性與失效歸因

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

AI 導讀

TraceTrial-Bench 評估 AI 智能體 harness 的可靠性,並追蹤執行軌跡、歸因失效。基準涵蓋七個組成部分,以 95 項任務測試 4 個 harness 和 9 個模型,共評估 3,420 次執行。harness 評分介乎 59.8% 至 72.0%,另有 31.1% 的執行雖未符合至少一項 harness 標準,仍成功完成任務。

正文

Published on Oct 23

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

TraceTrial-Bench evaluates agent harness reliability across seven components using mechanism-oriented tasks, trajectory tracing, and failure attribution to compare harness designs and diagnose failures.

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.32285

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.32285 in a model README.md to link it from this page.

Datasets citing this paper 1

TraceLite-AI/TraceTrial-Bench

Viewer •

Updated about 11 hours ago

•

95

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co