跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 28 天前AI 評分41

SearchHorizonBench:評估 AI 智能體搜尋價值的基準測試

SearchHorizonBench: A Benchmark for Search Value in AI Agents

AI 導讀

SearchHorizonBench 評估 AI 智能體如何根據執行歷史估算額外搜尋的價值;歷史校準能提升預測準確度,但未能解釋大部分搜尋收益的差異。基準涵蓋 3,168 條問題的 45,632 次執行;在 1,632 條留出問題上,歷史校準將 Brier 誤差降低 16.2%,但測試中的歷史估算方法對搜尋收益差異的解釋不足 2%。

正文

Published on Oct 26

Authors:

,

Abstract

SearchHorizonBench evaluates how agents learn to estimate the value of additional search from execution history, showing that historical calibration improves prediction accuracy but leaves most search-gain variance unexplained.

AI agents must decide when another search is worth its cost. Answer-success probabilities capture question difficulty, while this decision requires estimating the additional benefit of search. We introduce SearchHorizonBench, a benchmark for learning and evaluating search value from execution experience. Paired continuations and repeated executions separate expected search gain from generation noise and connect prediction quality to decision utility. The resource contains 45,632 executions across 3,168 questions. On 1,632 held-out questions, historical calibration reduces Brier error by 16.2%, with the improvement concentrated in success-level information. Search gains have a standard deviation of 21.4 percentage points, and tested historical estimators explain less than 2% of this variance in point estimates. Matched-configuration learning curves probe this distinction across training scales. At the reference token price, free selection observations yield a 2.49-point utility opportunity over the best constant action, driven primarily by gain information. Reusable histories, frozen splits, strong baselines and scoring tools provide a common foundation for developing and evaluating value-aware search agents.

View arXiv page View PDF Project page GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2610.34714

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.34714 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.34714 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.34714 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co