Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads
Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads
研究以 HotpotQA 和 MuSiQue 比較智能體 RAG 的評測預算分配,涵蓋問題數、搜尋軌跡及重複作答。在 34.14--34.39M 模型 tokens 的預算下,增加問題覆蓋率,較重複讀取五次的標準誤低 33%,較三條搜尋軌跡低 12.6%。按記錄的模型費用,搜尋價格為每 1,000 次請求 $0--1 時,增加問題數優於增加搜尋軌跡;增加問題數或重複讀取的費用排名則未有定論。
Published on Oct 4
Authors:
,
Abstract
Evaluation budgets in agentic retrieval-augmented generation span questions, search trajectories, and repeated answers. We measure allocation precision, reading efficiency, and cost boundaries using a retrieval-feedback comparison on HotpotQA and MuSiQue. At 34.14--34.39M model tokens, broader question coverage lowers standard error by 33\% versus five reads and 12.6\% versus three trajectories. Archived nested and Q-only forecasts predict these allocations within 4.0\% and 3.5\%, respectively. Depth subsets establish no clear forecasting advantage beyond the two-trajectory audit. One-read variance penalties relative to the fitted optimum at the same token budget are 0--9.9\%, with substantial Pro uncertainty. Under recorded model fees, more questions beat more trajectories at search prices of \$0--1 per 1,000 requests; question-versus-read fee rankings remain unresolved. Temperature zero cuts answer disagreement from 14.3\% to 3.4\% while comparison precision stays similar. \par\medskip\noindentKeywords: Agentic RAG; Evaluation budget; Generalizability theory; Repeated sampling.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.05034
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.05034 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.05034 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.05034 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co