SIRIS:LLM 能否判斷模擬器誤差何時會影響 AI 系統研究?
SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research
SIRIS 是一項基準測試,旨在評估 LLM 能否判斷模擬器的限制是否會令系統設計建議失效。研究評估了 13 個前沿 LLM 在 LLM 服務設計問題上的表現,發現辨識出限制不代表能作出正確選擇,最強模型有時也會推翻有效建議。加入覆核回合有助模型恢復有效選擇,但也令部分模型出現新錯誤。
Published on Nov 6
Authors:
,
,
,
,
Abstract
A benchmark probes whether large language models can determine when imperfect simulator limitations invalidate recommended system designs, finding that identifying a limitation does not ensure correct design choices and that review-based self-correction yields mixed results.
LLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2611.10329
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.10329 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.10329 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.10329 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co