S1CogBench:System One 決策模型在認知實驗中能做甚麼、不能做甚麼
S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
S1CogBench 評估 System One 決策模型在 24 項認知任務上的表現,揭示其能力與限制。評測涵蓋 27 種評估模式、七個開源模型及 API 模型 Jev。結果顯示,模型在部分學習任務表現良好,但可能未能蒐集資訊或因應回饋調整;快速決策亦未必有充分證據支持。
Published on Nov 14
Authors:
Abstract
S1CogBench evaluates System One decision models on 24 cognitive tasks, revealing that fast decisions often lack evidence-based reasoning and vary widely in reliability.
Decision-making in humans and AI involves multiple cognitive abilities. System One decision models produce fast, structured choices. Their cognitive strengths and limitations, however, remain poorly understood. We introduce S1CogBench, a benchmark of 24 established cognitive tasks and 27 evaluation modes, evaluating seven open models and the API model Jev. It combines human-response prediction, autonomous execution, and controlled task manipulations to reveal strengths and failures that aggregate scores miss. We find distinct behavioral profiles across the evaluated models. In these models, strong performance on some learning tasks coexists with failures to gather information or adapt to feedback, while high accuracy does not guarantee human-like responses to changing task demands. Shared readout controls show that model rankings depend partly on how choices are extracted. Separate latency comparisons with generative language models show that long responses are much slower than direct decisions, whereas matched one-token generation adds little overhead. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments. It provides a common empirical basis for identifying their capabilities and limitations before integrating them into automated decision systems.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.00911
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.00911 in a model README.md to link it from this page.
Datasets citing this paper 1
dengzhe-hou/S1CogBench
Viewer • Updated 9 days ago • 3.48k • 205 • 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.00911 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co