CogBattery:以人類為參照評估語言模型的認知任務行為與結構
CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
CogBattery 以 13 種認知範式的文字改編測試 54 個開放權重語言模型,並與各範式原研究的人類數據比較。模型在 n-back 誘餌上的反應方向與人類相同,但對一般非目標項目出現更多誤報;多數模型的信心區分正確與錯誤答案的能力不及人類,且多數模型總是選擇同一賭注,包括分類準確的模型。模型規模與人類相似度並不一致,預設認知分組的預測表現也未可靠勝過同規模的隨機分組。
Published on Oct 30
Authors:
,
,
Abstract
CogBattery compares open-weight language models to humans across cognitive tasks, revealing that matching effect directions does not imply human-like error patterns or confidence calibration.
Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
View arXiv page View PDF Project page GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2607.24999
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2607.24999 in a model README.md to link it from this page.
Datasets citing this paper 1
dengzhe-hou/CogBattery
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2607.24999 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co