跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分54

CogBattery:以人類為參照評估語言模型的認知任務行為與結構

CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models

AI 導讀

CogBattery 以 13 種認知範式的文字改編測試 54 個開放權重語言模型,並與各範式原研究的人類數據比較。模型在 n-back 誘餌上的反應方向與人類相同,但對一般非目標項目出現更多誤報;多數模型的信心區分正確與錯誤答案的能力不及人類,且多數模型總是選擇同一賭注,包括分類準確的模型。模型規模與人類相似度並不一致,預設認知分組的預測表現也未可靠勝過同規模的隨機分組。

正文

Published on Oct 30

Authors:

,

,

Abstract

CogBattery compares open-weight language models to humans across cognitive tasks, revealing that matching effect directions does not imply human-like error patterns or confidence calibration.

Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.

View arXiv page View PDF Project page GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2607.24999

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2607.24999 in a model README.md to link it from this page.

Datasets citing this paper 1

dengzhe-hou/CogBattery

Updated 9 days ago

•

44

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2607.24999 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co