跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 21 天前AI 評分56

IdeaTaste:大語言模型的研究品味是否接近人類研究者?

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

AI 導讀

IdeaTaste 將 AI 智能體根據 2025 年文獻提出的研究構想,與同一社羣 2026 年發表的論文比較,發現其能跟上大方向,卻難以匹配具體方法。測試涵蓋六個當代語言模型、四個研究領域及逾 90,000 份提案;引用匹配約為人類水平的 26%,構想匹配僅 18%。

正文

Published on Nov 2

Authors:

,

,

,

,

Abstract

IdeaTaste benchmarks LLM research proposals against future publications, revealing that AI agents match human communities in broad directions but fall far behind in concrete approaches and expected success.

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.03424

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.03424 in a model README.md to link it from this page.

Datasets citing this paper 1

cx-cmu/IdeaTaste

Viewer •

Updated 17 minutes ago

•

275k

•

218

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.03424 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co