跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分45

rTaste:研究智能體有研究品味嗎?早期研究判斷評估

rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

AI 導讀

rTaste以7篇近期NLP論文測試六個前沿模型的研究智能體,評估它們提出和排序新實驗、設定門檻及預測結果的能力。415項實驗中,194項被評為可靠、有價值且具新意,當中只有73項同時超越種子論文且並非顯而易見。智能體首選比隨機選擇更常符合評審最看重的實驗,但預測表現未能證明優於只閲讀論文和實驗計劃的同模型實例。

正文

Do research agents have taste?

We gave six frontier models seven recent NLP papers and asked each, as a single agent and as a team, to use the paper as inspiration for novel research. Each proposed five small experiments, set a threshold for each, forecast whether it would pass, and ranked them.

A separate agent then attempted every experiment, including the ones ranked last, and three model auditors judged the results.

Of 415 experiments:

  • 194 were judged sound, valuable and new
  • only 73 of those were also beyond the seed paper and non obvious

The agents' first choices matched the auditors' most valued experiment more often than a random pick, but were not shown to beat the experiment they wrote first. Their forecasts beat a base rate, but were not measurably better than those of a second instance of the same model that read only the paper and the plans.

Reach out to me on X with any questions: https://x.com/artkanke?s=11

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co