跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 10 天前AI 評分43

PaperCritic:訓練 AI 評審評估論文可靠性

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

AI 導讀

PaperCritic 以強化學習訓練,透過同時提升論文可靠性審查要點的精確率與召回率,在同領域測試中勝過前沿模型及其他訓練方法。研究以 1K 篇 ICLR 投稿及 23K 個人手撰寫和 LLM 生成的審查要點,建立 PaperSound-23K 數據集。模型亦能遷移至 Nature 系列論文審查,以及程式碼審查、法律合約審查和故事寫作回饋等任務。

正文

Abstract

An AI reviewer trained by reinforcement learning to jointly maximize precision and recall of soundness review points outperforms frontier models and other training methods and transfers to other review domains.

As AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2611.17834

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.17834 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.17834 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.17834 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co