跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分37

輔助獎勵如何幫助小型語言模型從經驗證的推理軌跡中自我改進並評估遷移能力

Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

AI 導讀

研究比較信心指標 SC 與結合全局熵變化和局部平滑度的 UID-Conj.,評估輔助軌跡偏好對小型語言模型自我改進及遷移的作用。Qwen3-1.7B 在 30 道 AIME 題目上的在線實驗中,UID-Conj. 較未修改的 TTRL 在適應基準及三個遷移基準的宏平均高 2.08 個百分點;離線加權 SFT 中則較均勻加權高 1.93 個百分點。

正文

Published on Oct 30

Authors:

Abstract

Auxiliary trajectory preferences based on entropy and confidence metrics improve small language model self-improvement and transfer by distinguishing learning value among equally correct reasoning traces.

Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.41296

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.41296 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.41296 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.41296 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co