跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-08AI 評分46

何時需要 On-policy 蒸餾?以學生模型離線 rollout 蒸餾往往更佳

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

AI 導讀

研究提出 Semi-OPD,以初始學生模型生成的離線 rollout 進行蒸餾,往往比 OPD 更準確、訓練效率更高。在 17 組涵蓋 1.5B 至 235B 參數的教師—學生配對中,Semi-OPD 有 14 組勝出,準確度最高提升 +13.6%,訓練速度提升 11.4x。OPD 僅在教師與學生高度對齊、輸出 token 重疊率高時有利。

正文

Published on Oct 8

Authors:

,

,

,

,

,

,

,

,

,

Abstract

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.11291

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.11291 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.11291 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.11291 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co