跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 25 天前AI 評分36

每個 Token 都需要自己的教師:多教師自適應 on-policy 自我蒸餾

Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

AI 導讀

研究提出多教師自適應 on-policy 自我蒸餾(MTA-OPSD),讓訓練中的每個 Token 都能在候選教師池中自適應選擇偏好的教師。Qwen3 模型的數學推理實驗顯示,MTA-OPSD 持續優於現有 OPSD 方法,相對基礎模型的平均增益最高達 +5.9。

正文

Published on Oct 29

Authors:

,

,

,

,

Abstract

Multi-teacher Adaptive On-policy Self-Distillation selects a preferred teacher for each token during training, improving mathematical reasoning performance over fixed-teacher methods.

On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to +5.9 over the base model, highlighting the importance of adapting teacher supervision at the token level. The code is available at https://github.com/xingzhejun/MTA-OPSD.

View arXiv page View PDF GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.40574

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.40574 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.40574 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.40574 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co