跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分40

MUON-RT 重用教師軌跡,實現面向 LLM 自蒸餾的無動量 Muon

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

AI 導讀

Muon-RT 重用教師與學生的位移,免除隱藏權重矩陣的獨立動量緩衝區。在 Qwen2.5-7B-Instruct 的 SDFT 中,其平均準確率較 AdamW 高 2.6 點、持久記憶體用量少 42%;較 momentum Muon 高 5.4 點、記憶體用量少 27%。在 Olmo-3-7B-Instruct 的 SDPO 中,平均準確率較 AdamW 高 2.0 點。

正文

Published on Oct 30

Authors:

,

Abstract

Muon-RT eliminates separate momentum buffers by reusing teacher-student displacement for hidden weight matrices, reducing memory while improving fine-tuning and policy optimization accuracy.

Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices. We show that this displacement encodes an exponentially weighted history of applied updates and smooths temporally uncorrelated noise under a local model. Muon-RT removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton-Schulz (NS) orthogonalization. Under explicit regularity and alignment assumptions, we establish linear contraction of an expected objective-gap bound toward a residual error level. With Self-Distillation Fine-Tuning (SDFT) on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less persistent memory. With Self-Distillation Policy Optimization (SDPO) on Olmo-3-7B-Instruct, it improves average accuracy over AdamW by 2.0 points.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.41104

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.41104 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.41104 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.41104 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co