跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分42

Muon 令首個混合專家層更難平衡

Muon Makes the First Mixture-of-Experts Layer Harder to Balance

AI 導讀

使用 Muon 訓練混合專家(MoE)語言模型會加劇首個 MoE 層的路由失衡,其他層受影響較小。在活躍參數為 55M 和 108M 的模型中,首層偏置跨度在每個隨機種子下均上升,平均約增至兩倍;從路由器輸入中減去批次平均值可消除過載,而降低較小模型的 Muon 學習率亦能消除大部分增幅。此規模下,將路由器改用 AdamW 會令負載變化更大,結果較支持讓路由器繼續使用 Muon。

正文

Published on Oct 27

Authors:

Abstract

Using Muon to train mixture-of-experts language models increases routing imbalance in early layers, which can be mitigated by centering router inputs or reducing the optimizer learning rate.

Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.37914

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.37914 in a model README.md to link it from this page.

Datasets citing this paper 2

marcoshernanz/llm-lab-fineweb-edu-sample10bt-bpe-16384-full

Updated about 11 hours ago

•

304

marcoshernanz/tfg-runs

Viewer •

Updated about 11 hours ago

•

13

•

141

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.37914 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co