跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分43

無需訓練的 MoE 專家剪枝:替換算子與路由語義

What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning

AI 導讀

研究比較無需訓練的 MoE 專家剪枝替換算子,發現閘門縮放及校準資料域會顯著影響效能,且沒有單一算子在所有設定中佔優。剪枝率為 50% 時,tilted novice 增加 1.2-1.8% 參數量,在 OLMoE 和 Moonlight 分別較常數替換提升 3.45 及 1.37 分。

正文

Published on Oct 30

Authors:

,

,

Abstract

Comparing replacement operators for pruned mixture-of-experts experts reveals that gate scaling and calibration domain strongly affect performance, with a proposed tilted novice operator improving results without dominating all settings.

Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.41366

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.41366 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.41366 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.41366 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co