透過逐層誤差模型搜尋提升基於 SVD 的 LLM 壓縮
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
LEMS 逐層誤差模型搜尋與 KFAC-SVD 透過改善秩分配及分解品質,提升大型語言模型的低秩 SVD 壓縮效果。在 Mistral、Qwen3 和 Llama 3 系列中,LEMS 的零樣本準確率提升最高達 4.8 p.p.,並可延伸至 70B 參數規模的模型;KFAC-SVD 的平均困惑度改善 15%。
Published on Oct 23
Authors:
,
,
,
,
,
,
,
,
Abstract
Layer-wise error modeling and token-wise Fisher statistics improve low-rank SVD compression of large language models by refining rank allocation and decomposition quality.
Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project page and code: https://lems-svd.github.io
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.32878
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 23
Text Generation • 5B • Updated 8 days ago • 252 • 1MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6
Text Generation • 7B • Updated 8 days ago • 210MoritzMo123/kfac-svd_lems_Qwen3-8B_0.9
Text Generation • 7B • Updated 8 days ago • 217MoritzMo123/kfac-svd_lems_llama-3-8b_0.8
Text Generation • 7B • Updated 8 days ago • 202MoritzMo123/kfac-svd_lems_mistral-7b_0.9
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.32878 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.32878 in a Space README.md to link it from this page.
Collections including this paper 2
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co