跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-23AI 評分46

透過逐層誤差模型搜尋提升基於 SVD 的 LLM 壓縮

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

AI 導讀

LEMS 逐層誤差模型搜尋與 KFAC-SVD 透過改善秩分配及分解品質,提升大型語言模型的低秩 SVD 壓縮效果。在 Mistral、Qwen3 和 Llama 3 系列中,LEMS 的零樣本準確率提升最高達 4.8 p.p.,並可延伸至 70B 參數規模的模型;KFAC-SVD 的平均困惑度改善 15%。

正文

Published on Oct 23

Authors:

,

,

,

,

,

,

,

,

Abstract

Layer-wise error modeling and token-wise Fisher statistics improve low-rank SVD compression of large language models by refining rank allocation and decomposition quality.

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project page and code: https://lems-svd.github.io

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.32878

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 23

MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6

Text Generation •

5B •

Updated 8 days ago

•

252

•

1

MoritzMo123/kfac-svd_lems_Qwen3-8B_0.9

Text Generation •

7B •

Updated 8 days ago

•

210

MoritzMo123/kfac-svd_lems_llama-3-8b_0.8

Text Generation •

7B •

Updated 8 days ago

•

217

MoritzMo123/kfac-svd_lems_mistral-7b_0.9

Text Generation •

7B •

Updated 8 days ago

•

202

Browse 23 models citing this paper

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.32878 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.32878 in a Space README.md to link it from this page.

Collections including this paper 2

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co