跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-20AI 評分46

Stratified Sparse Attention:在極端稀疏度下縮小稀疏與密集註意力的差距

Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity

AI 導讀

Stratified Sparse Attention(SSA)結合確定性檢索與分數引導抽樣,在多項基準測試中提升長上下文推理的準確度與速度。於 RULER-HARD 上,Ministral-3-8B 在 50 倍注意力稀疏度下,32K 和 128K 的準確度分別由 81.8% 升至 84.2%、由 63.8% 升至 69.3%。

正文

Published on Oct 20

Authors:

,

,

,

,

,

Abstract

Stratified Sparse Attention unifies deterministic retrieval and score-guided sampling to recover attention mass, improving long-context inference accuracy and speed across multiple benchmarks.

Large Language Model inference over long contexts is increasingly bottlenecked by the cost of attending to a growing Key-Value (KV) cache. Sparse attention reduces this cost by accessing only a fraction of the cache, but existing methods face a trade-off between deterministic retrieval, which captures high-scoring tokens but discards residual attention mass, and sampling, which improves coverage at the cost of higher variance. Hybrid methods combine top-k retrieval with uniform sampling, yet still exhibit a substantial gap from dense attention at aggressive sparsity. We introduce Stratified Sparse Attention (SSA), a unified framework combining deterministic retrieval with score-guided sampling. SSA deterministically attends to high-scoring tokens while sampling the residual using proxy attention scores and correcting for their inclusion probabilities, thereby recovering attention mass beyond the retrieval set. SSA encompasses top-k, top-p, and hybrid top-k+sampling as special cases. Across RULER, LongBench, LOFT, and AIME, and five models ranging from 3B to 27B parameters, SSA consistently improves over the baselines at matched sparsity. On RULER-HARD with Ministral-3-8B, SSA improves accuracy over the strongest practical baseline from 81.8\% to 84.2\% at 32K and from 63.8\% to 69.3\% at 128K under 50times attention sparsity, recovering 94\% and 85\% of dense-attention performance, respectively. With optimized CUDA kernels, SSA achieves up to 2.96times decode speedup over FlashInfer at 1M context, demonstrating that its accuracy gains translate into practical long-context inference acceleration. We open-source our code at https://github.com/skylight-org/SSA-Paper-Code.

View arXiv page View PDF GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2610.28441

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.28441 in a model README.md to link it from this page.

Datasets citing this paper 1

Skylight1/stratified-sparse-attention

Updated about 22 hours ago

•

9

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.28441 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co