跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分39

LowRAM:以低秩鍵與攤銷值實現可組合的測試時 KV cache 壓縮

LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

AI 導讀

LowRAM 提出一種測試時 KV cache 壓縮方法,以低秩分解壓縮鍵,並用小型逐頭網絡根據鍵預測值。該方法將值快取壓縮 4×,效能接近無損;在 262K 上下文及 4× KV 壓縮下,選擇性重建令峯值解碼吞吐量最高提升 5×。結合量化與淘汰後,壓縮率達 26.9×,並保留基準效能的 94.5%。

正文

We are excited to share LowRAM, our new KV cache compression method at test-time!

LowRAM compresses keys using low-rank factorisation, while small per-head networks predict values from keys. We then keep residual corrections for the hardest-to-reconstruct tokens.

This achieves 4× value-cache compression with near-lossless performance. With selective reconstruction, we also reach up to 5× higher peak decode throughput at 4× KV compression on at 262K context!

Because LowRAM exploits redundancy between keys and values, it adds a different axis of compression alongside quantisation and eviction. Combining all three lets us push compression to 26.9×, while retaining 94.5% of baseline performance.

We see promising applications in long-document QA and serving multiple queries over a shared context, where the predictors can be fitted once and reused. We’d love to hear what you think!

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co