LowRAM:以低秩鍵與攤銷值實現可組合的測試時 KV cache 壓縮
LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
LowRAM 提出一種測試時 KV cache 壓縮方法,以低秩分解壓縮鍵,並用小型逐頭網絡根據鍵預測值。該方法將值快取壓縮 4×,效能接近無損;在 262K 上下文及 4× KV 壓縮下,選擇性重建令峯值解碼吞吐量最高提升 5×。結合量化與淘汰後,壓縮率達 26.9×,並保留基準效能的 94.5%。
We are excited to share LowRAM, our new KV cache compression method at test-time!
LowRAM compresses keys using low-rank factorisation, while small per-head networks predict values from keys. We then keep residual corrections for the hardest-to-reconstruct tokens.
This achieves 4× value-cache compression with near-lossless performance. With selective reconstruction, we also reach up to 5× higher peak decode throughput at 4× KV compression on at 262K context!
Because LowRAM exploits redundancy between keys and values, it adds a different axis of compression alongside quantisation and eviction. Combining all three lets us push compression to 26.9×, while retaining 94.5% of baseline performance.
We see promising applications in long-document QA and serving multiple queries over a shared context, where the predictors can be fitted once and reused. We’d love to hear what you think!
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co