跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分50

DAMI:用於提升門控線性注意力長上下文檢索的雙軸記憶初始化

DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

AI 導讀

DAMI 為 GDN、GDN2、KDA、Mamba-2 和 RWKV-7 提出雙軸記憶初始化方法,聯合設定控制衰減係數的 A_log 與 dt_bias 初始值。100M 參數、5B tokens 的預訓練實驗中,五種架構的 RULER 檢索準確率平均較隨機初始化高 5.29–16.71,整體能力基本不變。這種方法無需增加參數或更改運算子,也不影響推理。

正文

DAMI 👇 — a fairly systematic study of the initialization of decay-coefficient-related parameters in mainstream linear attention architectures (GDN, GDN2, KDA, Mamba-2, RWKV-7). Comments and discussion are very welcome

Motivation: Linear attention replaces the KV cache with a fixed-size state. Inference gets cheaper, but the old problem of "failing to remember far away" persists, and one of the key control components is the decay coefficient. Through 100M-parameter, 5B-token pretraining experiments, we found that after training, the correlation between the A_log and dt_bias that determine the decay coefficients and their initial values is above 0.9 — the decay-memory structure is essentially fixed at initialization. So how do we set these two parameters correctly before training? We carried out a series of theoretical derivations and controlled experiments, and arrived at DAMI.

DAMI: We spread A_log, which governs the timescale, across heads as a geometric progression (4096 → 1 token), and make dt_bias, which governs input sensitivity, vary along model depth in a "low–high–low" profile (0.25 → 0.75 → 0.25); the two are solved jointly to obtain the initialization values. In practice it requires only a small batch of activations, layer-wise gradient-free forward passes, and a bisection solve — no extra parameters, no operator changes, no impact on inference. The result: across five architectures (GDN / KDA / GDN2 / Mamba-2 / RWKV-7), average RULER retrieval accuracy is 5.29–16.71 higher than random initialization (GDN 30.42 → 44.36, KDA 26.43 → 43.14, GDN2 30.33 → 43.77, Mamba-2 13.7 → 28.2, RWKV-7 23.9 → 29.2), while general capabilities remain essentially unchanged.

Limitations: We only studied the decay coefficients, not the write/update coefficients; larger data scales and model sizes still need validation.

📑 Paper: https://huggingface.co/papers/2610.41994
📎 Blog: https://speedai.com/research/dami
💻 Code: https://github.com/SpeedAI-Lab/DAMI
⚖️ Weights: https://huggingface.co/SpeedAI-Lab/DAMI

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co