DAMI:用於提升門控線性注意力長上下文檢索的雙軸記憶初始化
DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
DAMI 為 GDN、GDN2、KDA、Mamba-2 和 RWKV-7 提出雙軸記憶初始化方法,聯合設定控制衰減係數的 A_log 與 dt_bias 初始值。100M 參數、5B tokens 的預訓練實驗中,五種架構的 RULER 檢索準確率平均較隨機初始化高 5.29–16.71,整體能力基本不變。這種方法無需增加參數或更改運算子,也不影響推理。
DAMI 👇 — a fairly systematic study of the initialization of decay-coefficient-related parameters in mainstream linear attention architectures (GDN, GDN2, KDA, Mamba-2, RWKV-7). Comments and discussion are very welcome
Motivation: Linear attention replaces the KV cache with a fixed-size state. Inference gets cheaper, but the old problem of "failing to remember far away" persists, and one of the key control components is the decay coefficient. Through 100M-parameter, 5B-token pretraining experiments, we found that after training, the correlation between the A_log and dt_bias that determine the decay coefficients and their initial values is above 0.9 — the decay-memory structure is essentially fixed at initialization. So how do we set these two parameters correctly before training? We carried out a series of theoretical derivations and controlled experiments, and arrived at DAMI.
DAMI: We spread A_log, which governs the timescale, across heads as a geometric progression (4096 → 1 token), and make dt_bias, which governs input sensitivity, vary along model depth in a "low–high–low" profile (0.25 → 0.75 → 0.25); the two are solved jointly to obtain the initialization values. In practice it requires only a small batch of activations, layer-wise gradient-free forward passes, and a bisection solve — no extra parameters, no operator changes, no impact on inference. The result: across five architectures (GDN / KDA / GDN2 / Mamba-2 / RWKV-7), average RULER retrieval accuracy is 5.29–16.71 higher than random initialization (GDN 30.42 → 44.36, KDA 26.43 → 43.14, GDN2 30.33 → 43.77, Mamba-2 13.7 → 28.2, RWKV-7 23.9 → 29.2), while general capabilities remain essentially unchanged.
Limitations: We only studied the decay coefficients, not the write/update coefficients; larger data scales and model sizes still need validation.
📑 Paper: https://huggingface.co/papers/2610.41994
📎 Blog: https://speedai.com/research/dami
💻 Code: https://github.com/SpeedAI-Lab/DAMI
⚖️ Weights: https://huggingface.co/SpeedAI-Lab/DAMI
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co