長度成本低,競爭不然:注意力可預測 Qwen2.5-1.5B 在檢索中的上下文退化
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
在鍵值檢索任務中,Qwen2.5-1.5B 面對 32K tokens 中性文字時,準確率由 0.97 升至 1.00;若填入證據格式的陳述,準確率則降至 0.11 至 0.21。
Author here. Thank you for taking a look.
Short version: in a key-value retrieval task, 32K tokens of neutral text cost Qwen2.5-1.5B almost nothing (accuracy 0.97 to 1.00). Filling the same positions with statements in the evidence format drops accuracy to 0.11 to 0.21. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110.
Inside the model, the attention mass left on the evidence predicts the loss (up to 58% of variance over 126,473 trials), while log length explains at most 8%.
We also replicate the masked-filler protocol of Du et al. (2025) on MMLU, and show that in retrieval the same masking costs 46 accuracy points at 32K tokens while the visible filler costs none.
Limits: correlational (no intervention on attention), one synthetic task, models up to 3B. All trial-level data are on the Hub: https://huggingface.co/datasets/yliusci/length-is-cheap
Happy to answer questions or run checks people suggest.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co