跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-13AI 評分35

Copy-Paste Language Models

Copy-Paste Language Models

AI 導讀

Copy-Paste Language Models(CPLMs)在傳統語言模型頭之外加入輕量級 Copy-Paste 頭,以明確建模提升大語言模型的語境複製能力。研究以 135M 至 1.7B 參數模型預訓練驗證,在不同規模及骨幹模型上均改善訓練效率與下游表現,並提升長上下文及檢索情境的表現,訓練和推理額外開銷可忽略。

正文

Published on Oct 13

Authors:

Abstract

Copy-Paste Language Models enhance in-context copying by adding a lightweight head that improves training efficiency, downstream performance, and long-context retrieval with minimal overhead.

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.19247

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.19247 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.19247 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.19247 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co