UltraBERT:超越 BERT 預訓練的極限
UltraBERT: Beyond the Limit of BERT Pre-Training
UltraBERT 以單一編碼器在分類與檢索任務均達最先進水平,GLUE 得分為 91.0,BEIR 為 47.7。模型在同一批次為低、高遮罩比例分設 LM head,並將知識蒸餾目標限制於低遮罩比例;研究指出,較低遮罩比例有利分類,較高比例則有利檢索。
Published on Oct 10
Authors:
,
Abstract
Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.15028
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 3
Fill-Mask • 0.2B • Updated 26 minutes ago • 80 • 6dmis-lab/UltraBERT-150M
Fill-Mask • 0.4B • Updated 26 minutes ago • 101 • 5dmis-lab/UltraBERT-380M
Sentence Similarity • 0.2B • Updated 26 minutes ago • 105 • 4dmis-lab/UltraBERT-Embed-150M
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.15028 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.15028 in a Space README.md to link it from this page.
Collections including this paper 1
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co