重複使用數據的預訓練擴展:批次大小、正則化與 GPU 時間取捨
Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
研究分析訓練數據有限、需要重複使用時的預訓練,發現驗證損失與批次大小呈 U 形關係,較小批次較典型預訓練更受青睞,可能因梯度噪聲提供隱式正則化。每個 epoch 重新打包文件、權重共享等訓練配方亦可調整;期限較緊時較大批次較有利,但固定時間預算下,較小批次配較少 GPU 仍可取得比大批次配較多 GPU 更低的驗證損失。
Published on Oct 11
Authors:
Abstract
We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.15965
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.15965 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.15965 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.15965 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co