On-Policy Self-Training:大型語言模型在無需人工問答下自我改進
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
On-Policy Self-Training(OPST)將 on-policy self-distillation 重構為自我訓練框架,讓大型語言模型在無需人工整理問答的情況下提升語言及多模態推理,並勝過其他訓練策略。模型根據高層次主題描述自行生成問題、答案及特權推理示範,再以偽驗證篩選可在新語境及改述下重現答案的問題。
Can a language model improve itself without a single human-written question or answer?
We introduce On-Policy Self-Training (OPST), which reformulates on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. Guided only by coarse, high-level topic descriptions, the model builds its own curriculum of queries and privileged reasoning traces, with no human-curated prompts or answers. A pseudo-verification filter keeps a problem only if the model can reproduce its answer in a fresh context and under a paraphrase.
We evaluate in-domain self-improvement as well as knowledge retention on out-of-domain benchmarks when mixing generation domains, and transfer to multimodal inputs with Visual OPST. OPST improves both language and multimodal reasoning without human-curated questions or answers, and outperforms alternative training strategies.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co