PACT 跨模型預填充至自回歸遷移的機遇與挑戰
On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
研究提出 PLatour,以直接透過強化學習訓練的輕量線性橋接器連接凍結模型,讓大型模型處理提示詞,再由小型模型獨立生成回應。使用 35B sender 時,PLatour 在共同數學及編碼基準上的結果超越 Mostik AI 公佈的 753B sender 結果,但訓練設定可能不同。
How much can a small model benefit from a larger model that only processes the prompt—and what actually drives the improvement?
We explore these questions in this new paper.
In prefill-to-autoregressive cross-model transfer (PACT), a larger sender processes the prompt and passes internal representations to a smaller receiver, which then generates the response independently.
We introduce PLatour, a simple and scalable framework that connects frozen models through lightweight linear bridges trained directly with reinforcement learning. It also supports hybrid-attention architectures.
On generation tasks, PLatour substantially outperforms the evaluated open-source PACT baselines. Using a 35B sender, it also exceeds Mostik AI’s reported results on shared math and coding benchmarks, compared with Mostik's 753B sender—although the training setups may differ.
But stronger benchmark results do not, by themselves, establish useful information transfer.
On some tasks, replacing sender representations with question-independent signals or training the receiver alone reproduces much of the improvement. Trainable bridges can provide substantial adaptation capacity even when both model backbones are frozen.
The evidence is task-dependent: controls nearly match PLatour on GSM8K, while PLatour retains gains beyond all tested controls on Chemistry with chain-of-thought generation.
To examine this systematically, we introduce PACT-Check. It combines interventions on sender information with separately trained adaptation baselines. PACT-Score measures the accuracy margin over the strongest applicable control.
Our takeaway: PACT shows promising performance, but understanding its benefits requires distinguishing useful sender information from gains achievable through adaptation.
We hope PLatour and PACT-Check help the community investigate when prefill-only communication helps—and how to make those benefits more reliable.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co