LambdaMix:以聯合訓練低秩適配器實現可調式多目標羣組相對偏好最佳化
Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters
LambdaMix 以羣組相對線上強化學習,在凍結的基礎模型上為每個目標聯合訓練一個低秩適配器,學習一系列由偏好條件化的策略。偏好向量會選擇生成回應的適配器組合,並加權用於更新適配器的獨立正規化獎勵;訓練完成後,可在推理時調整偏好而無須再次最佳化。LambdaMix 在四個基礎 LLM 上的超體積高於所有受評估基線,所需訓練補全數量及適配器儲存空間亦少於一組 GRPO 專家模型。
🤔 Post-training usually gives you one policy for one fixed weighting of objectives, but different users and settings call for different trade-offs.
🎛️ We introduce LambdaMix, a parameter-efficient method that learns a whole family of preference-conditioned policies with group-relative online RL.
🧩 LambdaMix trains one low-rank adapter per objective together on a frozen base model. A single preference vector does two things: it picks the adapter mixture that generates responses, and it weights the separately normalized rewards used to update the adapters.
🔁 After one training run, you can change preferences at inference with no further optimization.
We test LambdaMix across four base LLMs:
- 🛡️ on helpfulness vs. harmlessness
- 🧮 on math with instruction following
- 🕵️ on rewriting comments while removing clues about personal attributes
📈 LambdaMix achieves higher hypervolume than all evaluated conditioned and policy-merging baselines.
⚡ It also uses fewer training completions and less adapter storage than a set of GRPO specialists.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co