長時程 LLM 智能體的多尺度策略最佳化
Multiscale Policy Optimization for Long-Horizon LLM Agents
多尺度策略最佳化(MPO)透過在回合尺度對齊信用分配與策略更新,改善長時程 LLM 智能體的訓練。MPO 結合多尺度優勢估計(MAE)與回合層級重要性抽樣,在使用 4B 和 30B-A3B 模型進行的搜尋、工具使用及自動化任務實驗中,持續勝過多個強化學習基線。
Published on Oct 26
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.35013 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.35013 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.35013 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co