細粒度信用分配何時有助、何時有害:以影片剪輯探討多模態智能體強化學習
When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing
研究發現,細粒度過程獎勵可改善多模態智能體的長時程影片剪輯任務,但固定權重會在較短任務中引發獎勵作弊。在平均超過 60 turns 的 Fine Cut 任務中,衰減過程獎勵權重較 outcome-only GRPO 相對提升 45.4%;三類平均 30–40 turns 的任務,提升最多為 10.4%。
Published on Oct 26
Authors:
,
Abstract
Fine-grained process rewards improve long-horizon video editing tasks but cause reward hacking in shorter tasks, which is mitigated by decaying their weight, while on-policy self-distillation provides little benefit.
Reinforcement learning (RL) has shown promise for training multimodal agents to reason and use tools. However, it remains unclear when fine-grained credit assignment improves learning and when it fails to improve task performance. Video editing provides a useful testbed for this question, combining multimodal perception, long-horizon planning, and tool use with delayed outcome feedback. We construct four movie-trailer editing task types---Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut---that reflect real-world editing workflows and span different levels of task complexity and interaction horizon. Decaying process rewards provides the largest benefit on Fine Cut, with a 45.4% relative gain over outcome-only GRPO on tasks averaging more than 60 turns under outcome-only GRPO. On the three shorter-horizon task types averaging 30--40 turns under the same baseline, outcome-only GRPO remains competitive, with decaying process rewards yielding at most a 10.4% relative gain. Process rewards from an LLM judge initially improve tool use and encourage visual inspection. As training progresses, however, GRPO+PRM with a fixed process-reward weight increasingly exhibits excessive reasoning and redundant tool calls: process rewards continue to rise while task completion rates and outcome scores decline. Decaying the process-reward weight to zero mitigates this reward-hacking behavior while preserving the early gains. Despite dense token-level supervision, on-policy self-distillation (OPSD) yields little improvement or substantial degradation and is highly sensitive to the choice of privileged information. Our analysis reveals differences in teacher--student disagreement and gradient magnitudes, and suggests possible dilution of informative signals over long trajectories.
View arXiv page View PDF Project page GitHub 1 Add to collection
Models citing this paper 2
Image-Text-to-Text • 35B • Updated 5 days ago • 52 • 1leizhao7/VideoEdit-Qwen3.6-35B-A3B
Image-Text-to-Text • 9B • Updated 5 days ago • 44leizhao7/VideoEdit-Qwen3.5-9B
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.35497 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.35497 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co