跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 28 天前AI 評分42

細粒度信用分配何時有助、何時有害:以影片剪輯探討多模態智能體強化學習

When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

AI 導讀

研究發現,細粒度過程獎勵可改善多模態智能體的長時程影片剪輯任務,但固定權重會在較短任務中引發獎勵作弊。在平均超過 60 turns 的 Fine Cut 任務中,衰減過程獎勵權重較 outcome-only GRPO 相對提升 45.4%;三類平均 30–40 turns 的任務,提升最多為 10.4%。

正文

Published on Oct 26

Authors:

,

Abstract

Fine-grained process rewards improve long-horizon video editing tasks but cause reward hacking in shorter tasks, which is mitigated by decaying their weight, while on-policy self-distillation provides little benefit.

Reinforcement learning (RL) has shown promise for training multimodal agents to reason and use tools. However, it remains unclear when fine-grained credit assignment improves learning and when it fails to improve task performance. Video editing provides a useful testbed for this question, combining multimodal perception, long-horizon planning, and tool use with delayed outcome feedback. We construct four movie-trailer editing task types---Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut---that reflect real-world editing workflows and span different levels of task complexity and interaction horizon. Decaying process rewards provides the largest benefit on Fine Cut, with a 45.4% relative gain over outcome-only GRPO on tasks averaging more than 60 turns under outcome-only GRPO. On the three shorter-horizon task types averaging 30--40 turns under the same baseline, outcome-only GRPO remains competitive, with decaying process rewards yielding at most a 10.4% relative gain. Process rewards from an LLM judge initially improve tool use and encourage visual inspection. As training progresses, however, GRPO+PRM with a fixed process-reward weight increasingly exhibits excessive reasoning and redundant tool calls: process rewards continue to rise while task completion rates and outcome scores decline. Decaying the process-reward weight to zero mitigates this reward-hacking behavior while preserving the early gains. Despite dense token-level supervision, on-policy self-distillation (OPSD) yields little improvement or substantial degradation and is highly sensitive to the choice of privileged information. Our analysis reveals differences in teacher--student disagreement and gradient magnitudes, and suggests possible dilution of informative signals over long trajectories.

View arXiv page View PDF Project page GitHub 1 Add to collection

Models citing this paper 2

leizhao7/VideoEdit-Qwen3.6-35B-A3B

Image-Text-to-Text •

35B •

Updated 5 days ago

•

52

•

1

leizhao7/VideoEdit-Qwen3.5-9B

Image-Text-to-Text •

9B •

Updated 5 days ago

•

44

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.35497 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.35497 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co