回合至關重要:以回合級信用分配推進串流影片強化學習
Turns Matter: Advancing Streaming Video RL via Turn-Level Credit
TurnWise提出回合級GRPO框架,透過逐回合信用分配改善串流影片理解的強化學習。框架平均各回合的 token 損失,讓回應與等待決策獲得相同權重,無需訓練 critic 或增加 rollout;實驗顯示其表現優於先前的串流影片理解強化學習方法。
Published on Nov 4
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
TurnWise improves streaming video understanding by assigning turn-level credit and balancing decision weights in group relative policy optimization.
Streaming video understanding requires models to repeatedly decide whether to respond or continue observing. Recently, Group Relative Policy Optimization (GRPO) has been applied to this setting, with existing approaches emphasizing reward design while retaining a single trajectory-level advantage across turns. Such shared credit can reinforce redundant replies after earlier successes and discourage useful follow-ups after earlier failures. We propose TurnWise, a turn-level GRPO framework that aligns learning with individual streaming decisions. TurnWise addresses the mismatch between trajectory-level feedback and turn-level interaction, promoting informative responses and reducing redundancy. Specifically, turn-level credit assignment uses silence-completed prefix scoring to construct group-relative advantages from each decision's remaining return, separating its learning signal from earlier rewards. Turn-mean policy optimization averages token losses within each turn and then across turns, giving response and waiting decisions equal weight regardless of output length. These complementary designs improve streaming policy learning without a learned critic or additional rollouts. Extensive experiments show our method outperforms prior RL approaches for streaming video understanding.
View arXiv page View PDF GitHub 53 Add to collection
Get this paper in your agent:
hf papers read 2611.06728
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
4B • Updated 22 minutes ago • 3shuai22/turns-matter-rl-ckpt
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.06728 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.06728 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co