DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
DiffusionART 提出一種擴散模型線上強化學習方法,透過聚合每個提示詞的獎勵改善方向,實現高效穩定的訓練。在 HunyuanVideo-13B 上,DiffusionART 達到 DiffusionNFT 和 Flow-GRPO-Fast 後期平均獎勵水平所需的 GPU 小時分別少 12.02 倍和 11.72 倍,平均訓練獎勵則分別高 28.8% 和 53.0%。
Published on Oct 8
Authors:
,
,
,
,
,
,
,
Abstract
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.12572
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 2
Text-to-Image • Updated about 2 hours agovanzl/DiffusionART-SD35M-MultiReward
Text-to-Video • 13B • Updated 8 minutes agovanzl/DiffusionART-HunyuanVideo-7-2-1
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.12572 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.12572 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co