跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-12AI 評分40

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

AI 導讀

SyncReward 研究視聽生成的同步評估與獎勵建模,並推出配套資料集、基準及獎勵模型。SyncReward-Data-80K 收錄 80K 項專業同步評分,SyncReward-Bench 包含 1K 段由多位專業人士獨立評分的片段;SyncReward 與受評方法相比,最能貼合人類評分。作為獎勵模型,它可透過推理時 best-of-N 選擇及基於強化學習的後訓練改善同步效果。

正文

Published on Oct 12

Authors:

,

,

,

,

,

Abstract

Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-N selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.16338

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.16338 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.16338 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.16338 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co