OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing
OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing
OmniXRef 提出統一框架,讓用戶結合不同數量的文字、圖片及影片參考,控制來源影片的多種屬性。框架採用 R^2-RoPE,並以單流擴散 Transformer 處理來源影片及來自 MLLM 的多模態上下文;團隊亦建構 OmniXRef-Data 和 OmniXRef-Bench。評估顯示,OmniXRef-Model 的表現優於特定任務基線及商業模型。
Published on Oct 15
Authors:
,
,
,
,
,
,
,
,
Abstract
Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce R^2-RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.
View arXiv page View PDF Project page Add to collection
Get this paper in your agent:
hf papers read 2610.22744
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.22744 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.22744 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.22744 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co