跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-22AI 評分36

讓 MLLM 潛在表示發揮作用:用於電影感影片生成的因素感知預測學習

Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

AI 導讀

研究提出因素感知預測學習(FAPL),讓影片擴散 Transformer 利用多模態大語言模型(MLLM)的表示,支援稀疏時間推理及電影感影片的因素控制。方法透過分階段訓練,分別加強 MLLM 潛在表示的生成應用、因素控制和時間推理。實驗顯示,FAPL 在稀疏輸入生成影片、因素感知編輯及多鏡頭生成三項任務中,表現持續優於基線方法。

正文

Published on Oct 22

·

Submitted by

Eddie

on Oct 23

Authors:

,

,

,

,

,

,

,

,

,

Abstract

Factor-Aware Predictive Learning enables video diffusion transformers to leverage multimodal language model representations for sparse temporal inference and factor-aware cinematic control through staged grounding and optimization.

Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.30926

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.30926 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.30926 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.30926 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co