跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 19 天前AI 評分48

DreamFree:面向無想像世界動作模型的未來表徵蒸餾

Future Representation Distillation for Imagination-free World Action Models

AI 導讀

DreamFree 透過未來表徵蒸餾,構建無需生成未來影片的世界動作模型,並能隨影片模型規模及影片數據增加而提升控制表現。在 DROID 數據集上訓練後,DreamFree 在 10 項真實世界任務中取得 64% 成功率,高於 DreamZero 的 56% 和 π0.5 的 16%。其 14B 影片骨幹在單張 H200 上以 11 Hz 運行,推理速度較 DreamZero 快 17.8 倍。

正文

Can we scale up imagination-free WAMs (e.g., Fast-WAM)?

Excited to share DreamFree — an imagination-free WAM that improves with stronger video models and more video data, running at 11 Hz with a 14B video backbone.

Video models learn how the world changes, providing useful knowledge for robot control. But generating future videos is costly, and prediction errors can affect actions.

Imagination-free WAMs, e.g., Fast-WAM, enable robot control without generating future videos. But for Fast-WAM, stronger video priors don't automatically mean better control. In our experiments, scaling from 5B to 14B did not improve success.

How can we make imagination-free WAMs benefit from stronger video models?

Our key idea: turn the pretrained video model into a predictive visual encoder

A frozen video teacher sees the future. The encoder learns to predict the teacher’s representation from the current view. This teaches the encoder to anticipate scene changes without generating future videos. Training the encoder needs only videos, without using action labels.

Next, we train the action DiT using robot demonstrations with action labels. From the encoder’s visual representation, a readout learns to extract action-relevant information. An action decoder uses this information to predict robot actions.

What we find:

Real-world performance: Trained on DROID alone, DreamFree achieves 64% success across 10 real-world tasks, outperforming DreamZero at 56% and π0.5 at 16%. We evaluate the policies without task-specific fine-tuning.

Fast inference: A 14B video backbone runs at 11 Hz on a single H200, with 17.8× faster inference than DreamZero.

Memory efficiency: With a 14B video backbone, DreamFree fits on a single A6000 GPU with 48 GB of memory, without future-video generation or classifier-free guidance at inference.

Better scaling: Scaling the video backbone from 5B to 14B significantly improves performance, enabling real-world manipulation on DROID and bimanual YAM. More video data further improves control without additional action labels.

Video-only task transfer: DreamFree learns new task sequences from videos, including demonstrations from a different robot embodiment, without action labels for the new tasks.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co