跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分44

NEO-VLA:從像素到動作,邁向原生視覺語言動作策略

From Pixels to Actions - Towards Native Vision-Language-Action Policies

AI 導讀

NEO-VLA 提出一種無固定視覺編碼器的視覺語言動作框架,讓圖像與語言在骨幹網絡中共同演化,供動作專家從不同深度讀取視覺表徵。在受控比較中,原生架構勝過依賴編碼器的架構;完整微調可縮小差距,但不能消除差距,單靠擴大規模亦無法追回。在多種操控情境及分佈轉移下,NEO-VLA 的控制表現更強、更穩健。

正文

Published on Nov 4

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

NEO-VLA eliminates fixed visual encoders to let action policies access evolving visual representations across network depths, improving manipulation robustness.

Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.

View arXiv page View PDF Project page Add to collection

Get this paper in your agent:

hf papers read 2611.06918

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.06918 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.06918 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.06918 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co