跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-23AI 評分40

感知運動對齊:預測具身策略成功的柏拉圖式指標

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

AI 導讀

研究提出感知運動對齊(S²),發現其與機械人任務成功率呈正相關,涵蓋模擬環境中的 VLA 和 WAM,以及真實機械人上的策略。S² 可作為低成本的離線指標,在閉環評估前篩選較有潛力的模型,亦可在昂貴的 VLA 微調前識別合適的 VLM 主幹模型。

正文

The Platonic Representation Hypothesis (Huh et al, 2024) argues that as vision and language models scale, their representation spaces converge toward a shared statistical model of reality. The idea is that images and text are different projections of the same underlying world. As models recover that world more faithfully, they begin to agree on which inputs are similar. For example, if a vision model places two images close together in the representation space, a language model tends to place their captions close together too.

In this work, we bring PRH to robotics and ask: can physical robot motion be viewed as another projection of that same world, such that more capable embodied models better align their vision and language representations with the structure of motion trajectories?

Our evidence suggests the answer is yes. To test this, we use Sensorimotor Alignment (S²), which measures the overlap between neighborhoods of a model's sensory representations and neighborhoods of end-effector trajectories. Higher alignment means episodes nearby in sensory representation space are also nearby in motor representation space.

Our first finding is that S² is positively related to downstream robot task success. We test a range of Vision-Language-Action models (VLAs) and World Action Models (WAMs) in simulation, as well as policies on real robots. Across these settings, models with higher S² tend to achieve higher task success rates. This pattern provides empirical support for extending PRH to robotics.

S² also explains, at the representation level, several phenomena that prior work has observed only through closed-loop results. For example, VLAs often ignore the language instruction and act mainly on what they see. We find that their task success correlates with the S² of their vision features, not their text features.

In practice, S² can serve as a low-cost offline proxy for identifying which models are likely to achieve higher task success before closed-loop evaluation. For instance, it can identify promising vision-language model (VLM) backbones even before costly VLA fine-tuning.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co