視覺基礎模型中的空間狀態表徵探究
Probing Spatial State Representations in Visual Foundation Models
研究以 10 種表徵檢視視覺基礎模型的空間能力,發現可讀取場景不代表能持續追蹤物件、預測空間更新或可靠地支援目標導向行動。物件持續性和可行動性建基於相機位姿讀取,特徵預測則建基於可見場景讀取。各項能力亦隨任務、網絡層及觀察前綴長度而異。
Published on Oct 29
Authors:
,
,
,
,
Abstract
Visual foundation models vary in supporting persistent spatial states across structure, persistence, prediction, and action, with readout accuracy not guaranteeing history-based updates or reliable goal-directed progress.
An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.
View arXiv page View PDF Project page Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.40636 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.40636 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.40636 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co