跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 25 天前AI 評分47

多模態推理中的參照落差:視覺輔助能解決和不能解決的問題

Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

AI 導讀

多模態語言模型在多步視覺推理中的失誤,可能源於語言未能連結相關視覺物件、區域或位置的「參照落差」,而非視覺感知不足。研究提出 ReGap-Bench,涵蓋五項視覺推理任務;結果顯示,配合任務的外部參照可在無需訓練下令準確率提高最多 25%,不匹配的參照則助益甚微,甚至會降低表現。

正文

Abstract

Multimodal language models fail at multi-step visual reasoning due to a reference gap rather than perception, and task-matched external references like numbering significantly improve accuracy without training.

Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25\%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.39419

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.39419 in a model README.md to link it from this page.

Datasets citing this paper 1

regap-bench/ReGap-Bench

Viewer •

Updated 3 days ago

•

1k

•

63

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.39419 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co