Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
知識型視覺問答研究發現,在數據稀缺的微調條件下,決策層晚期融合優於深度注意力修改,顯示更深入的知識整合未必更有效。研究以 Llama 3.3 70B 生成引導資訊,並在 OK-VQA 比較兩種策略;晚期融合取得 39.4% exact match,比無引導基線高 10.0,深度注意力修改則為 31.8%。研究亦以問題改寫擴增訓練數據,令數據量增加逾十倍。
Published on Oct 12
Authors:
,
,
,
Abstract
Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge. While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored. In this work, we generate structured reasoning guidance, visual focus concepts, distractor elements and external knowledge using Llama~3.3~70B and systematically compare two integration strategies within a unified frozen ViT + FLAN-T5 framework on the OK-VQA benchmark: decision level late fusion and attention-matrix injection. Contrary to the prevailing assumption that deeper integration is superior, we find that shallow late fusion achieves the strongest results (39.4% exact match, +10.0 over the no guidance baseline), outperforming deep attention modification (31.8%) under data scarce fine-tuning. We further contribute an LLM based augmentation pipeline that expands training data more than tenfold via question paraphrasing. Our findings indicate that integration depth must be matched to the data regime.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.17079
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
Visual Question Answering • Updated about 4 hours agorishii100/knowledge-guided-vqa-models
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.17079 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.17079 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co