跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分39

GAFU:解鎖統一多模態模型的細粒度理解能力

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

AI 導讀

GAFU 是一種後訓練框架,將生成端視覺表徵壓縮為精簡的 G tokens,供理解路徑按任務檢索,以提升統一多模態模型的細粒度視覺理解能力。與使用相同數據訓練的 SFT 基線相比,GAFU 在 VStar-Bench、OmniSpatial 和 VSR-Bench 的相對提升最高分別為 20.00%、17.94% 和 8.62%。

正文

Published on Oct 14

Authors:

,

,

,

,

,

,

,

Abstract

GAFU leverages native generation-side representations to improve fine-grained visual understanding in unified multimodal models by compressing them into compact tokens for task-conditioned retrieval.

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact \(G\) tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00\%, 17.94\%, and 8.62\% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.23987

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.23987 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.23987 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.23987 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co