跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分44

YANAMI:面向多輪有狀態互動的大規模多模態數據集與基準

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

AI 導讀

YANAMI 提出面向有狀態互動的大規模多模態對話數據集及基準,評估模型追蹤和實現跨輪次變化的對話狀態。YANAMI-400K 包含 400K 段多輪對話、超過 2M 張圖片及 2.6M 個對話輪次;以其微調 BAGEL,圖像編輯、生成及多模態有狀態互動能力分別提升 10.3%、2.0% 及 13.23%。評測顯示,現有統一多模態模型仍難以維持、更新並以視覺輸出實現變化中的對話狀態。

正文

Published on Nov 3

Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

YANAMI introduces a large-scale multimodal dialogue dataset and benchmark for stateful interaction, requiring models to track and realize evolving dialogue states across turns, and reveals that current unified multimodal models still struggle with this task.

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.

View arXiv page View PDF Project page Add to collection

Get this paper in your agent:

hf papers read 2611.05280

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.05280 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.05280 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.05280 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co