IntReal:面向多模態交錯生成的真實世界數據集與評測基準
IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
IntReal 提出面向真實世界多模態交錯生成的數據集與評測框架,用於診斷統一多模態模型的能力並改善生成效果。框架包括涵蓋四大類別的 IntReal-100K,以及按七個維度評估模型的 IntReal Evaluation Suite;評測與人類判斷的一致準確率為 83.3%。實驗顯示,使用 IntReal-100K 訓練可改善交錯生成,單模態生成訓練亦可帶來效益。
Authors:
,
,
,
,
,
,
Abstract
IntReal introduces a real-world interleaved multimodal dataset and evaluation framework to diagnose unified model strengths and improve generation through targeted training.
Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's ρleq 0.42 ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's ρ> 0.9 ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.
View arXiv page View PDF Project page GitHub 2 Add to collection
Community
Paper submitter
Models citing this paper 4
Any-to-Any • 18B • Updated 16 minutes agoyushaohan/IntReal-U1-8B-MoT-SFT-RL
Any-to-Any • 18B • Updated 16 minutes agoyushaohan/IntReal-U1-8B-MoT-SFT
Any-to-Any • Updated 15 minutes agoyushaohan/IntReal-Bagel-SFT
Any-to-Any • Updated 16 minutes agoyushaohan/IntReal-Bagel-SFT-RL
Datasets citing this paper 1
yushaohan/IntReal-100K
Viewer • Updated 17 minutes ago • 99.4k
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.38425 in a Space README.md to link it from this page.
Collections including this paper 1
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co