跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分39

VidUEU-Agent:面向多模態理解、編輯與統一任務的數據整理智能體

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

AI 導讀

VidUEU-Agent 是一個數據整理框架,可從原始影片自動構建用於多模態理解、圖像編輯及統一理解與生成任務的指令數據。團隊以 8,268 段影片構建 VidUEU-100K,包含近 20,000 條密集中繼資料標註及 100,000 個指令樣本。框架透過四輪訓練及評測結果調整數據分佈,每輪以 30K 個樣本從同一初始檢查點微調 BAGEL,三個反饋基準的整體表現均有提升。

正文

🚀 We are excited to share our latest work: “VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks”!

The scarcity and high annotation cost of high-quality multimodal instruction data remain major bottlenecks for multimodal model development. Raw videos contain rich visual knowledge and temporal information, but transforming them into accurate, verifiable, and task-oriented instruction data remains challenging.

To address this, we introduce VidUEU-Agent, an agentic data-curation framework that automatically constructs instruction data from raw videos for multimodal understanding, image editing, and unified understanding-and-generation tasks.

🔑 Key Highlights

🎬 Four-Stage Data Construction Pipeline:
VidUEU-Agent performs global perception to analyze video structure and filter low-quality shots, mines informative keyframes and multimodal signals, constructs structured metadata, and routes each metadata entry to suitable task types for instruction synthesis. Agent planning, tool use, and multi-stage quality control support the entire process.

🔍 Evidence-Grounded Data Synthesis:
The framework combines vision-language models with visual tools for quality assessment, semantic analysis, and information verification. During instruction synthesis, the model cites the evidence signals it uses, improving data traceability and factual consistency.

🔄 Autonomous Iteration with Training Feedback:
VidUEU-Agent uses downstream training and evaluation results to identify weaknesses and refine data distributions across capabilities, domains, and task types. Across four iterations, each using 30K samples to fine-tune BAGEL from the same initial checkpoint, performance improves overall on the three feedback benchmarks:

📦 VidUEU-100K Dataset:
Using this pipeline, we construct VidUEU-100K from 8,268 diverse videos, containing nearly 20,000 dense metadata annotations and 100,000 instruction samples: 40K understanding, 30K editing, and 30K unified samples, spanning nine task paradigms and five scenario categories.

Our work demonstrates how raw videos can be transformed into diverse, evidence-grounded instruction data through agent planning, tool augmentation, and training-feedback-driven refinement. We hope VidUEU-Agent and VidUEU-100K provide a scalable and reproducible foundation for developing unified multimodal models.

📄 Paper: https://arxiv.org/abs/2611.04418

💻 Code: https://github.com/LRuca/VidUEU-Agent

🤗 Dataset: https://huggingface.co/datasets/Lruca/VidUEU100k

We welcome feedback and discussions!

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co