跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分49

VideoMemBench:多模態大語言模型在影片對話中的記憶能力如何?

VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

AI 導讀

VideoMemBench 評測多模態大語言模型在長篇多輪影片對話中的記憶能力,結果顯示模型表現差距顯著。基準涵蓋 326 段影片及 2,476 個對話輪次,其中 1,387 個輪次依賴較早的對話內容。逾 20 款開源及閉源模型中,Claude Opus 5 整體得分最高,為 45.7%;數據集及評測已公開。

正文

Published on Nov 3

Authors:

,

,

,

,

,

Abstract

VideoMemBench evaluates multimodal language models on long multi-turn video dialogues requiring retrieval of earlier visual and conversational context, revealing substantial performance gaps.

Multimodal large language models (MLLMs) can now process hour-long videos, making extended multi-turn dialogue over a single video a natural mode of use. Sustaining such a dialogue requires more than answering each question in isolation: the model must retrieve and reuse information established earlier in the conversation, both what it has seen and what it has said, an ability we term in-context video memory. Existing video QA benchmarks leave this ability unmeasured: their questions are mutually independent, never reference one another, and each is answerable from the video alone, so nothing ever requires carrying information across turns. We introduce VideoMemBench, in which each sample pairs a single video with a long multi-turn dialogue, whose turns are connected by typed cross-turn dependencies: an Episodic Dependency refers back to a visual episode described in an earlier turn, probing memory of what the model has seen, while an Answer Dependency builds on the model's own earlier answer, probing memory of what the model has said. Every dependency is verified considering two properties: 1) necessity, meaning the question cannot be answered without its earlier turns, and 2) non-leakage, meaning the question does not restate the information from its earlier turns. To enforce these properties, we design a construction pipeline with three LLM roles: a planner, a generator, and a judger, with each dialogue reviewed by human experts. VideoMemBench comprises 326 videos with 2,476 turns, of which 1,387 depend on earlier turns, including multi-parent dependencies and long-range call-back. Among over 20 evaluated open- and closed-source MLLMs, Claude Opus 5 performs the best with an overall score of only 45.7%, highlighting the challenge of in-context video memory. Our dataset and evaluation are released.

View arXiv page View PDF Project page GitHub 1 Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.40131 in a model README.md to link it from this page.

Datasets citing this paper 1

VideoMemBench/data

Viewer •

Updated 3 days ago

•

326

•

543

•

1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.40131 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co