VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval
VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval
VEDJE 將影片壓縮成小型、具時間結構的快取,再以 33M 參數的聯合編碼器進行影片文字檢索。它在 MSR-VTT 取得 59.8 R@1,線上參數量比 LamRA 少約 49×;每段影片的快取為 48 KiB,較原始視覺特徵小 128.5×。保留壓縮後的時間結構有助檢索,特徵變化預測目標亦能提升檢索表現,且不增加推理時計算。
🚀 Can we get the accuracy of a 7.6B-parameter video-language model using a much smaller retrieval system?
Introducing VEDJE: efficient video-text retrieval with compact, reusable video representations.
Rather than repeatedly processing full video features, VEDJE compresses each video once into a tiny, temporally structured cache. A lightweight 33M-parameter joint encoder then matches queries against the cached candidates.
The results:
📈 59.8 R@1 on MSR-VTT, matching LamRA with ~49× fewer online parameters.
📦 48 KiB per video, 128.5× smaller than the original visual features.
⚡ +3.6 to +7.2 R@1 points over matched first-stage retrievers on MSR-VTT.
The interesting finding: preserving temporal structure during compression matters. We also introduce a feature-change prediction objective that improves retrieval without extra inference-time computation.
🔗 Interactive project page: https://gabrieleserussi.github.io/vedje/
💻 Open-source implementation: https://github.com/GabrieleSerussi/vedje
▶️ Google Colab: https://colab.research.google.com/github/GabrieleSerussi/vedje/blob/main/colab/vedje_tour.ipynb
We're curious: could compact, reusable video representations become a practical foundation for large-scale multimodal search?
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co