🐺 Wolf: Captioning Everything
with a World Summarization Framework

Boyi Li1,2 Ligeng Zhu1,3 Ran Tian1,2 Shuhan Tan1,4 Yuxiao Chen1 Yao Lu1
Yin Cui1 Sushant Veer1 Max Ehrlich1 Jonah Philion1,5 Xinshuo Weng1 Fuzhao Xue1
Andrew Tao1 Ming-Yu Liu1 Sanja Fidler1,5 Boris Ivanovic1 Trevor Darrell2 Jitendra Malik2 Song Han1,3 Marco Pavone1,6
1NVIDIA, 2UC Berkeley, 3MIT, 4UT Austin, 5University of Toronto, 6Stanford University
Paper arXiv Video Code Leaderboard
Main Image

Wolf: Captioning Everything with a World Summarization Framework

Abstract

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore (caption quality) by 55.6% and CapScore (caption similarity) by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.

Udpates

Thanks for your interest in Wolf! We will update more details shortly. Stay tuned!

BibTeX

@article{li2024wolf,
  title={Wolf: Captioning Everything with a World Summarization Framework},
  author={Li, Boyi and Zhu, Ligeng and Tian, Ran and Tan, Shuhan and Chen, Yuxiao and Lu, Yao and Cui, Yin and Veer, Sushant and Ehrlich, Max and Philion, Jonah and others},
  journal={arXiv preprint arXiv:2407.18908},
  year={2024}
}