🐺 Wolf: Captioning Everything
with a World Summarization Framework

Boyi Li1,2 Ligeng Zhu1,3 Ran Tian1,2 Shuhan Tan1,4 Yuxiao Chen1 Yao Lu1
Yin Cui1 Sushant Veer1 Max Ehrlich1 Jonah Philion1,5 Xinshuo Weng1 Fuzhao Xue1 Jim Fan1 Yuke Zhu1,4
Jan Kautz1 Andrew Tao1 Ming-Yu Liu1 Sanja Fidler1,5 Boris Ivanovic1 Trevor Darrell2 Jitendra Malik2 Song Han1,3 Marco Pavone1,6
1NVIDIA, 2UC Berkeley, 3MIT, 4UT Austin, 5University of Toronto, 6Stanford University
Paper arXiv Video Code Homepage

Leaderboard

Method Caption Similarity ↑ Caption Quality (eg. reduced hallucination) ↑
Scenarios Organization Nuscenes Pexels Robotics Nuscenes Pexels Robotics
CogAgent Tsinghua&Zhipu AI 0.18 0.68 0.38 0.24 0.72 0.43
GPT-4V OpenAI 0.31 0.72 0.34 0.36 0.75 0.35
VILA-1.5-13b NVIDIA 0.21 0.85 0.62 0.25 0.86 0.67
Gemini-Pro-1.5 Google 0.42 0.87 0.63 0.45 0.87 0.67
Wolf(ours) NVIDIA 0.55 0.88 0.72 0.56 0.89 0.75

We are continuously adding more models to the leaderboard. If you would like to add your model, please contact us!