VIBE Benchmark: Can a VLM read the room?

Frontier VLMs and MLLMs can see a smile. But they can't tell you what it means.

Blog post by Eylon Caplan · October 2026

Based on the paper

VIBE: Can a VLM Read the Room?

Tania Chakraborty, Eylon Caplan, Dan Goldwasser · Findings of EMNLP 2025

Paper Dataset

Watch the clip (no sound), then pick the interpretation that fits best.

Seeing isn't understanding

Most work on VLM failures is about perception: did the model hallucinate something that isn't there? VIBE targets the next step. The cue is guaranteed to be in the video; the question is what it means in context. We call this visual social-pragmatic (VSP) inference.

1 · Perceive2 · Interpret

cue: a smile ✓

What's in VIBE

Each of the 994 questions pairs a short clip (433 clips, all under 4 seconds, from English and Chinese TV shows) with one observable fact about the speaker and two interpretations of it. One is right. Every fact and answer was checked by at least two human annotators who agreed perfectly.

Try three. These are hard cases we picked because most models missed them.

Nobody is close to humans

We ran four current open models and four frontier API models on all 994 questions, giving every model the same 30 frames per clip. The best model, GPT-6.1 Sol, scores . People score .

Accuracy on all 994 questions, 30 frames per clip. Bars start at chance (50%). Hatched = thinking on. Whiskers: 95% CI. Paper models from the original VIBE paper. *InternVL2-26B is footnoted there: the dataset was partly built from its mistakes.

Performance gains are expensive

The API models are the most accurate. One pass over the benchmark costs through their APIs, against roughly $0.30 to $1.65 of GPU time for the open models, and the best API model is only 4.5 points more accurate than the best open one. The most expensive run, Claude Opus 5.5, still finishes 12 points below people.

Cost of one pass over all 994 questions (log scale). API models: measured, at batch prices (half of list). Open models: estimated from measured GPU time at $2.50 per H100-hour; hollow points.

Thinking more doesn't help

We compared each open model with reasoning off and on, and each Gemini model at medium and high reasoning effort. Every change is within about a point, and every 95% interval includes zero.

Paired change in accuracy, same 994 questions, with 95% bootstrap CI. Open models: thinking off → on (on = mean of 3 seeds). API: reasoning effort medium → high.

More frames also don't help

Social cues can be very brief. So we think closing the gap to humans likely requires a high frame rate, close to the clip's full frame rate (about 24 per second). But with current models, that clearly isn't sufficient. Accuracy with 4 frames is within about a point of accuracy with every frame of the clip. Pick a frame budget to see what the model gets from this clip.

Frames sent

Mode

Accuracy vs. frames per clip for the open models (all 994 questions; y-axis zoomed). Clips are short: at 60 requested, models get every frame of the clip, 50 on average. For the Qwen models, more frames also means lower resolution per frame.

Takeaway

It's been a year and a half since the VIBE paper came out. In that time, the best score went from 75% to 81%. That's progress, but the best model is still 11 points behind people.

PaperFindings of EMNLP 2025 DatasetVIBE on Hugging Face, MIT license

@inproceedings{chakraborty-etal-2025-vibe,
  title     = {{VIBE}: Can a {VLM} Read the Room?},
  author    = {Chakraborty, Tania and Caplan, Eylon and Goldwasser, Dan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  pages     = {22992--23008},
  year      = {2025}
}