PIVOTSBench

People do not read a relationship as a category. They read it as a set of graded, directional judgments — and they read it from faces, posture and timing as much as from words.

PIVOTS is the first benchmark that asks whether multimodal LLMs can do the same: score a relationship on six psychology-grounded dimensions, separately for each participant's point of view, from a 30-second clip and its transcript. Two clips can share the label parent–child and sit at opposite ends of nearly every dimension.

Alongside scoring, two auxiliary tasks ask the model to point at its evidence: which frame shows it, and which visual cue in that frame carries it.

−2 −1 0 +1 +2 Sign follows the dimension’s own axis, not good vs. bad.

191video clips, 30s each

765directional pairs

4,590scoring items

483visual-reasoning items

0.81Krippendorff's α

Six dimensions

Four axes come from Wish, Deutsch and Kaplan (1976); valence and permanence extend them. Each is a five-point bipolar scale from −2 to +2, scored once for A’s view of B and once for B’s view of A.

PPowerEgalitarian vs. hierarchical — influence, decision-making agency, status

IInvolvementSuperficial vs. intense — intimacy, self-disclosure, interdependence

VValencePositive vs. negative — the emotional quality of the interaction

OObjectiveSocioemotional vs. task-oriented — driven by the bond or by the goal

TPermanenceTemporary vs. enduring — the expected lifespan of the bond

SStanceCooperative vs. competitive — are the two sets of interests linked or opposed

Three tasks

Six-dimensional scoring

Given the clip and its transcript, predict the six scores for A→B. Five options per item, so chance is 20%.

Key-frame identification

Five candidate frames; pick the one that best shows the dimension being scored. Distractors sit ±5s and ±10s from the annotated evidence frame.

Visual cue causal analysis

Given that frame, pick the two cues that carry the judgment — expression, posture, gaze, gesture, attire, context. Half credit for one, so chance is 40%.

What the results say

GPT-5 leads at 56.9% average on the Social-IQ 2.0 subset; the strongest open model tested, Qwen3-32B-VL, reaches 34.5%. For reference, the three human annotators agreed exactly with each other 83.9% of the time.

Human agreement83.9

GPT-556.9

Qwen3-32B-VL34.5

Average scoring accuracy, Social-IQ 2.0 subset. The human bar is inter-annotator exact agreement; the tick marks chance at 20%.

The gap is not uniform. Power is comparatively easy for every model. Valence and objective are the hard ones: valence needs affective signal that most current models do not represent, and objective breaks down whenever a task-driven interaction is emotionally loud.

Two findings cut against the obvious. Redacting kinship terms and honorifics from the transcript costs only 2.5 points — models are not simply reading the labels. And scoring all six dimensions at once beats scoring them one at a time on casual interactions (+8.7), while on in-depth interviews it hurts.

All results and ablations →