Dataset · PIVOTSBench

At a glance

Sources: Social-IQ 2.0 (121 clips) and YouTube (70 clips). Social-IQ 2.0 carries salient, everyday interaction; the YouTube half was collected for restrained, in-depth exchanges — persuasion, conflict resolution, former partners, estranged siblings, strangers across an ideological divide.
Selection: Multiple speakers visible and audible; interaction longer than 30 seconds; social dynamics not readable off a trivial symbol. News interviews and clips with text overlays were removed to limit visual leakage.
Unit: One directional pair per clip pair-of-characters, scored on all six dimensions. 765 directional pairs, 4,590 scores, plus 483 auxiliary items for Tasks 2 and 3.
Annotation: GPT-5 produced initial scores from the rubric; annotators verified every one against the definitions, adjusted 25.3%, and attached an evidence timestamp, cue tags and a written rationale. On a 35-clip validation set, fully manual and model-assisted annotation agree at Cohen’s κ = 0.977.
Agreement: Krippendorff’s α (ordinal) 0.8125 overall; 83.87% exact agreement. Power scores lowest on α (0.643) purely because over 70% of its labels are 0, which deflates the coefficient at 86.2% exact agreement.
Characters: At most three per clip, kept by dominance and role. Where the source identifiers were weak, characters were relabeled descriptively — “woman in yellow”, “man on the right”. All five evaluated models identified characters correctly on all 43 test questions.

Label distribution

765 scores per dimension. The shape of each bar is itself a finding: power is overwhelmingly egalitarian in this material, while objective and permanence spread across the full range.

−2 −1 0 +1 +2

Counts per dimension. Also at `data/stats/score_distribution.csv`.
Dim	−2	−1	0	+1	+2	Total
P	13	89	544	104	15	765
I	54	123	105	345	138	765
V	24	86	147	424	84	765
O	96	64	87	195	323	765
T	121	34	229	238	143	765
S	13	73	124	425	130	765

Inter-annotator agreement per dimension. Also at `data/stats/agreement.csv`.
Dim	Exact agreement	Krippendorff’s α
P	86.23%	0.6434
I	83.48%	0.8402
V	81.98%	0.7693
O	79.97%	0.7803
T	87.59%	0.8395
S	83.96%	0.7906
Overall	83.87%	0.8125

Getting the data

Annotations live in the repository as JSONL, one record per directional pair, with a JSON Schema for each task. Video is not redistributed: Social-IQ 2.0 clips come from the original challenge distribution under its license, and YouTube material is referenced by identifier and time span only.

git clone https://github.com/ciossayin/pivots-bench.git
cd pivotsbench
pip install -r requirements.txt
python scripts/validate_data.py

# evaluate any OpenAI-compatible endpoint
python -m pivots.run_eval --task 1 --model gpt-5

Prompts for all three tasks, all three prediction settings and both heuristic strategies are in prompts/, transcribed from the paper.

Intended use and limits

PIVOTS is an analysis and evaluation instrument, not a deployable system. It should not be used to score real people without their knowledge, and it should not be put into high-stakes or persuasive settings without human oversight.

The material is English-only. All annotators share a cultural and linguistic background and are not native English speakers, which is a live source of bias on the subjective dimensions — valence and objective above all.

Annotations are released under CC BY-NC-SA 4.0; the code under MIT. The underlying videos remain under their original terms.

Cite

@inproceedings{pivotsbench2026,
  title     = {PIVOTSBench: Evaluating Fine-Grained Interpersonal Relationship
               Reasoning in Multimodal Large Language Models},
  author    = {Zhang, Shuxiang and Yin, Yiting and Song, Wenxuan and
               Wu, Yuhang and Liu, Miao},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026}
}

Dim	−2	−1	0	+1	+2	Total
P	13	89	544	104	15	765
I	54	123	105	345	138	765
V	24	86	147	424	84	765
O	96	64	87	195	323	765
T	121	34	229	238	143	765
S	13	73	124	425	130	765

Dim	−2	−1	0	+1	+2	Total
P	13	89	544	104	15	765
I	54	123	105	345	138	765
V	24	86	147	424	84	765
O	96	64	87	195	323	765
T	121	34	229	238	143	765
S	13	73	124	425	130	765

Dim	−2	−1	0	+1	+2	Total
P	13	89	544	104	15	765
I	54	123	105	345	138	765
V	24	86	147	424	84	765
O	96	64	87	195	323	765
T	121	34	229	238	143	765
S	13	73	124	425	130	765