Diagnostic Benchmark Under Review

ActionLens

Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

Gueter Josmy Faure1, Min-Hung Chen2, Hao Ping Wang1, Timothée Lardy1, Hung-Ting Su1, Winston H. Hsu1
1National Taiwan University  •  2NVIDIA
arXiv Code & Eval Dataset (Anonymous Repo) Explore Samples Leaderboard (20 VLMs)
6,701
Questions
5
Targeted Diagnostics
20
VLMs Benchmarked
100%
Deterministic GT
91.0%
Human Reference
25.4%
Random Chance
ActionLens Teaser - Spatial-Temporal Binding Failures in VLMs
ActionLens asks video questions humans answer in seconds. State-of-the-art vision-language models score above 80% on scene-level benchmarks like VideoMME, yet they systematically fail at spatial-temporal binding: associating the right action with the right actor at the right moment. ActionLens decomposes this fundamental capability into five targeted diagnostics with 100% deterministic ground truth derived from 1.58 million per-second annotations in AVA v2.2.

A Diagnostic Probe, Not Just a Leaderboard

Standard benchmarks measure scene-level recognition: identifying what is broadly happening or naming objects. ActionLens isolates specific capabilities: detecting when an action changes, resolving who did what in multi-person scenes, recognizing simultaneous multitasking, tracking asymmetric roles, and inferring 3D gaze direction.

14 Rounds of Human Quality Engineering

Every ground truth answer is derived deterministically from dense AVA annotations—zero language model hallucination. Through 14 iterative human validation rounds, question clarity rose from 53% to 91.0% human accuracy, defending against binding traps, temporal reversals, and language shortcuts.

Interactive Explorer

Explore the 5 Core Diagnostics

Click on any diagnostic below to view actual video clips, questions, multiple-choice options, and model failure modes.

194 Questions Sample 1: Grabbing to Lifting

What action transition does the person in the red box undergo during this clip?

Empirical Analysis

Key Diagnostic Findings

ActionLens isolates five critical failure mechanisms that remain invisible on general video benchmarks.

Finding 1 The Binding Trap

Binding Trap Hit Rate on D2

In two-person scenes, when VLMs make an error, up to 53% of their mistakes select the other person's action! Models successfully recognize actions occurring anywhere in the scene, but fail to bind the action to the queried person.

Finding 2 Visual Grounding Deficit

Box vs Static Coordinate References

Replacing tracked visual bounding boxes with static text coordinates drops accuracy by up to 19.2% on actor disambiguation. Relational descriptions recover part of this gap, proving that numeric coordinate parsing is a major hurdle for current VLMs.

Finding 3 Frame Sweeps & Temporal Binding

Frame Sweep Heatmap

Scaling visual input from 1 to 64 frames provides substantial gains on person-binding diagnostics (+25.3% on D2 for Qwen3.5), confirming that spatial-temporal binding demands temporal continuity rather than single-frame static shortcuts.

Finding 4 Gaze Collapses Near Chance

89.6% vs 25.8%
Human vs VLM Gaze Accuracy

Random chance is 25.0%. Even frontier models struggle with 3D gaze and head orientation.

While human raters achieve 89.6% on gaze tracking, state-of-the-art models hover between 25% and 47%. Gaze detection requires fine-grained 3D head and eye orientation tracking that current visual tokenizers fail to preserve.

Differentiators

Benchmark Comparison

ActionLens is the only video benchmark providing per-person spatial grounding, programmatic derivation, and iterative human validation.

Benchmark QA Pairs Clips Per-Sec Temporal GT Per-Person Spatial GT Multi-Person Compositional Iterated Human Audit
VideoMME 2,700 900 — — — —
MVBench 4,000 — — — Partial —
EgoSchema 5,031 5,031 — — — —
STAR 60,000 22,000 Partial — — —
PerceptionTest 11,619 691 — Partial —
TempCompass 7,540 500 — — —
ActionLens (Ours) 6,701 6,701
Evaluation

ActionLens Leaderboard

Exact-match accuracy (%) across 20 Vision-Language Models, evaluated under standardized prompts on NVIDIA H100 GPUs.

Rank Model Size D1: Transition D2: Actor Dis. D3: Concurrent D4: Interaction D5: Gaze Weighted Avg

* Weighted Avg = accuracy across all 6,701 questions (question-micro aggregate). Human pooled reference evaluated across 1,194 review items. Random chance accounts for D4's 3-option items (26.7%).

Reproducibility

Evaluate Any Model with lmms-eval

ActionLens is fully integrated into lmms-eval. Reproduce all benchmark results with a single command.

# 1. Install lmms-eval with ActionLens support
git clone -b add-actionlens-dataset https://github.com/joslefaure/lmms-eval.git
cd lmms-eval && pip install -e .

# 2. Run one-command evaluation on any supported model (e.g. Qwen3.5-27B)
accelerate launch --num_processes=1 -m lmms_eval   --model vllm   --model_args pretrained="Qwen/Qwen3.5-27B-Instruct",tensor_parallel_size=2   --tasks actionlens   --batch_size 1   --output_path ./results/actionlens_qwen35_27b

Citation

@article{faure2026actionlens,
  title   = {ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models},
  author  = {Faure, Gueter Josmy and Chen, Min-Hung and Wang, Hao Ping and Lardy, Timoth{\'e}e and Su, Hung-Ting and Hsu, Winston H.},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://joslefaure.github.io/actionlens}
}