Standard benchmarks measure scene-level recognition: identifying what is broadly happening or naming objects. ActionLens isolates specific capabilities: detecting when an action changes, resolving who did what in multi-person scenes, recognizing simultaneous multitasking, tracking asymmetric roles, and inferring 3D gaze direction.
Every ground-truth answer is derived deterministically from dense AVA annotations—zero language model hallucination. Through 14 iterative human validation rounds, question clarity rose from 53% to 91.0% human accuracy, defending against binding traps, temporal reversals, and language shortcuts.
Select a diagnostic and sample to view video clips, questions, multiple-choice options, and model error patterns.
ActionLens isolates spatial-temporal binding failure mechanisms across model families. Click any figure to enlarge.
In two-person scenes, when VLMs fail, up to 53% of their mistakes select the other person's action! Models successfully recognize actions occurring anywhere in the scene, but fail to bind the action to the queried person.
Replacing tracked visual bounding boxes with static text coordinates drops accuracy by up to 19.2% on actor disambiguation. Relational descriptions recover part of this gap, proving numeric coordinate parsing is a major hurdle.
Systematic evaluation across visual boxes, static coordinates, and relational references reveals that text coordinates remain a critical bottleneck across model scales, whereas visual boxes unlock strongest alignment.
Scaling visual input from 1 to 64 frames provides substantial gains on person-binding diagnostics (+25.3% on D2 for Qwen3.5), confirming that spatial-temporal binding demands temporal continuity rather than single-frame static shortcuts.
Comparison of ActionLens with established video understanding benchmarks.
| Benchmark | QA Pairs | Clips | Per-Sec Temporal GT | Per-Person Spatial GT | Multi-Person Compositional | Iterated Human Audit |
|---|---|---|---|---|---|---|
| VideoMME | 2,700 | 900 | — | — | — | — |
| MVBench | 4,000 | — | — | — | Partial | — |
| EgoSchema | 5,031 | 5,031 | — | — | — | — |
| STAR | 60,000 | 22,000 | Partial | — | — | — |
| PerceptionTest | 11,619 | 691 | ✓ | — | Partial | — |
| TempCompass | 7,540 | 500 | ✓ | — | — | — |
| ActionLens (Ours) | 6,701 | 6,701 | ✓ | ✓ | ✓ | ✓ |
Exact-match accuracy (%) across 20 Vision-Language Models on ActionLens. Click any column header to sort ascending or descending.
| Rank | Model | Size | D1: Transition | D2: Actor Dis. | D3: Concurrent | D4: Interaction | D5: Gaze | Weighted Avg |
|---|
* Weighted Avg = accuracy across all 6,701 questions (question-micro aggregate). Human pooled reference evaluated across 1,194 review items. Random chance accounts for D4's 3-option items (26.7%).
ActionLens is integrated into lmms-eval. Run evaluation with:
# 1. Install lmms-eval with ActionLens support
git clone -b add-actionlens-dataset https://github.com/joslefaure/lmms-eval.git
cd lmms-eval && pip install -e .
# 2. Run one-command evaluation on any supported model (e.g. Qwen3.5-27B)
accelerate launch --num_processes=1 -m lmms_eval \
--model vllm \
--model_args pretrained="Qwen/Qwen3.5-27B-Instruct",tensor_parallel_size=2 \
--tasks actionlens \
--batch_size 1 \
--output_path ./results/actionlens_qwen35_27b@article{faure2026actionlens,
title = {ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models},
author = {Faure, Gueter Josmy and Chen, Min-Hung and Wang, Hao Ping and Lardy, Timoth{'e}e and Su, Hung-Ting and Hsu, Winston H.},
journal = {arXiv preprint},
year = {2026},
url = {https://joslefaure.github.io/actionlens}
}