PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
Qiran Zhang, Yuheng Wang, Runde Yang, Lin Wu, Jingru Fan, Shu Yao, Jie Zhang, Tianle Zhou
Abstract
Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluating whether language models can produce spatially correct animated outputs remains an open problem. We introduce PRISM, a large-scale benchmark of 10,372 human-calibrated instruction-code pairs (20 times larger than prior programmatic video generation benchmarks), grounded in real-world knowledge visualization scenarios across English and Chinese and spanning 437 subject categories. We further propose a funnel-style evaluation framework with four complementary metrics: Code-Level Reliability for executability, Spatial Reasoning for layout correctness over full animation sequences, and Prompt-Aware Dynamic Visual Complexity (PADVC) and Temporal Density (TD) for diagnosing dynamic expression and temporal activity. Systematic evaluation of seven mainstream LLMs reveals a striking Execution-Spatial Gap: the average drop from execution success rate to spatial pass rate is approximately 41%, showing that runnable code does not necessarily yield spatially coherent visual output. These findings show that programmatic video generation evaluation should go beyond executability. PRISM provides a principled benchmark for advancing spatially coherent code generation.
AI Impact Assessments
(1 model)Scientific Impact Assessment: PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
1. Core Contribution
PRISM introduces a large-scale benchmark of 10,372 human-calibrated instruction-code pairs for evaluating LLMs on programmatic video generation via Manim, a Python-based mathematical animation framework. The benchmark is bilingual (English/Chinese), spans 437 subject categories, and is approximately 20× larger than prior programmatic video generation benchmarks. The paper's central conceptual contribution is the identification and quantification of the Execution-Spatial Gap: the finding that code executability does not imply spatial correctness, with an average ~41% drop from execution success rate to spatial pass rate across seven evaluated LLMs.
The paper also proposes a four-metric "funnel-style" evaluation framework: Code-Level Reliability (executability), Spatial Reasoning (layout correctness across animation frames), Prompt-Aware Dynamic Visual Complexity (PADVC), and Temporal Density (TD). This multi-layered evaluation moves beyond binary pass/fail code execution metrics toward nuanced assessment of visual output quality.
2. Methodological Rigor
Data curation: The hierarchical pipeline—deterministic hard filtering via low-level rendering-engine parsing, boundary calibration with 3,000 expert-annotated examples, and human spot-check review—is methodologically sound. The decision to avoid VLM-based quality assessment is well-justified given known VLM weaknesses in fine-grained spatial reasoning. Starting from 30,000+ raw candidates and filtering to 10,372 demonstrates substantial quality control effort.
Evaluation metrics: The spatial reasoning metric is carefully designed with three mutually exclusive failure modes (overlap, out-of-bounds, leakage) and includes a hierarchy-aware parser with false-positive suppression. PADVC and TD are mathematically formulated with centered scores fitted to reference distributions, which is reasonable for detecting both under- and over-production of dynamics.
Experimental concerns: The evaluation uses only a 20% random subset rather than the full benchmark, which is somewhat unusual for a benchmark paper and raises questions about statistical power and representativeness. The bootstrap confidence interval for the Execution-Spatial Gap is reported, which is good practice. Temperature is set to 0.7 with single-attempt evaluation, which is standard but limits assessment of model variability.
One methodological weakness is that the PADVC metric involves several design choices (the concave exponent p=0.7, the specific formulation of PVD, the Gaussian centering) that appear somewhat ad hoc. While reasonable, the sensitivity of results to these choices is not thoroughly explored. The paper also acknowledges that temporal ordering is not explicitly evaluated since sequential script execution generally preserves it—this is a reasonable simplification but limits the benchmark's temporal reasoning assessment.
3. Potential Impact
Direct impact on programmatic generation: PRISM fills a genuine gap. Prior benchmarks for Manim/programmatic video generation were small (hundreds of samples) and primarily evaluated executability. The 20× scale increase and spatial evaluation framework provide a much-needed standardized testbed.
Broader implications for LLM evaluation: The Execution-Spatial Gap finding is the paper's most impactful insight—it demonstrates that code correctness metrics are insufficient proxies for output quality in generative settings. This lesson extends beyond Manim to any domain where LLMs generate code that produces visual or spatial outputs (CAD, web design, SVG generation, robotics).
Training data resource: The 10,372 curated instruction-code pairs could serve as high-quality training data for fine-tuning models on programmatic video generation, though the paper does not explore this.
Limitations on broader applicability: The benchmark is tightly coupled to Manim despite claims of generalizability. The educational knowledge visualization domain, while practical, is somewhat niche. The bilingual (English/Chinese) scope is useful but limited.
4. Timeliness & Relevance
The paper addresses a timely need. With the explosion of LLM-based code generation and growing interest in automated content creation, the gap between code that runs and code that produces correct visual output is increasingly important. The educational video generation use case is particularly relevant given the proliferation of AI-assisted educational tools.
The benchmark evaluates very recent models (GPT-5.4, Gemini 3.1 Pro Preview, Claude Sonnet 4.6, Kimi K2), which makes the findings immediately relevant to the current state of the art. However, this also means the benchmark's longevity depends on how quickly models improve on these tasks.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Summary
PRISM makes a solid empirical contribution by providing a substantially larger benchmark for programmatic video generation and introducing the Execution-Spatial Gap concept, which is both well-supported and practically important. The evaluation framework is more rigorous than prior work, though the diagnostic metrics (PADVC, TD) add complexity without proportional insight. The paper's primary limitation is its tight coupling to Manim and the educational domain, which constrains generalizability. Nevertheless, the core finding—that runnable code ≠ spatially correct output—is valuable for the broader code generation community.
Generated May 20, 2026
Comparison History (22)
Paper 2 has higher likely scientific impact: it introduces an end-to-end, censoring-aware reformulation that makes LLMs directly usable for survival analysis, a high-value and widely applicable clinical/statistical problem. The approach targets real-world deployment (local, compact models), shows improvements over Cox and multiple deep survival baselines on two cohorts, and could influence both medical ML and survival methodology. Paper 1 is a strong, timely benchmark with useful metrics, but its impact is more bounded to programmatic video/code-generation evaluation, whereas Paper 2’s applications and cross-field relevance (clinical prediction, biostatistics, ML) are broader.
Paper 1 is likely to have higher impact because it targets the broadly relevant and timely problem of always-on personal assistants, introducing a benchmark that meaningfully expands context (long-horizon histories, multi-service dependencies, GUI/CLI across devices) and enables evaluation of proactive assistance under realistic noise. Its automated data-generation pipeline and evidence of scalable training gains increase practical adoption and downstream research utility. Paper 2 is strong and novel for programmatic spatiotemporal reasoning, but its application scope is narrower (code-based video generation) and may influence fewer adjacent fields than general agentic personal-assistant evaluation.
Paper 1 addresses a fundamental and well-studied problem in knowledge graph reasoning (complex query answering) with a novel framework that scales to multi-variable queries—a significant gap in existing methods. It introduces both a new method (NS3) and a benchmark for joint-ranking evaluation, advancing the field methodologically. Paper 2 introduces a useful benchmark for programmatic video generation evaluation, but benchmarks generally have lower lasting impact than novel algorithmic frameworks. Paper 1's contribution to knowledge representation and reasoning has broader applicability across AI subfields and addresses a more foundational computational challenge.
PRISM introduces a large-scale, well-structured benchmark (10,372 pairs, 437 categories) addressing a timely gap in programmatic video generation evaluation. Its novel funnel-style evaluation framework and the discovery of the 'Execution-Spatial Gap' provide actionable insights with broad applicability across code generation, video synthesis, and spatial reasoning. Paper 1 (OSCToM) makes solid contributions to Theory of Mind benchmarking but addresses a narrower problem domain. PRISM's scale, bilingual coverage, and relevance to the rapidly growing field of code-generating LLMs give it broader potential impact.
Paper 2 (PRISM) is likely to have higher scientific impact because it delivers a large, reusable benchmark (10,372 pairs, bilingual) with concrete, automatable metrics and a clear empirical finding (Execution–Spatial Gap) that can drive model and method development. Its applications span programmatic video generation, spatial-temporal reasoning, code generation, and multimodal evaluation, making it broadly useful and timely. Paper 1 offers valuable evaluation taxonomies and insights, but is positioned as a protocol demonstration with a small-scale run and no benchmark release, limiting immediate adoption and downstream impact.
Paper 2 addresses a more fundamental and broadly applicable problem—systematic diagnosis of LLM agent failures at scale—which impacts the entire LLM agent ecosystem. It introduces a novel formalization (corpus-level trace diagnostics) and demonstrates concrete downstream improvements (30.4pp gains). Paper 1, while valuable as a benchmark for programmatic video generation, serves a narrower community. Paper 2's methodology (multi-agent diagnostic system) has wider applicability across any LLM agent deployment, making it more likely to influence research and practice broadly.
Paper 1 addresses a critical, timely issue in frontier AI evaluation by advocating for open-world, real-world tasks to supplement automated benchmarks. Its focus on long-horizon capabilities and safety has profound implications for AI policy, safety, and broad real-world applications. While Paper 2 offers a rigorous benchmark for a specific domain (programmatic video generation), Paper 1's conceptual shift in how we evaluate state-of-the-art AI systems ensures a much broader and more significant impact across the entire AI ecosystem.
Paper 2 addresses a fundamental and critical challenge in AI: the limitations of static benchmarks for evaluating frontier models. By proposing a framework for open-world, long-horizon evaluations, it has a broader potential impact across the entire field of AI development and safety compared to Paper 1, which focuses on a more specific niche of programmatic video generation.
Paper 2 (PRISM) likely has higher impact because it introduces a large, multilingual benchmark and a multi-metric evaluation framework that can become shared infrastructure for the community, enabling standardized comparison and driving progress across program synthesis, vision-language, and temporal/spatial reasoning. Its dataset scale and the identified “Execution–Spatial Gap” provide actionable, broadly relevant insights. Paper 1 offers useful controlled experiments and negative findings about LLM agent behavior in code optimization, but its scope is narrower and more diagnostic than enabling, which may limit cross-field uptake.
Paper 1 addresses a fundamental and highly timely problem in AI safety and agentic systems (trust calibration and human oversight) by introducing a rigorous mathematical framework based on Preferential Bayesian Optimization. Its formalized approach to progressive autonomy has broader applicability and higher potential impact across various domains of autonomous AI compared to Paper 2, which, while valuable, is restricted to benchmarking programmatic video generation.