Kurate.org
Rankings
Back to Rankings

TriPhysBench: Evaluating Physical Intelligence for Embodied Agents

Zhongxiang Sun, Chenghao Yang, Haojie Ding, Boheng Zhang, Fan Yang, Qingsong Wen, Kun Wang

Nov 1, 2026arXiv:2611.01934v1
cs.AIcs.CV
Share
Scorecard· 16/16
5.5/10 impact

Competent benchmark plus training baseline filling a real niche, but incremental ideas, a small 110-question test set, single-seed results, and noisy reference assumptions limit impact.

Abstract

Embodied agents need physical intelligence to understand their surroundings through physical perception, physical knowledge retrieval, and physical reasoning. These capabilities must work together to interpret events and make quantitative judgments. Existing benchmarks advance physical understanding and general-purpose tool use, yet the coordination of these capabilities in physical scenarios remains insufficiently characterized. Inspired by Popper's Three Worlds, we introduce TriPhysBench, a benchmark of 410 questions from 200 videos that brings physical perception, physical knowledge retrieval, and physical reasoning into a shared evaluation setting. Its measurement-grounded and knowledge-grounded subtasks impose complementary demands on obtaining observations and acquiring external specifications, then applying them to quantitative problems. Among six representative vision-language models, the best achieves an answer accuracy of only 50.38%, revealing limited performance on tasks that require these capabilities to work together. We further introduce TriPhys, a framework that learns a unified policy for physical perception, knowledge retrieval, and quantitative reasoning from complete tool-integrated trajectories. The policy uses intermediate reasoning to guide retrieval and computation, then incorporates tool feedback into subsequent physical inference. On 110 questions from videos excluded from training, TriPhys improves answer and solution scores across three Qwen configurations, with a 17.7% relative gain in answer accuracy for TriPhys-27B over its Qwen3.6-27B backbone. This work provides a benchmark and a learning pathway toward embodied agents that move beyond recognizing events to understanding and reasoning about the physical processes behind them.

AI Impact Assessments

(1 model)

Impact Assessment: TriPhysBench

Core Contribution

The paper introduces TriPhysBench, a benchmark of 410 open-form quantitative physics questions drawn from 200 real videos (EgoExo4D for MPR, Physics-aware-videos for KPR). It targets the joint demand of physical perception, external knowledge retrieval, and quantitative reasoning. Difficulty is controlled by withholding observations (MPR) or external specifications (KPR). Questions come in four answer forms, and the construction pipeline uses physical event graphs. The second contribution is TriPhys, a recipe of self-sampled, verified, tool-integrated trajectories with answer-block rewriting, followed by LoRA SFT and length-normalized full-trajectory DPO. It is applied to three Qwen backbones.

The conceptual novelty is modest. The "Popper's Three Worlds" framing is largely decorative, and both the event-graph construction (from OmniGAIA) and the SFT+DPO recipe (ToRA, ETO-style) are adapted from prior work. The combination is useful: real video, retrieval, and numerical computation in one benchmark, with partial-credit answer scoring and a rubric-based solution score. Table 1 shows no existing benchmark covers all of these.

Methodological Rigor

Strengths:

  • The video-level train/test split avoids leakage across questions from the same recording.
  • Answer scoring is detailed (exponential decay for point estimates, IoU plus center calibration for intervals).
  • The LLM-judge robustness check is good: a second judge from a different family, Pearson 0.877, identical model rankings, and no self-preference.
  • The three-judge review reports 95.6% agreement.
  • The paper is candid about limitations: no DPO-only ablation, no separate ablation of rewriting or loss masking, single seed, and no confidence intervals.

    • Weaknesses:

  • The test set is only 110 questions, with a stated standard error of about 0.048 on ACC. The 16–18% relative gains are around 0.08 absolute, so they are roughly 1.6 SE. That is suggestive but not strongly significant, and with one training seed it is fragile. Individual stage increments (e.g., +0.012 from DPO on 9B, SFT leaving 27B ACC unchanged) are well within noise. The sensitivity analysis shows 0.03–0.05 swings per subtask from hyperparameters alone.
  • Reference answers depend on assumptions: a retrieved "typical" hand mass of 409.6 g, a 26" rim diameter, and a whole-hand approximation of wrist speed. The ground truth is therefore partly an arbitrary modeling choice, and the benchmark rewards guessing the annotators' constants. The 30% cutoff softens this, but it weakens the claim of measuring physical intelligence. The review pass flagged 18 questions, 16 of them in the training split, and ten concerned modeling simplifications.
  • The review model calls disable tools, so the reference calculations were not independently executed.
  • Training reward and trajectory filtering use the same ACC and SCS metrics as evaluation, with the same judge family, which invites metric overfitting.
  • Test questions come from the same distribution and annotation process as training, so the generalization claim is narrow.
  • The "perception" component is weak. Many MPR observations are given when difficulty is Easy. The video-dependence labels are judgments made during construction, not measured by ablation. The paper never shows that models actually need the video, or that the three capabilities are separable or coordinate.
  • Measurement tolerances of ±5/12/25% are construction tolerances, and the quadratic amplification for kinetic energy makes some answers noisy, which bounds achievable accuracy.

    • Potential Impact

    The benchmark fills a niche between QuantiPhy, PhysBench, and OmniGAIA-style tool agents. It may be adopted by groups working on agentic video reasoning and physical VLMs. The data pipeline and trajectory-learning recipe are reusable but not groundbreaking. With 410 questions and a relatively narrow mechanics focus (Work & energy dominates, 160 of 410), it is small by modern benchmark standards. The 50% frontier accuracy leaves headroom, but saturation may come quickly given that fine-tuning a 27B model on 300 questions surpasses GPT-5.5. This suggests much of the "gap" is format and constant-selection alignment rather than deep physical capability. Embodied relevance is largely rhetorical, since there is no actual embodied control or interaction.

    Timeliness & Relevance

    Agentic multimodal reasoning with search and code tools, together with the push for quantitative grounding of VLMs, is a hot area. The paper fits the trend of benchmarks plus RL/DPO post-training. It is moderately timely, though the field has many crowded physical-reasoning benchmarks.

    Strengths & Limitations

    Strengths:

  • Coherent integration of perception, retrieval, and computation on real video.
  • Process-level solution scoring with validated judge consistency.
  • Thorough appendices: prompts, hyperparameters, and partition details, with a code and data link.
  • Clear failure diagnosis (variable identification is the dominant failure, 56–92%).
  • An honest limitations discussion.

    • Limitations:

  • Small test set and single seed, so the evidence is only moderately strong.
  • Ground-truth assumptions are noisy.
  • Limited model diversity (Qwen only for training, six models evaluated).
  • No ablation of the key design choices.
  • The diagnostic failure analysis covers only three models and relies on LLM-judge thresholds.
  • Evaluation scores are not checked against human annotation.
  • The framing overreaches ("embodied agents").

    • Overall this is a solid, competently executed benchmark-plus-baseline paper. It is useful to a subcommunity but incremental in ideas, and the empirical claims are limited by small-sample noise.

    Rating:5.5/ 10
    Significance 5Rigor 5.5Novelty 5Clarity 6.5

    Generated Nov 3, 2026

    Comparison History (0)

    No comparisons yet.