This assessment is based on version 1 of this paper. Version 2 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
Philip Schroeder, Thomas Weng, Karl Schmeckpeper, Eric Rosen, Stephen Hart, Ondrej Biza
Abstract
Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. To address this limitation, we introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including GPT-5 and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking.
AI Impact Assessments
(3 models)Scientific Impact Assessment: SOLE-R1
1. Core Contribution
SOLE-R1 addresses a critical bottleneck in robot learning: the reliance on hand-engineered reward functions or human supervision for reinforcement learning. The paper introduces a video-language reasoning model that produces per-timestep chain-of-thought (CoT) reasoning and dense progress estimates from raw video and natural language goals, serving as the sole reward signal for online RL. The key novelty lies in the integration of three components: (a) a large-scale video trajectory and reasoning synthesis pipeline generating temporally grounded CoT traces with continuous progress supervision, (b) a hybrid training framework combining SFT with RLVR, and (c) the demonstration that this approach enables zero-shot online RL from random initialization across diverse tasks and environments without any ground-truth rewards, demonstrations, or task-specific tuning.
The problem is well-motivated: current frontier VLMs (GPT-5, Gemini-3-Pro) systematically fail as reward functions due to reward hacking, where RL policies exploit perceptual errors rather than solve tasks. SOLE-R1's explicit spatiotemporal reasoning mechanism directly targets this vulnerability.
2. Methodological Rigor
Strengths in experimental design: The evaluation is remarkably comprehensive—40 tasks across 4 simulation environments (RoboSuite, ManiSkill, Meta-World, LIBERO) and real-robot experiments with a Franka arm. The zero-shot protocol is strict: no ground-truth rewards, no demonstrations, no task-specific tuning, no offline trajectories. This is a genuinely challenging evaluation setting that prior work typically relaxes.
Baselines are strong and appropriate: Comparing against GPT-5, Gemini-3-Pro, Gemini Robotics-ER 1.5, plus specialized reward models (LIV, ReWiND, VLAC, RoboReward) provides a thorough competitive landscape. The inclusion of frontier commercial models strengthens credibility.
Ablation studies are well-designed: The paper systematically ablates CoT reasoning, non-expert training data, and foundational spatial reasoning, using the perceived-vs-true success framework to distinguish reward hacking from signal-limited failures. The scaling analysis (Figure 6) showing performance as a function of training task diversity is particularly informative.
Potential concerns: (1) Real-world experiments use only a single seed, limiting statistical confidence. (2) The 50% success threshold for "solving" a task is somewhat arbitrary—14 of 40 tasks fall below this threshold even for SOLE-R1. (3) The progress supervision for real-world videos relies on temporal order as a proxy, which is a relatively weak signal. (4) The paper compares an 8B parameter specialized model against general-purpose models used zero-shot, which is not entirely apples-to-apples—though this is acknowledged and the specialized reward model baselines partially address this.
3. Potential Impact
Immediate impact on robot learning: If the claimed results hold, SOLE-R1 could substantially reduce the engineering burden for deploying RL on new manipulation tasks. The ability to specify tasks in natural language and learn from scratch is a significant step toward general-purpose robot learning.
Broader methodological contributions: The video trajectory synthesis pipeline (generating non-expert trajectories with grounded CoT) is reusable and could benefit other video understanding tasks. The hybrid SFT+RLVR training framework for reward models could inspire similar approaches in other domains requiring calibrated scalar predictions from language models.
Limitations on immediate practical impact: (1) Inference speed (1s/frame for SOLE-R1, much slower for baselines) creates a bottleneck for real-time RL. (2) The approach still requires ~2 hours of real-world interaction for simple tasks. (3) The model is trained on specific manipulation task families, and generalization to fundamentally different task categories remains uncertain.
4. Timeliness & Relevance
This work is highly timely. It sits at the intersection of three active research fronts: (a) foundation models for robotics, (b) reward learning/specification for RL, and (c) reasoning-augmented VLMs (following DeepSeek-R1 and similar). The reward hacking problem it addresses is increasingly recognized as a fundamental barrier. The release of model checkpoints, training data, and code further amplifies potential impact.
The work also arrives as the field debates whether to pursue end-to-end VLA fine-tuning versus modular approaches with separate perception/reward/policy components. SOLE-R1 provides evidence for the modular approach's viability.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Additional observations: The paper's framing as comparing against "GPT-5" and "Gemini-3-Pro" suggests frontier models circa late 2025/early 2026. The systematic superiority of a specialized 8B model over much larger general-purpose models reinforces the value of domain-specific training for grounded robotics tasks. The finding that expert-only training increases reward hacking is a particularly valuable insight for the community.
Generated Mar 31, 2026
Comparison History (220)
Paper 2 tackles a fundamental bottleneck in embodied AI: the reward specification problem. By enabling zero-shot online RL from random initialization using a VLM-based reward signal resistant to reward hacking, it unlocks highly scalable, autonomous robot learning without human-engineered rewards or demonstrations. While Paper 1 presents an impressive hierarchical orchestration method for existing policies, Paper 2's paradigm-shifting approach to automated reward generation offers broader theoretical implications for the future of reinforcement learning and self-supervised agents across diverse environments.
Paper 2 addresses a fundamental bottleneck in reinforcement learning: automated reward specification. By developing a video-language reasoning model that provides robust, zero-shot rewards without reward hacking, it enables robots to learn unseen tasks from random initialization using only natural language goals. This capability drastically reduces human engineering, offering broader, transformative impact across AI and robotics compared to Paper 1. While Paper 1 presents a highly innovative visuo-tactile world model for contact-rich manipulation, Paper 2's potential to enable scalable, open-ended real-world learning gives it a higher overall scientific impact.
Paper 1 addresses a fundamental and timely problem: reward specification for robot RL, which is a major bottleneck. Its approach of using VLM reasoning as the sole reward with demonstrated robustness to reward hacking outperforming GPT-5 and Gemini-3-Pro shows strong methodological rigor and broad applicability. Paper 2's fast one-shot imitation is impressive and practical, but more incremental within imitation learning. Paper 1's contribution to autonomous reward generation has broader potential to transform how robots learn without human supervision, giving it wider cross-field impact.
Paper 1 presents a novel eversion-based robotic platform validated in intact human cadaver anatomy, addressing a clear unmet clinical need in spinal intrathecal interventions with rigorous multimodal validation and quantified safety benefits. Its potential to enable safer neurological procedures has direct, high-value real-world medical impact. Paper 2 offers a strong ML contribution on VLM-based rewards for robot RL, but operates in a rapidly evolving, crowded field where advances are quickly superseded. Paper 1's demonstrated clinical translation pathway and tangible patient-safety implications suggest broader and more durable scientific impact.
Paper 1 presents a paradigm-shifting approach by solving the reward specification bottleneck in reinforcement learning. By enabling zero-shot online RL using a video-language model for dense reward generation, it eliminates the need for manual reward design or demonstrations, paving the way for scalable autonomous learning. While Paper 2 offers significant computational and architectural efficiencies, Paper 1's integration of spatiotemporal CoT reasoning to prevent reward hacking addresses a fundamental, long-standing challenge in AI and robotics, resulting in broader cross-disciplinary impact and higher potential for long-term scientific transformation.
Paper 1 is more novel and broadly impactful: it reframes reward specification for online RL by introducing a dedicated video-language reward model that outputs dense progress signals via spatiotemporal reasoning, enabling zero-shot RL without ground-truth rewards or demonstrations and explicitly addressing reward hacking under partial observability. This directly targets a major bottleneck in scalable robot learning and could generalize across many embodiments and tasks. Paper 2 is strong and practical for humanoid execution via persistent 3D object state, but its core contribution is more domain-specific and likely narrower in cross-field influence.
Paper 2 (SOLE-R1) likely has higher scientific impact due to its broader, more general contribution: enabling online RL with a single video-language reward model, removing reliance on task-specific rewards, demonstrations, or tuning. This could transfer across many domains where reward specification is a bottleneck and addresses a key failure mode (reward hacking under partial observability/distribution shift). The methodology includes a scalable data/reasoning synthesis pipeline and hybrid SFT+RL training, with results spanning multiple sims plus real robot and many unseen tasks. Paper 1 is strong but more specialized to long-context policy scaling via fast-weight TTT.
Paper 2 has higher potential impact because it addresses the fundamental reward specification bottleneck in reinforcement learning. While Paper 1 offers an impressive synthetic data augmentation pipeline for 1-shot learning, Paper 2 entirely eliminates the need for demonstrations or manual reward engineering by using a VLM with spatiotemporal chain-of-thought reasoning as a dense reward generator. Enabling zero-shot online RL from random initialization while resisting reward hacking is a major breakthrough that applies broadly across robotics and general RL, offering wider theoretical and practical implications.
Paper 1 is more novel and broadly impactful: it reframes on-robot RL by using a dedicated video-language reasoning model as the sole reward, addressing a central bottleneck (reward specification and reward hacking under partial observability) with a scalable data/reasoning synthesis pipeline and hybrid SFT+RL training. The demonstrated capability—zero-shot online RL from scratch on real robots without rewards, demos, or tuning—has wide applicability beyond a single domain. Paper 2 is methodologically strong and timely for driving, but its core innovation is more domain-specific and incremental relative to existing Dreamer/world-model conditioning work.
Paper 1 is more novel and timely: it tackles a central bottleneck in VLM-supervised RL—reward unreliability under partial observability and reward hacking—by introducing a dedicated video-language reasoning reward model that yields dense, per-timestep progress signals and enables zero-shot online RL without task rewards, demos, or tuning. This “sole reward” framing plus a scalable reasoning-synthesis pipeline and real-robot validation suggests strong practical applicability and broad influence across robot learning, RLHF-style supervision, and evaluation. Paper 2 is impactful but more incremental within existing world-model/alignment trends.