DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong
Abstract
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
AI Impact Assessments
(3 models)Scientific Impact Assessment: DreamDojo
1. Core Contribution
DreamDojo addresses a fundamental bottleneck in robot world modeling: the scarcity of diverse, action-labeled robot data. The paper's key insight is to exploit the massive availability of egocentric human videos as a pretraining source for robot world models, bridging the embodiment gap through continuous latent actions that serve as unified proxy action labels. The system comprises several interconnected contributions:
The problem framing is compelling: rather than scaling expensive robot teleoperation data, leverage the nearly unlimited supply of human activity videos that share underlying physics with robot manipulation.
2. Methodological Rigor
The experimental design is thorough with several commendable aspects:
Systematic ablations: The paper carefully isolates contributions from (a) latent vs. action-free vs. ground-truth action conditioning (Tab. 2), (b) data mixture scaling (Tab. 3), (c) architectural choices (Tab. 5), and (d) distillation benefits (Tab. 6-7). Each ablation answers a specific question clearly.
Benchmark construction: Six OOD evaluation benchmarks are designed to test generalization — including novel objects, novel environments (via Gemini-generated background edits), and counterfactual actions. This is more rigorous than the in-distribution evaluation typical of prior robot world models.
Human preference evaluation: For scenarios without ground truth, 12 evaluators assess physics correctness and action following via a randomized side-by-side protocol, which is appropriate.
Downstream validation: Policy evaluation (Pearson r=0.995 with real-world success rates), model-based planning (17% improvement, ~2× over uniform sampling), and live teleoperation demonstrate practical utility.
However, there are methodological concerns:
3. Potential Impact
Near-term applications: The demonstrated downstream applications — policy evaluation without real deployment, model-based planning for test-time correction, and real-time teleoperation — address practical needs in robotics development. Policy evaluation alone could save significant real-world testing costs.
Scaling paradigm: The core idea of using human videos to bootstrap robot world models establishes a scalable data pipeline. If the approach generalizes as claimed, it could fundamentally change how robot simulators are developed — shifting from hand-crafted physics engines to learned generative models trained on internet-scale data.
Cross-field influence: The latent action framework for cross-embodiment transfer could influence broader areas including imitation learning from observation, sim-to-real transfer, and embodied AI development. The temporal consistency loss (Eq. 4) is a simple but potentially broadly useful technique for video prediction models.
Infrastructure contribution: The 44k-hour DreamDojo-HV dataset, if released, would be a significant community resource, though the paper doesn't explicitly commit to full public release (it's described as "in-house").
4. Timeliness & Relevance
This work arrives at an inflection point where video generation models (Cosmos, WAN, Sora) have matured sufficiently to serve as backbones for world models, and the robotics community is actively seeking scalable simulation approaches. The paper directly addresses the data-efficiency bottleneck that limits current robot world models to narrow domains. The use of latent actions to bridge unlabeled video to action-conditioned prediction is timely, building on a rapidly growing line of work (LAPA, AdaWorld, Genie).
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Scalability: The 256-GPU training setup raises questions about accessibility, though the 2B model variant offers a more practical option. The distillation to RTX 5090-compatible inference is a positive step.
Summary
DreamDojo represents a significant engineering and scientific achievement in scaling world models for robotics through human video pretraining. The continuous latent action framework elegantly solves the action label gap, and the comprehensive evaluation demonstrates meaningful OOD generalization. While the absolute performance gaps in some metrics are modest and the comparison scope could be broader, the paradigm of scaling human videos for robot world models is well-motivated and potentially transformative for the field.
Generated Apr 19, 2026
Comparison History (500)
DreamDojo presents a foundation world model trained on 44k hours of human video—the largest such dataset for world model pretraining—addressing fundamental challenges in robot learning (data scarcity, action label scarcity) with a unified latent action framework. Its breadth of applications (teleoperation, policy evaluation, model-based planning) and generalist open-world capabilities give it broader impact across robotics, computer vision, and generative modeling. ParkourFormer, while impressive for humanoid parkour locomotion with strong results, addresses a narrower problem domain. DreamDojo's paradigm of learning world models from massive unlabeled human video has greater potential to influence the field at large.
Paper 1 offers higher potential impact by tackling a fundamental bottleneck in embodied AI: action label scarcity. By leveraging a massive 44k-hour human video dataset using continuous latent proxy actions, DreamDojo creates a generalist world model bridging human video and robotic control. This broadly impacts simulation, planning, and policy evaluation. While Paper 2 presents a rigorous, practical improvement for VLA models via CoT representation shaping, Paper 1's paradigm of extracting physical dynamics from unlabeled, internet-scale human video represents a more transformative and scalable leap toward open-world generalist agents.
Paper 2 has higher potential impact due to its more novel and broadly enabling contribution: a large-scale, generalist robot world model trained from 44k hours of human egocentric video, addressing a key bottleneck (scarce action labels) via continuous latent proxy actions and adding practical real-time distillation. World models can underpin many downstream areas (planning, control, evaluation, teleop, simulation) and generalize across robots and tasks, giving wider cross-field reach. Paper 1 is strong engineering/SOTA for embodied VLMs, but is more benchmark- and recipe-centric with narrower conceptual novelty.
Paper 2 (DreamDojo) likely has higher scientific impact due to its broader, foundational contribution: a large-scale generalist robot world model pretrained on an unprecedented 44k hours of human egocentric video, plus a principled approach to missing action labels via continuous latent actions. This can enable multiple downstream capabilities (planning, evaluation, teleoperation) across many agents and domains, improving cross-field impact and timeliness. Paper 1 shows strong performance gains by integrating world-action priors into VLA policies, but is more specific to policy steering and narrower in generality.
Paper 2 presents a massive-scale foundation world model trained on 44k hours of video, addressing broader challenges in simulating open-world, contact-rich tasks. Its introduction of latent actions and real-time generation enables versatile applications like policy evaluation and model-based planning, offering wider impact across robotics and AI compared to Paper 1's more focused, albeit effective, keypoint-based policy transfer method.
Paper 1 likely has higher impact due to its foundation-model framing: a large-scale (44k hours) pretraining recipe for a generalist robot world model with a novel latent-action proxy to exploit unlabeled human video, plus distillation to real-time use. This can broadly enable teleoperation, planning, and policy evaluation across many tasks and platforms, making it timely and widely applicable. Paper 2 is strong and methodologically complete for single-video sim2real dexterous skill acquisition, but its scope is narrower and may generalize less broadly than a generalist world model.
DreamDojo has higher potential impact due to its broader scope as a foundation world model trained on 44k hours of human videos, addressing the fundamental challenge of learning world dynamics from unlabeled data at unprecedented scale. Its contributions—continuous latent actions for cross-embodiment transfer, real-time distillation, and multiple downstream applications (teleoperation, policy evaluation, planning)—represent a more foundational advance. While RoboDream offers valuable practical contributions for data synthesis, DreamDojo's approach of leveraging massive human video data for generalizable robot world models addresses a more fundamental bottleneck and has broader applicability across robotics.
Qwen-VLA presents a more comprehensive unified framework addressing manipulation, navigation, and trajectory prediction across multiple embodiments with strong empirical results on numerous benchmarks. Its breadth of unification (tasks, environments, embodiments) and demonstrated real-world performance (ALOHA experiments, zero-shot dynamic manipulation) suggest broader immediate impact. DreamDojo is innovative in leveraging human videos for world modeling, but its impact is more focused on simulation/prediction rather than direct action generation. Qwen-VLA's practical applicability across diverse robotic platforms gives it an edge in real-world deployment potential.
DreamDojo tackles a fundamental bottleneck in generalist robotics by leveraging massive unlabelled human video data to create a foundation world model. While ElegantVLA provides valuable efficiency improvements, DreamDojo's approach to learning diverse interactions and enabling simulation, planning, and policy evaluation at scale represents a more paradigm-shifting contribution with broader implications for embodied AI.
Paper 2 likely has higher scientific impact due to its timeliness and broad relevance: foundation world models from massive human video data align with a major robotics/AI trend and can generalize across many tasks and embodiments. Its potential applications (teleoperation, planning, policy evaluation) span robotics, simulation, and ML, and the scale (44k hours) plus techniques (latent actions, distillation to real-time) suggest strong downstream influence. Paper 1 is novel and rigorous in robot design, but its impact is more domain-specific to certain platforms/morphologies.