Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Open-H-Embodiment Consortium, :, Nigel Nelson, Juo-Tung Chen, Jesse Haworth, Xinhao Chen, Lukas Zbinden, Dianye Huang
First large-scale open cross-embodiment surgical robotics dataset that unblocks a decade-long data bottleneck, released openly with models and code, likely to become reference infrastructure for the subfield.
Abstract
Autonomous medical robots hold promise to improve patient outcomes, reduce provider workload, democratize access to care, and enable superhuman precision. However, autonomous medical robotics has been limited by a fundamental data problem: existing medical robotic datasets are small, single-embodiment, and rarely shared openly, restricting the development of foundation models that the field needs to advance. We introduce Open-H-Embodiment, the largest open dataset of medical robotic video with synchronized kinematics to date, spanning more than 50 institutions and multiple robotic platforms including the CMR Versius, Intuitive Surgical's da Vinci, da Vinci Research Kit (dVRK), Rob Surgical BiTrack, Virtual Incision's MIRA, Moon Surgical Maestro, and a variety of custom systems, spanning surgical manipulation, robotic ultrasound, and endoscopy procedures. We demonstrate the research enabled by this dataset through two foundation models. GR00T-H is the first open foundation vision-language-action model for medical robotics, which is the only evaluated model to achieve full end-to-end task completion on a structured suturing benchmark (25% of trials vs. 0% for all others) and achieves 64% average success across a 29-step ex vivo suturing sequence. We also train Cosmos-H-Surgical-Simulator, the first action-conditioned world model to enable multi-embodiment surgical simulation from a single checkpoint, spanning nine robotic platforms and supporting in silico policy evaluation and synthetic data generation for the medical domain. These results suggest that open, large-scale medical robot data collection can serve as critical infrastructure for the research community, enabling advances in robot learning, world modeling, and beyond.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Open-H-Embodiment
1. Core Contribution
Open-H-Embodiment addresses a fundamental infrastructure gap in medical robotics: the absence of a large-scale, multi-embodiment, openly shared dataset with synchronized video and kinematics. The paper delivers three interlinked contributions: (1) a 770-hour dataset spanning 20 robotic platforms, 49+ institutions, and 119 constituent datasets; (2) GR00T-H, a vision-language-action (VLA) model post-trained on this corpus that achieves the first reported end-to-end task completion on the SutureBot suturing benchmark (25% vs. 0% for all baselines); and (3) Cosmos-H-Surgical-Simulator, the first multi-embodiment action-conditioned world model for surgical simulation from a single checkpoint.
The paper's central thesis—that the scaling laws demonstrated in general-purpose robotics (Open-X-Embodiment, RT-2-X) can transfer to the medical domain given sufficient diverse data—is directly tested and supported by the experimental results. This is a significant conceptual validation because the surgical domain has unique challenges (deformable tissues, endoscopic optics, irreversible actions, regulatory constraints) that could plausibly invalidate scaling assumptions from rigid-object manipulation.
2. Methodological Rigor
Dataset construction is handled with considerable care. The adoption of LeRobot v2.1 format, structured READMEs documenting platform-specific nuances (clutching mechanisms, operator skill levels), and per-dataset normalization statistics demonstrate attention to the practical challenges of multi-institution data aggregation. The five-tier realism spectrum (simulation through clinical) is well-structured for progressive validation.
GR00T-H evaluation is reasonably thorough. The paper evaluates across four dimensions: end-to-end task completion (n=20 per model), out-of-distribution generalization (n=30), data efficiency (33% vs. 100% fine-tuning data), and multi-embodiment transfer (three platforms). Statistical reporting includes Clopper-Pearson confidence intervals and Fisher's exact tests with Holm-Bonferroni correction. The per-setup evaluation protocol for end-to-end suturing (holding setup constant across models) strengthens internal validity.
However, there are methodological concerns. The 25% end-to-end success rate, while the only non-zero result, is based on 5/20 trials—a small absolute number. The LingBot-VA baseline was evaluated in a separate session with "best effort" to mirror conditions, introducing potential confounds. The ex vivo evaluation (64% average across 29 subtasks) tests individual subtask success but doesn't report end-to-end completion rates on biological tissue, which would be the most clinically meaningful metric.
Cosmos-H-Surgical-Simulator evaluation relies on L1 and SSIM metrics against ground-truth replay, which the authors themselves acknowledge are limited—they don't capture surgery-specific fidelity (instrument positioning, tool-tissue interaction plausibility). The absence of closed-loop evaluation or domain-specific metrics weakens the world model contribution relative to the dataset and VLA contributions.
3. Potential Impact
Immediate impact: The dataset itself, released under CC-BY-4.0, could become foundational infrastructure for the surgical robotics community, analogous to what ImageNet was for computer vision or Open-X-Embodiment for general robotics. The 770-hour scale represents a >35x increase over the largest prior dataset (ImitateCholec, ~20 hours), and the multi-embodiment nature is entirely novel.
Downstream applications: Beyond VLA training and world modeling, the authors correctly note the dataset supports monocular depth estimation, instrument segmentation, procedural phase recognition, and language-conditioned planning. The clinical procedure annotations (phase and gesture labels for hundreds of hours of Versius data) add value for surgical workflow analysis.
Industry and clinical pathway: The involvement of major surgical robotics companies (CMR Surgical contributing 489 hours, plus Intuitive/dVRK, Virtual Incision, Rob Surgical, Moon Surgical) signals potential for translational impact, though the paper is appropriately cautious about clinical deployment timelines.
Broader field influence: This work could catalyze a paradigm shift in surgical robotics from single-institution, single-platform research toward community-driven data sharing, analogous to the shift seen in NLP and general robotics.
4. Timeliness & Relevance
The paper arrives at a critical inflection point. Foundation models have transformed general robotics (OpenVLA, π0, GR00T-N1), but SutureBot recently demonstrated their complete failure on surgical tasks. This paper directly addresses that documented failure with a data-centric solution. The projected surgeon shortage (13,000–86,000 by 2036) provides genuine clinical urgency. The convergence of VLA architectures, video foundation models, and the demonstrated need for domain-specific pretraining makes this contribution highly timely.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
6. Additional Observations
The paper represents a significant organizational achievement—coordinating data collection across 49 institutions with heterogeneous hardware and regulatory environments. The practical contribution of establishing data standards, conversion tools, and documentation templates may have impact beyond the specific models trained. The observation about hardware drift robustness (GR00T-H maintaining performance despite instrument wear) is a practically important finding for real-world deployment.
Generated May 5, 2026
Comparison History (90)
Paper 1 addresses a fundamental data-scarcity bottleneck in a high-stakes, historically closed field (medical robotics) by releasing the largest multi-institutional dataset and the first foundation and world models for this domain. Large-scale open datasets historically trigger paradigm shifts by serving as critical infrastructure for future research. While Paper 2 presents a highly effective orchestration framework for general robotics, Paper 1's democratization of surgical data and multi-platform simulation enables entirely new sub-fields of research, promising broader and more transformative long-term impact across both AI and medicine.
Paper 1 offers higher potential impact by solving a critical bottleneck in medical robotics: data scarcity. By releasing the largest open, multi-embodiment dataset across 49 institutions, it democratizes a highly siloed field. Additionally, introducing the first open foundation VLA and multi-embodiment surgical world models enables massive downstream breakthroughs in healthcare automation. While Paper 2 presents a novel methodological advance in visuo-tactile manipulation, Paper 1 provides foundational infrastructure at an unprecedented scale, carrying broader real-world implications for patient care and catalyzing field-wide advancement.
Paper 1 provides critical open infrastructure (largest multi-embodiment medical robotics dataset across 49+ institutions) plus two foundation models addressing a fundamental data bottleneck in a high-stakes domain. Datasets that enable foundation models tend to have broad, lasting community impact and catalyze many downstream works. Paper 2 presents an elegant, efficient one-shot learning method with strong results, but its impact is narrower and more algorithm-specific. Paper 1's breadth, timeliness for medical AI, and role as reusable infrastructure give it higher estimated scientific impact.
Paper 2 has higher potential impact because it addresses a critical data bottleneck in a high-stakes domain by open-sourcing the largest multi-embodiment medical robotics dataset to date. While Paper 1 offers strong methodological improvements for scalable navigation, Paper 2 unlocks an entirely siloed field. By providing foundational infrastructure-including the first open vision-language-action model and an action-conditioned world model for surgical robotics across 49 institutions-Paper 2 catalyzes broad downstream research in healthcare, fundamentally advancing both surgical autonomy and future patient care.
Paper 1 offers higher scientific impact by providing foundational infrastructure—a massive open dataset and the first multi-embodiment foundation models for medical robotics. This solves a critical field-wide bottleneck, enabling a paradigm shift toward embodied AI across various surgical procedures and platforms. In contrast, Paper 2 presents a highly rigorous but narrowly focused hardware innovation for spinal navigation. Paper 1's generalizable dataset and AI models will catalyze widespread research and drive rapid advancements across the entire field of autonomous medical robotics, granting it significantly broader scientific influence.
Paper 2 likely has higher impact due to its creation of large-scale, multi-institution, multi-platform open infrastructure for medical robotics—an area with strong real-world relevance and significant data scarcity. The dataset can enable many downstream works (foundation models, benchmarking, simulation, policy learning) across robotics, ML, and healthcare, amplifying breadth and timeliness. It also demonstrates clear methodological follow-through via two substantial models (VLA and world model). Paper 1 is a strong, efficient architectural innovation, but its impact is narrower and more incremental relative to the field-wide leverage of a major open dataset in a high-stakes domain.
Paper 2 introduces critical infrastructure—the largest open medical robotics dataset spanning 49 institutions and multiple platforms—addressing a fundamental data bottleneck. It enables broad community advances via two foundation models with demonstrated results in a high-stakes medical domain. Its breadth of impact, real-world clinical applications, and enabling nature give it greater potential reach. Paper 1 presents a solid, well-validated method for object persistence in humanoid VLA, but its impact is narrower and more incremental compared to the field-enabling dataset and foundation models in Paper 2.
Paper 2 addresses a critical bottleneck—the lack of open medical robotics data—by creating the largest multi-institution, multi-embodiment dataset (49 institutions, multiple platforms). Such infrastructure enables broad community advances and has direct high-stakes real-world applications in healthcare. It also demonstrates two foundation models. While Paper 1 offers an elegant methodological innovation (context scaling via test-time training), its impact is more incremental within robot learning. Paper 2's dataset serves as durable, field-defining infrastructure with clearer breadth of impact, timeliness, and life-saving relevance.
Paper 2 likely has higher impact due to its large, open, multi-institution, multi-embodiment dataset and demonstrated foundation models, creating reusable infrastructure for a safety-critical, high-value domain. Its real-world applicability in medical robotics is immediate, and broad adoption could catalyze progress across robot learning, simulation, and world modeling. Paper 1 is highly novel in synthetic data generation from minimal demos and may advance open-world manipulation, but its impact depends more on uptake and generalization beyond showcased setups, whereas Paper 2’s dataset-first contribution is broadly enabling and timely.
Paper 2 introduces critical infrastructure—the largest open multi-embodiment medical robotics dataset spanning 49+ institutions and multiple platforms—addressing a fundamental data bottleneck that constrains an entire field. It enables foundation models with demonstrated real-world clinical relevance (suturing, ultrasound, endoscopy) and provides lasting community value. Paper 1 is a solid, well-formulated methodological contribution with rigorous theory and strong results, but its impact is narrower and incremental within driving world models. Paper 2's breadth, high-stakes healthcare applications, and enabling role for future research give it substantially greater potential impact.
First large-scale open cross-embodiment surgical robotics dataset that unblocks a decade-long data bottleneck, released openly with models and code, likely to become reference infrastructure for the subfield.