Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Open-H-Embodiment Consortium, :, Nigel Nelson, Juo-Tung Chen, Jesse Haworth, Xinhao Chen, Lukas Zbinden, Dianye Huang
Abstract
Autonomous medical robots hold promise to improve patient outcomes, reduce provider workload, democratize access to care, and enable superhuman precision. However, autonomous medical robotics has been limited by a fundamental data problem: existing medical robotic datasets are small, single-embodiment, and rarely shared openly, restricting the development of foundation models that the field needs to advance. We introduce Open-H-Embodiment, the largest open dataset of medical robotic video with synchronized kinematics to date, spanning more than 49 institutions and multiple robotic platforms including the CMR Versius, Intuitive Surgical's da Vinci, da Vinci Research Kit (dVRK), Rob Surgical BiTrack, Virtual Incision's MIRA, Moon Surgical Maestro, and a variety of custom systems, spanning surgical manipulation, robotic ultrasound, and endoscopy procedures. We demonstrate the research enabled by this dataset through two foundation models. GR00T-H is the first open foundation vision-language-action model for medical robotics, which is the only evaluated model to achieve full end-to-end task completion on a structured suturing benchmark (25% of trials vs. 0% for all others) and achieves 64% average success across a 29-step ex vivo suturing sequence. We also train Cosmos-H-Surgical-Simulator, the first action-conditioned world model to enable multi-embodiment surgical simulation from a single checkpoint, spanning nine robotic platforms and supporting in silico policy evaluation and synthetic data generation for the medical domain. These results suggest that open, large-scale medical robot data collection can serve as critical infrastructure for the research community, enabling advances in robot learning, world modeling, and beyond.
AI Impact Assessments
(3 models)Scientific Impact Assessment: Open-H-Embodiment
1. Core Contribution
Open-H-Embodiment addresses a well-recognized structural bottleneck in autonomous surgical robotics: the absence of a large-scale, multi-embodiment, openly shared dataset of medical robotic video with synchronized kinematics. The paper delivers three interrelated contributions: (1) a 770-hour dataset spanning 20 robotic platforms, 49+ institutions, and 119 constituent datasets; (2) GR00T-H, a vision-language-action (VLA) model post-trained on this corpus that achieves the first reported end-to-end autonomous suturing completion on the SutureBot benchmark; and (3) Cosmos-H-Surgical-Simulator, the first multi-embodiment action-conditioned world model for surgical simulation.
The core insight — that the scaling laws validated in general-purpose robotics (via Open-X-Embodiment and successors) should transfer to the medical domain given sufficient domain-specific data — is not itself novel, but the execution is significant. The paper correctly identifies that general-purpose VLAs fail on surgical tasks not due to architectural limitations but because of a domain gap that cannot be bridged without in-domain data. Open-H is the first serious attempt to close this gap at scale.
2. Methodological Rigor
Dataset construction is handled with considerable care. The adoption of LeRobot v2.1 format with healthcare-specific schema extensions, structured READMEs documenting platform nuances (clutching, cable wear, operator skill), and the explicit acknowledgment of data heterogeneity challenges are commendable engineering decisions. The five-tier realism spectrum (simulation through clinical) is well-motivated.
GR00T-H evaluation follows a reasonably rigorous protocol. The end-to-end suturing evaluation uses a per-setup matched design where all models attempt the same configuration. Statistical reporting includes Clopper-Pearson confidence intervals and Fisher's exact tests with Holm-Bonferroni correction. However, several concerns arise:
Cosmos-H-Surgical-Simulator evaluation is acknowledged as preliminary. The reliance on L1 and SSIM metrics for action-conditioned generation is a known limitation — these metrics don't capture surgical-specific fidelity (instrument position accuracy, tissue interaction plausibility). The paper is transparent about this gap.
Dataset composition bias is a significant concern: CMR Versius clinical data accounts for ~65% of total hours (499/770), creating a heavily skewed distribution. While sampling caps at 20% during GR00T-H training partially mitigate this, the effective diversity is less than the headline numbers suggest.
3. Potential Impact
The potential impact is substantial across multiple dimensions:
Infrastructure contribution: The dataset itself, released under CC-BY-4.0 with model weights and code publicly available, could become foundational infrastructure analogous to what ImageNet was for vision or Open-X-Embodiment for general robotics. The multi-institutional, multi-platform nature lowers barriers for researchers without access to specific surgical platforms.
Clinical trajectory: While far from clinical deployment (the paper appropriately notes this), the work establishes a credible research pipeline from data collection through foundation model training to policy evaluation. The ex vivo results on pork belly tissue represent meaningful progress toward clinical relevance.
World modeling: Cosmos-H-Surgical-Simulator opens a new direction — using world models for in silico policy evaluation could dramatically reduce the cost of surgical robot policy development, which currently requires expensive physical robot time.
Community building: The 49-institution consortium itself is a significant organizational achievement that could catalyze sustained collaboration in a field historically fragmented by proprietary platforms and institutional silos.
4. Timeliness & Relevance
The paper arrives at an inflection point where foundation models have demonstrated clear scaling benefits in general robotics, but surgical robotics remains locked out due to data scarcity. The projected surgeon shortage provides genuine urgency. The timing relative to the explosion of VLA models (OpenVLA, π0, GR00T-N1) makes this contribution directly actionable — the community now has both the architectures and (with Open-H) the data to explore surgical foundation models.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Overall Assessment: This is a high-impact community resource paper that addresses a genuine bottleneck. The dataset contribution alone justifies publication at a top venue. The foundation model results, while preliminary, provide compelling evidence that domain-specific pretraining matters for surgical robotics. The work's greatest impact will likely be measured not by the specific models presented, but by what the community builds on this infrastructure over the coming years.
Generated Apr 24, 2026
Comparison History (99)
Paper 1 addresses a fundamental infrastructure gap by releasing the largest open medical robotics dataset spanning 49 institutions and multiple platforms, plus demonstrates two foundation models. Datasets serve as durable community infrastructure enabling broad downstream research, and medical robotics has enormous real-world impact potential. Paper 2 offers a clever orchestration approach with strong benchmark gains, but is a methodological contribution more incremental in scope. Paper 1's combination of critical shared data, high-stakes clinical domain, novel multi-embodiment models, and enabling role for the field gives it broader and more lasting impact.
Paper 1 introduces critical, large-scale open infrastructure spanning 49+ institutions and multiple robotic platforms, addressing a fundamental data bottleneck in medical robotics. It enables two foundation models with demonstrated real-world clinical relevance and high potential for improving patient outcomes and democratizing care. The dataset itself provides lasting community value. Paper 2 presents solid technical work on visuo-tactile world models but has narrower scope and impact. Paper 1's breadth, timeliness, real-world medical applications, and role as reusable infrastructure suggest substantially higher scientific impact.
Paper 1 likely has higher impact due to creating open, large-scale, multi-institution, multi-embodiment medical-robotics infrastructure (dataset + foundation models), enabling broad downstream research (robot learning, simulation/world models, benchmarking) in a high-stakes, timely domain. Its real-world application potential is substantial (surgical/ultrasound/endoscopy) and the openness/scale can catalyze community progress. Paper 2 is methodologically novel and compelling for fast skill acquisition, but appears narrower in scope and lacks the same field-building infrastructure and cross-platform medical relevance.
While Paper 1 presents impressive methodological advances in scalable robot navigation, Paper 2 has higher potential impact because it overcomes a critical, long-standing bottleneck in medical robotics: data scarcity. By releasing the largest multi-institutional, multi-embodiment open dataset alongside the first foundation VLA and world models for surgical robotics, Paper 2 democratizes research in a traditionally closed, high-stakes field. Its potential to accelerate autonomous healthcare applications and surgical simulation promises a more profound societal and cross-disciplinary paradigm shift than state-of-the-art improvements in general navigation.
Paper 1 offers significantly broader impact by providing critical infrastructure—the largest open dataset and first foundation models for medical robotics. By solving a fundamental data bottleneck across multiple platforms and 49 institutions, it will catalyze widespread downstream research in embodied AI, surgical autonomy, and world modeling. In contrast, while Paper 2 is highly rigorous and clinically valuable, it describes a specialized device for a specific anatomical region, limiting its breadth of scientific impact compared to the paradigm-shifting foundational approach of Paper 1.
Paper 1 likely has higher impact due to major infrastructure contribution: the largest open multi-institution, multi-embodiment medical robotics dataset with synchronized video+kinematics, enabling broad downstream research and standardization. It targets a high-stakes, application-rich domain (medical robotics) and demonstrates two foundation models including cross-embodiment simulation and end-to-end suturing progress. Paper 2 is a strong, efficient architectural improvement with clear performance gains, but its scope is narrower and less likely to reshape community resources than a large open dataset plus baseline models in a critical domain.
Paper 2 has higher potential scientific impact because it addresses a fundamental data bottleneck by providing the largest multi-embodiment open dataset for medical robotics. Large-scale foundational datasets historically catalyze massive field-wide advancements. Furthermore, its applications in autonomous surgery hold profound real-world, life-saving potential. While Paper 1 offers a valuable algorithmic improvement for humanoid control, Paper 2 provides critical infrastructure and novel foundation models that will likely empower widespread, cross-disciplinary research and drive the next generation of medical AI.
Paper 2 likely has higher scientific impact because it provides broadly enabling infrastructure: the largest open multi-institution, multi-robot medical robotics dataset with synchronized kinematics, plus baseline foundation and world models. Open datasets typically catalyze widespread follow-on work, benchmarking, reproducibility, and cross-lab progress, especially in a high-stakes, data-constrained domain (medical robotics). While Paper 1 is methodologically novel and strong on long-context robot control, Paper 2’s openness, scale, and direct clinical relevance increase breadth, adoption potential, and timeliness.
Paper 1 likely has higher impact: it delivers a large, open, multi-institution, multi-embodiment medical robotics dataset plus two foundational models (vision-language-action and action-conditioned world model), creating durable community infrastructure in a high-stakes domain. The openness, scale, and clinical relevance can catalyze broad follow-on work (robot learning, simulation, evaluation benchmarks) and address a central bottleneck in medical autonomy. Paper 2 is novel and broadly applicable, but is primarily a method; Paper 1 combines dataset + models + ecosystem effects with strong timeliness for foundation-model robotics.
Paper 2 introduces a large-scale, multi-institution, multi-embodiment open dataset addressing a fundamental data bottleneck in medical robotics, and demonstrates two foundation models built upon it. Its impact is broad: enabling community-wide advances in robot learning, world modeling, and surgical automation with direct healthcare relevance. As critical infrastructure spanning 49 institutions and multiple platforms, it has enduring, field-shaping value. Paper 1 offers a solid, well-analyzed methodological contribution (ego-dynamics-augmented world model with cross-chassis adaptation) but is narrower in scope and incremental within autonomous driving. Paper 2's timeliness, breadth, and infrastructural nature give it substantially higher potential impact.