Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Open-H-Embodiment Consortium, :, Nigel Nelson, Juo-Tung Chen, Jesse Haworth, Xinhao Chen, Lukas Zbinden, Dianye Huang
Abstract
Autonomous medical robots hold promise to improve patient outcomes, reduce provider workload, democratize access to care, and enable superhuman precision. However, autonomous medical robotics has been limited by a fundamental data problem: existing medical robotic datasets are small, single-embodiment, and rarely shared openly, restricting the development of foundation models that the field needs to advance. We introduce Open-H-Embodiment, the largest open dataset of medical robotic video with synchronized kinematics to date, spanning more than 49 institutions and multiple robotic platforms including the CMR Versius, Intuitive Surgical's da Vinci, da Vinci Research Kit (dVRK), Rob Surgical BiTrack, Virtual Incision's MIRA, Moon Surgical Maestro, and a variety of custom systems, spanning surgical manipulation, robotic ultrasound, and endoscopy procedures. We demonstrate the research enabled by this dataset through two foundation models. GR00T-H is the first open foundation vision-language-action model for medical robotics, which is the only evaluated model to achieve full end-to-end task completion on a structured suturing benchmark (25% of trials vs. 0% for all others) and achieves 64% average success across a 29-step ex vivo suturing sequence. We also train Cosmos-H-Surgical-Simulator, the first action-conditioned world model to enable multi-embodiment surgical simulation from a single checkpoint, spanning nine robotic platforms and supporting in silico policy evaluation and synthetic data generation for the medical domain. These results suggest that open, large-scale medical robot data collection can serve as critical infrastructure for the research community, enabling advances in robot learning, world modeling, and beyond.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Open-H-Embodiment
1. Core Contribution
Open-H-Embodiment addresses a fundamental infrastructure gap in medical robotics: the absence of a large-scale, multi-embodiment, openly shared dataset with synchronized video and kinematics. The paper delivers three interlinked contributions: (1) a 770-hour dataset spanning 20 robotic platforms, 49+ institutions, and 119 constituent datasets; (2) GR00T-H, a vision-language-action (VLA) model post-trained on this corpus that achieves the first reported end-to-end task completion on the SutureBot suturing benchmark (25% vs. 0% for all baselines); and (3) Cosmos-H-Surgical-Simulator, the first multi-embodiment action-conditioned world model for surgical simulation from a single checkpoint.
The paper's central thesis—that the scaling laws demonstrated in general-purpose robotics (Open-X-Embodiment, RT-2-X) can transfer to the medical domain given sufficient diverse data—is directly tested and supported by the experimental results. This is a significant conceptual validation because the surgical domain has unique challenges (deformable tissues, endoscopic optics, irreversible actions, regulatory constraints) that could plausibly invalidate scaling assumptions from rigid-object manipulation.
2. Methodological Rigor
Dataset construction is handled with considerable care. The adoption of LeRobot v2.1 format, structured READMEs documenting platform-specific nuances (clutching mechanisms, operator skill levels), and per-dataset normalization statistics demonstrate attention to the practical challenges of multi-institution data aggregation. The five-tier realism spectrum (simulation through clinical) is well-structured for progressive validation.
GR00T-H evaluation is reasonably thorough. The paper evaluates across four dimensions: end-to-end task completion (n=20 per model), out-of-distribution generalization (n=30), data efficiency (33% vs. 100% fine-tuning data), and multi-embodiment transfer (three platforms). Statistical reporting includes Clopper-Pearson confidence intervals and Fisher's exact tests with Holm-Bonferroni correction. The per-setup evaluation protocol for end-to-end suturing (holding setup constant across models) strengthens internal validity.
However, there are methodological concerns. The 25% end-to-end success rate, while the only non-zero result, is based on 5/20 trials—a small absolute number. The LingBot-VA baseline was evaluated in a separate session with "best effort" to mirror conditions, introducing potential confounds. The ex vivo evaluation (64% average across 29 subtasks) tests individual subtask success but doesn't report end-to-end completion rates on biological tissue, which would be the most clinically meaningful metric.
Cosmos-H-Surgical-Simulator evaluation relies on L1 and SSIM metrics against ground-truth replay, which the authors themselves acknowledge are limited—they don't capture surgery-specific fidelity (instrument positioning, tool-tissue interaction plausibility). The absence of closed-loop evaluation or domain-specific metrics weakens the world model contribution relative to the dataset and VLA contributions.
3. Potential Impact
Immediate impact: The dataset itself, released under CC-BY-4.0, could become foundational infrastructure for the surgical robotics community, analogous to what ImageNet was for computer vision or Open-X-Embodiment for general robotics. The 770-hour scale represents a >35x increase over the largest prior dataset (ImitateCholec, ~20 hours), and the multi-embodiment nature is entirely novel.
Downstream applications: Beyond VLA training and world modeling, the authors correctly note the dataset supports monocular depth estimation, instrument segmentation, procedural phase recognition, and language-conditioned planning. The clinical procedure annotations (phase and gesture labels for hundreds of hours of Versius data) add value for surgical workflow analysis.
Industry and clinical pathway: The involvement of major surgical robotics companies (CMR Surgical contributing 489 hours, plus Intuitive/dVRK, Virtual Incision, Rob Surgical, Moon Surgical) signals potential for translational impact, though the paper is appropriately cautious about clinical deployment timelines.
Broader field influence: This work could catalyze a paradigm shift in surgical robotics from single-institution, single-platform research toward community-driven data sharing, analogous to the shift seen in NLP and general robotics.
4. Timeliness & Relevance
The paper arrives at a critical inflection point. Foundation models have transformed general robotics (OpenVLA, π0, GR00T-N1), but SutureBot recently demonstrated their complete failure on surgical tasks. This paper directly addresses that documented failure with a data-centric solution. The projected surgeon shortage (13,000–86,000 by 2036) provides genuine clinical urgency. The convergence of VLA architectures, video foundation models, and the demonstrated need for domain-specific pretraining makes this contribution highly timely.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
6. Additional Observations
The paper represents a significant organizational achievement—coordinating data collection across 49 institutions with heterogeneous hardware and regulatory environments. The practical contribution of establishing data standards, conversion tools, and documentation templates may have impact beyond the specific models trained. The observation about hardware drift robustness (GR00T-H maintaining performance despite instrument wear) is a practically important finding for real-world deployment.
Generated May 5, 2026