This assessment is based on version 1 of this paper. Version 2 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
A multimodal and temporal foundation model for virtual patient representations at healthcare system scale
Andrew Zhang, Tong Ding, Sophia J. Wagner, Caiwei Tian, Ming Y. Lu, Rowland Pettit, Joshua E. Lewis, Alexandre Misrahi
Abstract
Modern medicine generates vast multimodal data across siloed systems, yet no existing model integrates the full breadth and temporal depth of the clinical record into a unified patient representation. We introduce Apollo, a multimodal temporal foundation model trained and evaluated on over three decades of longitudinal hospital records from a major US hospital system, composed of 25 billion records from 7.2 million patients, representing 28 distinct medical modalities and 12 major medical specialties. Apollo learns a unified representation space integrating over 100 thousand unique medical events in our clinical vocabulary as well as images and clinical text. This "atlas of medical concepts" forms a computational substrate for modeling entire patient care journeys comprised of sequences of structured and unstructured events, which are compressed by Apollo into virtual patient representations. To assess the potential of these whole-patient representations, we created 322 prognosis and retrieval tasks from a held-out test set of 1.4 million patients. We demonstrate the generalized clinical forecasting potential of Apollo embeddings, including predicting new disease onset risk up to five years in advance (95 tasks), disease progression (78 tasks), treatment response (59 tasks), risk of treatment-related adverse events (17 tasks), and hospital operations endpoints (12 tasks). Using feature attribution techniques, we show that model predictions align with clinically-interpretable multimodal biomarkers. We evaluate semantic similarity search on 61 retrieval tasks, and moreover demonstrate the potential of Apollo as a multimodal medical search engine using text and image queries. Together, these modeling capabilities establish the foundation for computable medicine, where the full context of patient care becomes accessible to computational reasoning.
AI Impact Assessments
(3 models)Scientific Impact Assessment: Apollo — A Multimodal Temporal Foundation Model for Virtual Patient Representations
Core Contribution
Apollo represents the most comprehensive attempt to date at building a unified foundation model over the full breadth of the electronic health record. The key novelty lies in the integration of 28 distinct medical modalities — structured data (diagnoses, labs, medications, vitals, flowsheets), clinical text (progress notes, diagnostic reports), and medical images (pathology WSIs, blood smears, EM) — into a single temporally-aware transformer trained on 25 billion records from 7.2 million patients spanning 33 years. Unlike prior EHR foundation models (BEHRT, Med-BERT, MOTOR, Chronoformer) that are limited to structured data or curated ICU cohorts like MIMIC-IV, Apollo operates across the full continuum of inpatient and outpatient care from a real multi-institutional health system.
The architectural approach is pragmatic: modality-specific frozen encoders (GatorTron for text, TITAN/CONCH for pathology, DinoBloom for hematology) produce embeddings that are projected into a shared space, then processed by a temporal transformer with masked token modeling. Patient embeddings are generated by appending a masked diagnosis token at inference. This design isolates the temporal integration module from raw PHI — a meaningful privacy consideration.
Methodological Rigor
Strengths: The evaluation framework is exceptionally thorough — 322 tasks across 5 prognostic categories and retrieval, with 1.4M held-out patients. The Cox proportional hazards framework with IPCW-based metrics is statistically appropriate for right-censored survival data. Case-cohort sampling, proper train/validation/test separation, and re-estimation of baseline hazards on unmodified validation sets demonstrate careful methodological practice. Bootstrap confidence intervals (100 iterations) with formal significance testing add rigor. Calibration analysis across all tasks is commendable.
Limitations: The primary baseline is age-sex Cox regression — an extremely weak comparator. While informative as a statistical reference, the absence of head-to-head comparisons with established clinical risk scores (ASCVD, CHA₂DS₂-VASc, Framingham, HEART) for well-studied endpoints limits clinical interpretability. The authors acknowledge this but argue computational infeasibility at scale. The supervised baseline (same architecture without pretraining) provides stronger evidence of pretraining value, but is only evaluated on 28 cancer progression tasks. The structured-only ablation (+0.025 mean AUROC from multimodal integration) shows a real but modest gain from unstructured modalities on the cancer benchmark, raising questions about cost-benefit tradeoffs.
The linear probing evaluation (frozen embeddings + Cox regression with PCA to 50 dimensions) is standard for foundation model assessment but understates potential performance with fine-tuning. Conversely, it means the reported numbers are conservative estimates.
Potential Impact
The paper establishes several capabilities with broad implications:
1. Clinical forecasting at scale: 261 prognostic tasks with strong performance (e.g., heart failure 0.88 AUROC, schizophrenia 0.92, type 2 diabetes 0.85) suggest genuine clinical utility for risk stratification, though deployment would require prospective validation.
2. Multimodal semantic retrieval: The ability to query a 1.4M patient database using text descriptions or pathology images is novel and practically valuable for cohort discovery and clinical trial matching. The glioblastoma image retrieval example (retrieving IDH-wildtype patients from an external TCGA slide) is particularly compelling.
3. Atlas of medical concepts: The unified embedding space showing clinically coherent clustering across modalities (Figure 2) has value for understanding medical ontology relationships and could serve as a foundation for downstream medical AI systems.
4. Interpretability: Both local (LOTO) and global (Integrated Gradients) attribution methods recover clinically plausible risk factors, though the authors appropriately caution these are associative, not causal.
Timeliness & Relevance
This work addresses a critical bottleneck: the fragmentation of clinical data across modality-specific silos. With healthcare generating massive data volumes but utilizing <3%, the need for unified representations is acute. The "virtual patient" concept parallels the "virtual cell" paradigm in computational biology, positioning this work at an emerging conceptual frontier. The scale of data (7.2M patients, 25B records) represents what is achievable within a single large health system, making this a realistic template for other institutions.
Strengths
Limitations
Overall Assessment
Apollo is a landmark systems-level contribution that demonstrates the feasibility and utility of whole-record multimodal patient modeling at healthcare-system scale. While individual task performances may not dramatically exceed specialized models, the generality and breadth of a single model across 322 tasks is the primary achievement. The work sets a new standard for EHR foundation model evaluation comprehensiveness, though stronger baselines and external validation would substantially strengthen the claims.
Generated Apr 21, 2026
Comparison History (285)
Paper 2 introduces a healthcare-scale multimodal foundation model trained on 25 billion records from 7.2 million patients, with immediate real-world clinical applications across prognosis, disease progression, and treatment response. Its breadth of impact spans medicine and AI, addressing a timely need for integrated patient representations. Paper 1 offers valuable theoretical clarity for discrete diffusion models, with rigorous unifying insights, but its impact is narrower and more specialized. Paper 2's scale, applicability, and potential to transform clinical practice give it substantially broader scientific and societal impact.
Paper 1 likely has higher impact due to its scale (7.2M patients, 25B records), broad multimodal integration, and immediate high-stakes clinical applications across many tasks and specialties. A unified temporal patient foundation model could influence clinical forecasting, retrieval, and downstream decision support, affecting healthcare, ML, and informatics communities. While Paper 2 is novel and valuable for materials/engineering inverse design, its domain scope is narrower and dependent on surrogate fidelity and dataset coverage. Paper 1 is also highly timely given foundation-model trends in medicine.
Paper 1 presents a large-scale multimodal foundation model trained on unprecedented clinical data (25 billion records, 7.2 million patients), with extensive validation across 322 clinically-relevant tasks. Its potential to transform healthcare through unified patient representations addresses a critical real-world need with broad impact. Paper 2 offers a theoretically rigorous but narrower contribution to copyright detection; while timely and methodologically sound, its impact scope is more limited and empirical validation appears less extensive. Paper 1's combination of scale, novelty, real-world applicability, and thorough evaluation gives it substantially higher scientific impact potential.
Paper 2 likely has higher impact due to its scale, breadth, and cross-domain relevance: a multimodal, longitudinal foundation model trained on decades of data from 7.2M patients across 28 modalities with 322 downstream tasks. It enables broad real-world applications (risk prediction, progression, treatment response, operations, retrieval/search) and could influence multiple fields (clinical informatics, ML, public health, systems engineering). Paper 1 is a strong, timely systems/ML contribution for efficient DiT quantization, but its impact is narrower and more incremental within model compression and diffusion inference.
While Paper 1 presents a highly practical engineering optimization for LLM inference, Paper 2 demonstrates transformative scientific impact. Apollo's unprecedented scale—integrating 30 years of longitudinal, multimodal data from 7.2 million patients across 28 modalities—creates a foundational model for healthcare. Its ability to predict disease onset, progression, and treatment response across 322 clinical tasks bridges AI and medicine. Paper 2 offers significantly broader cross-disciplinary applications, greater potential to improve human health, and represents a massive systemic leap compared to Paper 1's narrower, albeit valuable, algorithmic efficiency improvement.
Paper 1 presents a massive-scale, multimodal foundation model for healthcare, integrating 30 years of data across 7.2 million patients. Its ability to predict disease onset, progression, and treatment response across 322 clinical tasks offers profound real-world medical applications. While Paper 2 provides a valuable algorithmic refinement for LLM post-training with impressive deployment, Paper 1 demonstrates a paradigm-shifting approach to computable medicine. Its novelty in temporal and multimodal virtual patient representation, combined with its capacity to fundamentally transform broad healthcare systems and clinical forecasting, gives it a higher potential scientific impact.
Paper 1 offers unprecedented scale and innovation by integrating 28 medical modalities across 7.2 million patients into a unified foundation model. Its demonstrated success across 322 clinical tasks highlights profound real-world applications that could transform clinical decision-making and healthcare globally. While Paper 2 presents a valuable algorithmic optimization for LLM training, Paper 1's massive interdisciplinary bridge between advanced AI and comprehensive, longitudinal clinical medicine promises a much broader, paradigm-shifting scientific and societal impact.
Paper 2 likely has higher scientific impact due to its unprecedented scale (7.2M patients, 25B records), breadth of modalities/specialties, and strong timeliness for healthcare foundation models. It enables many real-world applications (risk prediction, progression, treatment response, operations, retrieval/search) and could become infrastructure for multiple downstream clinical/ML tasks, affecting medicine, informatics, and policy. Methodologically, large held-out evaluation across 322 tasks suggests rigor and generality. Paper 1 is innovative for structure-predictor-guided molecule optimization but relies heavily on proxy/held-out computational metrics and faces larger translation hurdles to validated drug discovery.
Apollo represents a landmark-scale multimodal foundation model trained on 25 billion records from 7.2 million patients across 28 modalities, addressing a fundamental unsolved problem in healthcare data integration. Its broad clinical applications (prognosis, treatment response, retrieval across 322 tasks) offer transformative real-world impact across medicine. While PolyQ is a solid, well-executed engineering contribution enabling efficient edge LLM inference, its impact is narrower and incremental within a crowded quantization field. Apollo's breadth, timeliness, and potential to enable 'computable medicine' give it substantially higher scientific impact.
Paper 1 likely has higher scientific impact due to its large-scale multimodal temporal foundation model trained on decades of real-world clinical data, broad applicability across prognosis, retrieval, and clinical operations, and potential to change healthcare decision support and “computable medicine.” Its scope (28 modalities, millions of patients, hundreds of tasks) suggests substantial translational and cross-disciplinary impact. Paper 2 is methodologically rigorous and timely for efficient LLM training, but its impact is narrower (optimizer/subspace theory and practice) and more incremental relative to broader clinical and societal consequences of Paper 1.