RareLens: Towards End-to-End Rare Disease Care via Aligning Divergent Large Language Model Reasoning
Xi Chen, Hongru Zhou, Shiyu Feng, Hanyu Zhou, Huahui Yi, Rongsheng Wang, Tiancheng He, Kun Wang
Ambitious end-to-end system with a transferable conceptual thesis (learning from model divergence) and comprehensive evaluation, tempered by case-report-heavy data, LLM-as-judge circularity, and no statistical testing.
Abstract
Rare diseases collectively affect an estimated 3.5% to 5.9% of the population, yet more than 70% of patients are misdiagnosed and many endure years of evaluation before a diagnosis is reached, because early presentations are nonspecific and relevant expertise is scarce and unevenly distributed. Artificial intelligence could provide support, but existing systems address isolated stages of care, overwhelmingly diagnosis. They typically depend on the results of downstream investigations, and they treat the variability between models as noise to be eliminated. Here we present RareLens, a system that supports clinical decision-making across the entire rare disease trajectory by exploiting this variability. When heterogeneous large language models evaluate the same case, they generate divergent but complementary reasoning, which RareLens aligns and calibrates into a single convergent, actionable decision at each stage. Four coordinated modules perform primary-visit risk screening, diagnosis, treatment planning and prognosis. Developed and evaluated on RareBench, a real-world dataset of 157,525 cases spanning all 33 Orphanet categories and more than 7,000 conditions, RareLens outperformed every frontier model tested, including GPT-5, DeepSeek-R1, Claude-3.7-Sonnet and Gemini-2.5-Pro, at each stage. It achieved an area under the curve of 0.917 for screening and top-1 accuracies of 65.5% and 89.8% for diagnosis and treatment. In an external study spanning 1,287 cases and 23 physicians, autonomous RareLens and physicians assisted by RareLens both substantially outperformed unaided physicians. These findings indicate that aligning divergent model reasoning, rather than scaling a single model, offers a generalizable strategy for high-uncertainty clinical decision-making.
AI Impact Assessments
(1 models)Scientific Impact Assessment: RareLens
1. Core Contribution
RareLens makes two interlocking contributions. First is a conceptual reframing: rather than treating inter-model disagreement between heterogeneous LLMs as noise to be averaged out or resolved by voting, the authors treat divergent reasoning as an *informative, complementary signal* that can be learned from via supervised alignment. Second is a systems contribution: an end-to-end rare disease decision-support pipeline spanning four coupled stages (screening → diagnosis → treatment → prognosis), reflecting the longitudinal clinical trajectory rather than the isolated diagnosis step that prior work has almost exclusively addressed.
The most genuinely novel structural choice is reframing rare disease recognition as a *universal population-scale screening task* at the primary visit — using only history and physical exam, with a large non-rare negative cohort — rather than diagnosis among already-suspected patients. This targets the real clinical bottleneck (the failure to suspect a rare disease at first contact) that upstream systems ignore.
2. Methodological Rigor
The methodology is unusually thorough for a clinical-AI paper. Strengths include: strict case-level partitioning to prevent leakage; staged information release to prevent cross-stage leakage; a large development corpus (157,525 cases across all 33 Orphanet categories); comparison against 12–16 frontier and open models per module; comparison against a full family of training-free ensemble baselines (score fusion, rank fusion, voting, central tendency); ablations on the number of contributing LLMs, feature groups, and reasoning paths; a reliability/reproducibility analysis under stochastic decoding (ICC 0.93, weighted κ up to 0.89); and an external human study with 23 physicians across 10 hospitals under a crossed design.
However, several rigor concerns temper this. The ground truth and much of the evaluation rely on LLM-as-judge adjudication (GPT-5/GPT-5-nano), validated against humans on subsamples (κ = 0.95 diagnosis, but only 0.63 for treatment — moderate). RareBench itself is constructed largely from PubMed case reports, which are curated, self-contained, and diagnostically "clean" — very different from messy real-world primary-visit notes; this likely inflates screening and diagnostic performance. Critically, the paper reports "results descriptively without statistical hypothesis testing" — no confidence intervals on most headline comparisons, no significance tests distinguishing RareLens from the best baseline. Some external-study downstream stages rest on very small paired samples (e.g., treatment rescue "80% vs 0%").
3. Potential Impact
The impact potential is high but bimodal. The conceptual claim — that learning from foundation-model diversity beats selecting or scaling a single model — is broadly generalizable and could influence ensemble strategies well beyond rare disease, in any high-uncertainty prediction domain. This is the paper's most transferable idea.
The clinical system addresses a large, underserved problem (rare diseases affect 3.5–5.9% of the population with 70%+ misdiagnosis rates). If validated prospectively, primary-visit screening could materially shorten the diagnostic odyssey. The distilled, locally deployable 32B RareAlert model is a pragmatic and deployment-conscious design choice.
An intellectually honest and impactful secondary finding is the human-AI collaboration result: RareLens alone outperformed physicians+RareLens, which underperformed... consistent with prior meta-analytic evidence. The candid discussion of algorithm aversion and workflow design is valuable and cautionary for the whole clinical-AI deployment community.
4. Timeliness & Relevance
Extremely timely. Multi-LLM ensembling, agentic clinical systems, and rare-disease AI (cf. the recent *Nature* agentic diagnosis paper cited as ref 9) are all active frontiers. The framing directly counters the prevailing "scale a single model" paradigm at a moment when that assumption is being questioned.
5. Strengths & Limitations
Strengths: end-to-end coverage of the full care trajectory (genuinely rare in this literature); a clear, testable conceptual thesis backed by divergence analyses and rescue-rate evidence concentrated on hard cases; comprehensive baselines and ablations; a substantial new dataset; honest reporting of a null/negative collaboration result and modest absolute end-to-end success (21.3%).
Limitations (several acknowledged): retrospective, predominantly English, case-report-heavy data limiting real-world generalizability; heavy reliance on LLM-as-judge for both labels and evaluation, risking circularity (base models generating candidates are similar to judge models); no statistical testing; text-only (cannot interpret images directly); dependence on drifting proprietary APIs, complicating reproducibility and long-term stability; the full dataset cannot be released (only 500 cases). Absolute performance, while state-of-the-art, remains modest end-to-end, underscoring that the problem is far from solved.
Additional Observations
The dataset (RareBench) is a meaningful standalone contribution — screening-oriented with negatives, longitudinal, category-comprehensive — but its restricted release limits its foundational value. The alignment mechanism, while effective, is fundamentally supervised stacking/learning-to-rank over model outputs; the *conceptual* framing is more novel than the *technical* machinery, which is well-established ML. Reproducibility is partial: code and pipelines are on GitHub, but full data and API-dependent base models cannot be fully reproduced.
Overall, this is a strong, ambitious, well-engineered paper with a genuinely transferable idea and a large evaluation, held back from top-tier certainty by data-provenance concerns, judge-model circularity, and absence of statistical inference.
Generated Jul 28, 2026
Comparison History (18)
Paper 1 addresses a critical real-world healthcare problem (rare disease diagnosis) affecting millions, with rigorous validation on 157,525 cases and clinical trials involving 23 physicians. Its novel approach of exploiting rather than eliminating model variability is innovative and generalizable. The demonstrated clinical improvements have direct, measurable patient impact. While Paper 2 offers valuable systems-level infrastructure for agent memory with strong formal verification, its impact is more niche and technical, targeting emerging agent architectures. Paper 1's combination of methodological rigor, breadth, immediate applicability, and timeliness gives it substantially higher potential scientific and societal impact.
While Paper 1 presents a strong advance in automated AI planning, Paper 2 demonstrates profound real-world potential by addressing a critical healthcare bottleneck: rare disease care. Paper 2's methodological rigor is exceptional, featuring a massive dataset of 157,525 cases, human-in-the-loop evaluation with 23 physicians, and benchmarking against next-generation models. Its novel approach to exploiting multi-LLM divergence for high-uncertainty clinical decision-making spans multiple medical stages. This combination of vast societal application, interdisciplinary breadth, and rigorous clinical validation gives Paper 2 a significantly higher potential for widespread scientific and real-world impact.
Paper 2 addresses a fundamental, broadly applicable problem—hallucination detection in LLMs—with strong methodological rigor, including novel theoretical guarantees (random-length DKW inequality, exponential convergence) and practical single-pass efficiency. Its impact spans all high-stakes LLM deployments across domains. Paper 1 is impressive and clinically valuable but domain-specific to rare diseases, with results dependent on a particular dataset and benchmark. Paper 2's combination of theoretical novelty, generalizability across models/tasks, and immediate real-world deployability gives it broader and more foundational scientific impact.
Paper 1 introduces a foundational paradigm shift by formalizing a 'third axis' of test-time compute (interaction), moving beyond current reasoning and sampling limits. While Paper 2 offers profound societal value and rigorous clinical validation for rare diseases, Paper 1's fundamental algorithmic insights into AI scaling will likely have broader, cross-disciplinary impact. Because it addresses core bottlenecks in LLM capability and agentic workflows, Paper 1's framework will influence a wider array of downstream fields, driving fundamental advances in how future AI systems interact with external environments.
Paper 1 addresses a critical, high-stakes real-world problem (rare disease misdiagnosis affecting millions), demonstrates rigorous validation on a massive dataset (157,525 cases), and includes a clinical study with physicians showing measurable patient-care improvements. Its end-to-end scope across the entire care trajectory and strong empirical results against frontier models suggest broad, immediate clinical impact. Paper 2 offers an elegant defense mechanism for LLM safety with novel consensus decoding, but its impact is narrower, confined to a specific technical subproblem, and validated only on controlled tasks without demonstrated real-world deployment.
Paper 2 (RareLens) has higher likely scientific impact due to its direct, high-stakes real-world application in healthcare and breadth across the full rare-disease care pathway (screening, diagnosis, treatment, prognosis). It introduces a potentially generalizable paradigm—aligning divergent reasoning from heterogeneous LLMs—validated on a very large, real-world dataset plus external physician studies, suggesting strong methodological rigor and translational relevance. Paper 1 is novel and valuable for RL-based computer-use agents, but its impact is narrower and primarily confined to agent evaluation/training infrastructure.
Paper 2 demonstrates profoundly higher potential scientific impact. While Paper 1 presents a useful engineering tool for LLM coding workflows, Paper 2 tackles a critical global health challenge with high real-world stakes. Its methodological rigor is exceptional, featuring a massive dataset of 157,525 cases and external validation with 23 human physicians. Furthermore, Paper 2 introduces a highly novel paradigm of aligning divergent reasoning across multiple frontier models rather than scaling a single model, offering broad implications for both medical AI and general high-uncertainty decision-making.
Paper 2 demonstrates higher potential scientific and societal impact. While Paper 1 offers a valuable systems-level optimization for world models, Paper 2 tackles a high-stakes medical challenge affecting millions. Its novel approach of aligning divergent LLM reasoning across the entire care trajectory, combined with exceptionally rigorous validation on over 157,000 real-world cases and clinical trials with human physicians, ensures profound real-world applicability. This methodology not only advances medical AI but also provides a highly generalizable framework for high-uncertainty decision-making across other fields.
Paper 1 addresses a critical, high-stakes medical problem affecting millions, with a novel methodological insight (exploiting rather than eliminating LLM reasoning variability). It demonstrates rigorous, large-scale validation (157,525 cases, external study with 23 physicians) and outperforms frontier models across the full care trajectory. Its impact spans clinical practice and AI methodology. Paper 2 tackles a narrower software engineering task with a smaller study (7 projects), modest F1 scores, and incremental prompting comparisons. While useful, its breadth, rigor, and real-world stakes are substantially lower than Paper 1's.
Paper 2 has higher estimated impact: it targets a major, widespread health problem (rare diseases) with clear clinical utility across screening, diagnosis, treatment, and prognosis, and reports large-scale evaluation (157,525 real-world cases) plus external physician study, suggesting stronger methodological breadth and translational potential. Its core idea—aligning divergent reasoning across heterogeneous LLMs—may generalize to other high-uncertainty domains beyond medicine. Paper 1 is novel and rigorous for network operations, but its applications are narrower and the evaluation appears more preliminary.
Ambitious end-to-end system with a transferable conceptual thesis (learning from model divergence) and comprehensive evaluation, tempered by case-report-heavy data, LLM-as-judge circularity, and no statistical testing.