This assessment is based on version 1 of this paper. Version 3 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
Self Driving Datasets: From 20 Million Papers to Nuanced Biomedical Knowledge at Scale
Haydn Jones, Yimeng Zeng, Alden Rose, Li S. Yifei, Yining Huang, Kaiwen Wu, Jiaming Liang, Maggie Ziyu Huan
Abstract
Manually curated biomedical repositories -- spanning bioactivity, genomics, and chemistry -- are expensive to maintain, lag behind primary literature, and discard experimental context, obscuring nuances needed to assess data correctness and coverage. We show that PubMed itself can be autonomously and cost-effectively turned into structured datasets that are larger, more nuanced, and more accurate than the curated databases they replace. We present three coupled contributions: (1) an LLM-based entity-tagging pipeline, grounded in nine biomedical ontologies, that tags 4.5B entities across 19 categories in a 22.5M-paper, 2.5T-token PubMed corpus; (2) hybrid sparse-dense retrieval supporting entity-filtered semantic queries over the tagged corpus; and (3) Starling, a multi-agent deep research system that, given only a natural-language task description, designs precision- and recall-targeted retrieval filters, induces an extraction schema, and emits structured records with nuance-rich fields and supporting passages. Across six tasks -- blood-brain barrier permeability, oral bioavailability, acute toxicity (LD50), gene-disease associations, protein subcellular localization, and chemical reactions -- Starling produces ~6.3M records (91K-3M per task); several are, to our knowledge, the largest public datasets for their property. Frontier-model rejection of our extractions is 0.6-7.7% across tasks, far below error rates we measure on widely used curated counterparts (e.g., 16.5% on BBB_Martins, 7.3% on Bioavailability_Ma). Beyond scale and accuracy, the supporting passages carry nuance tabular databases discard -- e.g., oral bioavailability may depend on fed vs. fasted state. Together, the corpus, retrieval, and agent establish a foundation for AI-driven therapeutic design. Code and datasets: https://github.com/starling-labs/starling.
AI Impact Assessments
(1 model)Scientific Impact Assessment: "Self Driving Datasets: From 20 Million Papers to Nuanced Biomedical Knowledge at Scale"
1. Core Contribution
This paper introduces Starling, an end-to-end system for autonomously converting the PubMed literature (22.5M papers, 2.5T tokens) into structured, nuanced biomedical datasets. The system comprises three integrated components: (1) an LLM-based entity tagging pipeline covering 19 biomedical entity types grounded in nine ontologies, producing 4.5B entity tags; (2) a hybrid sparse-dense retrieval system enabling entity-filtered semantic queries; and (3) a multi-agent "deep research" system that autonomously designs extraction filters, induces schemas, and emits structured records with supporting passages.
The central claim is provocative and well-supported: automated extraction from primary literature can produce datasets that are simultaneously larger, more accurate, and more contextually rich than manually curated gold-standard databases. Across six tasks (BBB permeability, oral bioavailability, LD50, gene-disease associations, protein subcellular localization, chemical reactions), Starling produces ~6.3M records, with several representing the largest public datasets for their respective properties.
2. Methodological Rigor
The methodology is carefully designed with multiple quality-control layers. The teacher-student distillation for entity tagging (gpt-oss-120b → gpt-oss-20b) with ontology-grounded normalization is well-motivated and executed. The probe-based retrieval strategy—where agents autonomously design precision- and recall-targeted filters—is an elegant solution to the corpus-scale extraction problem that distinguishes this from brute-force approaches.
The evaluation is thorough and multi-faceted:
A notable methodological concern is the reliance on frontier LLM judges (GPT-5.4, Claude Opus 4.7) for quality assessment, creating a potential circularity since extraction also uses LLMs. However, the human calibration study partially mitigates this. The comparison with baseline deep research systems (Table 4) is somewhat generous to Starling since baselines receive no per-row judging while Starling counts are post-filter, though the authors acknowledge this.
3. Potential Impact
The potential impact is substantial across multiple dimensions:
For drug discovery and therapeutic design: The nuance captured by Starling—food state effects on bioavailability, dosing route effects on LD50, transport mechanism effects on BBB permeability—represents genuinely new training signal. The ANOVA analysis (Fig. 2) quantifying that dosing route explains 56% of within-molecule LD50 variance is a powerful argument that structure-only models hit a fundamental ceiling. Table 5's demonstration that TDC-trained models fail on active influx BBB molecules but Starling-augmented models succeed is directly actionable.
For dataset curation paradigm: If the quality claims hold, this paper challenges the dominant paradigm of manual expert curation, suggesting that autonomous extraction with quality filtering can be faster, cheaper ($0.001/record), and more accurate. This could democratize dataset creation for smaller labs.
For chemical reactions: The discovery of ~401K novel product SMILES not in any reference database is notable, though the 0.51× scale relative to ORD and patent-domain differences temper this.
4. Timeliness & Relevance
This work arrives at an inflection point where LLM capabilities have made corpus-scale extraction feasible but existing biomedical AI still relies on static, context-stripped datasets. The paper directly addresses the increasingly recognized problem that benchmark datasets contain non-trivial error rates and that context-free labels limit model ceiling performance. The timing is excellent given the rapid scaling of AI-driven drug discovery.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Reproducibility: While code and datasets are released, the corpus licensing barrier is significant. The MCP server access plan is commendable but introduces dependency on the authors' infrastructure.
Summary
This is a high-impact systems paper that makes a convincing case for autonomous, literature-scale biomedical dataset construction. Its strongest contributions are the demonstrated accuracy advantage over manual curation and the quantified importance of contextual nuance. The work is likely to influence how the field approaches biomedical dataset creation and could accelerate context-aware therapeutic AI models.
Generated May 11, 2026
Comparison History (109)
Paper 1 offers a paradigm-shifting approach to automating biomedical dataset curation, directly addressing a major bottleneck in scientific research. By successfully converting 22.5 million papers into highly accurate, scalable datasets for therapeutic design, it demonstrates massive, immediate real-world utility. Its interdisciplinary reach—spanning AI, biology, and medicine—and potential to replace expensive manual curation give it broader, more tangible scientific impact than Paper 2, which, while methodologically rigorous, provides theoretical advancements limited primarily to the machine learning community.
Paper 2 presents a transformative, large-scale application of LLM agents to resolve a critical bottleneck in biomedical research: manual dataset curation. By autonomously extracting structured, nuanced data from 22.5 million papers, it outperforms widely used manual databases in both scale and accuracy. This has immense breadth of impact across pharmacology, genomics, and clinical research. While Paper 1 is highly innovative for materials design, Paper 2's cross-disciplinary utility, massive open-source dataset release, and foundational role for AI-driven therapeutic design give it significantly higher potential scientific impact.
Paper 1 likely has higher scientific impact: it proposes an end-to-end, scalable system that converts the entire PubMed corpus into ontology-grounded, queryable, nuance-preserving structured datasets, demonstrated across six high-value biomedical tasks with millions of records and quantitative error comparisons against curated baselines. The real-world applications (drug discovery, toxicology, genomics) are immediate and broad, and releasing code/data amplifies adoption. Paper 2 is novel and timely for AI governance, but its impact may be narrower and hinges on stronger empirical validation and practical deployment pathways in legal/forensic settings.
Paper 1 demonstrates a paradigm-shifting application of LLMs to solve a major bottleneck in biomedical research: manual data curation. By autonomously converting 22.5M papers into highly accurate, nuanced datasets, it broadly impacts bioinformatics, drug discovery, and genomics. In contrast, Paper 2 presents a valuable but narrower technical optimization (W4A4 quantization for Diffusion Transformers). While Paper 2 improves generative AI efficiency, Paper 1 introduces a highly scalable foundation for AI-driven science, offering far greater cross-disciplinary utility, real-world therapeutic application, and lasting methodological innovation. Thus, Paper 1 has significantly higher potential scientific impact.
Paper 1 addresses a foundational bottleneck in biomedical research—transforming 22.5M papers into structured, nuance-rich datasets that surpass curated databases in scale and accuracy. Its broad impact spans genomics, chemistry, and therapeutic design, with demonstrated superiority over widely-used benchmarks and released public artifacts. Paper 2 is a rigorous, elegant engineering optimization for speculative decoding at million-token context, but its impact is narrower and incremental within LLM inference efficiency. Paper 1's cross-field applicability, novelty in autonomous curation, and enabling role for AI-driven drug discovery give it substantially higher potential impact.
While Paper 1 presents a highly successful deployed method for LLM alignment, Paper 2 offers a broader transformational impact across multiple scientific disciplines. By autonomously converting 20 million PubMed papers into structured, nuanced datasets with lower error rates than manual curation, it solves a major bottleneck in biomedical research. Its contribution of the largest public datasets for key pharmacological properties and an open-source multi-agent extraction framework provides foundational infrastructure that will significantly accelerate AI-driven drug discovery and cross-disciplinary scientific research.
Paper 1 addresses a major real-world bottleneck—biomedical knowledge curation—and demonstrates concrete, large-scale outputs (6.3M records, several the largest public datasets) with measured accuracy exceeding gold-standard curated databases. Its impact spans drug discovery, genomics, and chemistry, with released code and data enabling broad adoption. Paper 2 offers an elegant, rigorous contribution to RLVR optimization, but its impact is narrower and confined to the ML methodology community. Paper 1's breadth, timeliness, and direct enablement of therapeutic design give it higher potential scientific impact.
Paper 1 offers a transformative paradigm shift, replacing expensive manual curation with autonomous dataset generation across 22.5M biomedical papers. By creating massive, highly accurate datasets for multiple properties, its impact spans genomics, pharmacology, and chemistry, resolving a critical data bottleneck for downstream AI models. While Paper 2 presents a highly innovative algorithmic approach for de novo drug design, its scope is confined to a specific subfield. Paper 1 provides foundational data infrastructure that will accelerate countless future studies across all of biomedicine, giving it broader and more profound scientific impact.
Paper 1 offers a paradigm shift in biomedical knowledge curation, utilizing LLMs to autonomously extract massive, highly accurate datasets from 22.5 million papers. This directly accelerates real-world applications in drug discovery and therapeutic design while bridging AI and biomedicine. In contrast, Paper 2 provides important but narrower insights into the optimization dynamics of low-rank LLM training. Paper 1's creation of foundational public datasets, broader cross-disciplinary impact, and demonstration of a highly scalable multi-agent research system make its potential scientific footprint significantly larger.
Paper 2 likely has higher scientific impact due to broader cross-field reach and downstream utility: it proposes an end-to-end, scalable pipeline that converts the entire PubMed corpus into ontology-grounded, queryable structured datasets and demonstrates millions of records across multiple high-value biomedical tasks. The resulting resources can directly accelerate drug discovery, knowledgebase construction, and biomedical ML, with immediate real-world applications and strong timeliness. Paper 1 is methodologically solid and impactful for edge LLM deployment, but its scope is narrower (CPU quantization/compiler co-design) and primarily benefits systems/ML inference communities.