Forecasting Scientific Progress with Artificial Intelligence
Sean Wu, Pan Lu, Yupeng Chen, Jonathan Bragg, Yutaro Yamada, Peter Clark, David Clifton, Philip Torr
Abstract
Artificial intelligence (AI) is increasingly embedded in scientific discovery, yet whether it can anticipate scientific progress remains unclear. To study this question, we introduce a temporally grounded evaluation framework for forecasting scientific progress under controlled knowledge constraints. We present CUSP (Cutoff-conditioned Unseen Scientific Progress), a multi-disciplinary and event-level benchmark that evaluates scientific forecasting in AI systems through feasibility assessment, mechanistic reasoning, generative solution design, and temporal prediction. Across 4,760 scientific events, we observe systematic and domain-dependent limitations in current frontier models. While models can identify plausible research directions from competing candidates, they fail to reliably predict whether scientific advances will be realized and systematically misestimate when they will occur. Performance is highly heterogeneous across domains, with the timing of AI progress more predictable than advances in biology, chemistry, and physics. Performance is largely insensitive to whether events occur before or after the training cutoff, suggesting these limitations cannot be explained solely by knowledge exposure in training data. Under controlled information access, additional pre-cutoff knowledge improves performance but does not close the gap to full-information settings, which becomes more pronounced for high-citation advances. Models also exhibit systematic overconfidence and strong response biases, indicating unreliable uncertainty estimation. Taken together, current AI systems fall short as predictive tools for scientific progress. Access to prior knowledge does not translate into reliable forecasting, and performance benefits more from post-event information than from forward-looking prediction.
AI Impact Assessments
(1 model)Scientific Impact Assessment: "Forecasting Scientific Progress with Artificial Intelligence"
1. Core Contribution
This paper introduces CUSP, a temporally grounded benchmark comprising 4,760 scientific milestones and 17,429 structured tasks designed to evaluate whether frontier LLMs can forecast scientific progress. The key novelty lies in the *operationalization* of scientific forecasting as a measurable capability, decomposed into four complementary dimensions: binary feasibility assessment, mechanistic reasoning (MCQ), generative solution design (FRQ), and temporal prediction (date estimation). The benchmark enforces strict temporal knowledge constraints tied to each model's training cutoff, enabling controlled disentanglement of knowledge access from genuine forecasting ability.
The central finding is negative but informative: current frontier AI systems systematically fail at scientific forecasting despite having access to substantial prior knowledge. They can identify plausible technical approaches (MCQ accuracy up to 0.819 for GPT-5.4) but cannot reliably predict *whether* advances will be realized (binary performance near chance at ~0.50) or *when* they will occur (systematic positive temporal bias of 4-36 months). The paper introduces the analytical decomposition of performance gaps into "knowledge gaps" (∆know) and "forecasting gaps" (∆fore), revealing that the forecasting gap dominates and grows with citation impact.
2. Methodological Rigor
Strengths in design: The two-track evaluation framework—combining deterministic outcome scoring with rubric-based reasoning evaluation—is well-motivated and addresses known issues with LLM evaluation (correct answers via flawed reasoning). The temporal stratification using earliest-observed DOI dates across multiple APIs (Crossref, Semantic Scholar, OpenAlex, etc.) is thorough. The web search augmentation experiments with controlled information access (WS+Cutoff vs. unrestricted WS) provide a clean experimental design for decomposing knowledge versus forecasting contributions.
Concerns: Several methodological choices warrant scrutiny. First, the binary task design has an inherent label imbalance (ground truth for originals is always "Yes," perturbed always "No"), and the "merged score" averaging these may mask important asymmetries—as Table 19 reveals, models exhibit extreme response biases (LLaMA 3.3: 93% "Yes" bias). Second, the FRQ evaluation relies on GPT-5.4-mini with web search as judge, which introduces its own biases; the human-AI judge correlation of r=0.34 (n=60), while statistically significant, is modest. Third, the date prediction metric (exponential decay with 0.1 scaling) is somewhat arbitrary—the choice of decay rate significantly affects relative model rankings. Fourth, the LLaMA 3.3 date prediction anomaly (best score despite worst MCQ) appears driven by temporal anchoring artifacts acknowledged but not fully resolved.
The validation pipeline using Grok-3 as an independent judge from GPT-4o (used for generation) is a reasonable precaution, and human validation agreement analysis (Appendix E.1) adds credibility. The statistical testing in web search experiments (with p-values) strengthens causal claims.
3. Potential Impact
Direct impact on AI evaluation: CUSP addresses a genuine gap in the benchmark landscape. Unlike retrospective scientific reasoning benchmarks (GPQA, MMLU-Pro) that are approaching saturation, CUSP remains substantially unsaturated (Figure 11), providing continued discriminative power. The continuously updatable design and Time Capsule component enable living evaluation.
Implications for AI-for-science: The finding that forecasting gaps grow with citation impact (Table 29) has practical consequences for using AI in research prioritization, funding allocation, and technology roadmapping. The domain-dependent predictability results (AI timing more predictable than biology/chemistry/physics) provide concrete guidance about where AI forecasting tools may be more or less reliable.
Broader implications: The systematic overconfidence findings (Table 20) carry weight for any deployment of LLMs as scientific advisors. The disconnect between knowledge access and forecasting ability challenges the assumption that larger, more knowledgeable models will naturally become better forecasters.
Limitations in impact: The benchmark, while multi-disciplinary, is heavily biased toward biology (1,234 papers) and AI (1,141), with chemistry (203) and environmental science (235) underrepresented. The restriction to Nature/Science/Cell for natural sciences introduces a prestige bias that may not represent the full spectrum of scientific progress.
4. Timeliness & Relevance
This work is highly timely. The rapid deployment of AI in scientific discovery (AlphaFold, AI Co-Scientist, Virtual Lab) has created urgent need to understand AI's predictive limitations. The paper directly addresses the emerging question of whether AI can move from scientific *assistance* to scientific *anticipation*—a distinction with significant implications for autonomous research agents. The benchmark spans January 2024–March 2026, ensuring evaluation against very recent frontier models including GPT-5.4.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
Overall assessment: This is a substantial contribution that introduces a well-designed benchmark addressing a genuinely novel evaluation dimension. The negative findings are robustly established and practically important. While methodological refinements are needed (particularly for FRQ evaluation and binary task design), the framework establishes a strong foundation for studying AI's capacity for scientific foresight.
Generated May 22, 2026
Comparison History (32)
Paper 1 addresses a highly critical and timely bottleneck in AI—test-time scaling for LLM reasoning. By proposing a concrete algorithmic improvement (stochastic backtracking) that consistently enhances the accuracy-token trade-off, it offers immediate, practical utility for deploying advanced reasoning models. While Paper 2 provides a valuable meta-analysis of AI's forecasting abilities, Paper 1's methodology directly advances the foundational capabilities of AI systems, promising broader and more immediate real-world downstream impact across all fields utilizing AI reasoning.
Paper 2 likely has higher impact due to a concrete, reusable evaluation framework (CUSP) and a large, multidisciplinary benchmark enabling standardized measurement and follow-on work across scientometrics, AI evaluation, and science policy. Its findings on forecasting limits, domain heterogeneity, cutoff insensitivity, and miscalibrated uncertainty are immediately actionable for model development and decision-making. Paper 1 is conceptually novel and relevant to AI governance, but its impact depends heavily on the generality of its formal assumptions and may remain more theoretical with fewer direct empirical artifacts for the community to adopt.
Paper 1 introduces CUSP, a novel benchmark addressing a fundamental question about AI's ability to forecast scientific progress—a topic with broad implications across all scientific disciplines. Its systematic evaluation of frontier models reveals important limitations (overconfidence, domain heterogeneity, failure modes) that inform the entire AI-for-science community. Paper 2, while technically sound, addresses a narrower problem (credit assignment in search-augmented reasoning) with an incremental methodological contribution. Paper 1's breadth of impact, timeliness given the AI-for-science movement, and foundational insights about AI capabilities give it higher potential impact.
Paper 1 introduces CUSP, a novel benchmark addressing a fundamental question about AI's ability to forecast scientific progress—a topic with broad implications across all scientific disciplines. Its comprehensive evaluation of frontier models reveals systematic limitations in scientific forecasting, contributing important insights to AI capabilities research, science of science, and epistemic calibration. Paper 2, while technically sound, addresses a more incremental improvement in search-augmented reasoning via self-distillation, with narrower scope. Paper 1's breadth of impact, timeliness given rapid AI integration in science, and its interdisciplinary relevance give it higher potential impact.
Paper 1 offers higher potential scientific impact due to its immediate, practical solutions to critical bottlenecks in LLM alignment and efficiency. By introducing 'Behavior Cues,' it provides a novel mechanism for monitorable reasoning, yielding significant efficiency gains (50% reduction in wasted tokens) and major safety improvements (recovering safe actions from 80% of unsafe traces). While Paper 2 introduces a valuable metascience benchmark, its primarily negative findings about current AI limitations offer less immediate utility compared to Paper 1's actionable framework for scalable oversight and safe AI deployment.
Paper 2 introduces a novel, large-scale benchmark (CUSP) for evaluating AI's ability to forecast scientific progress—a fundamental and broadly relevant question. Its multi-disciplinary scope (4,760 events across many fields), systematic evaluation of frontier models, and insights about AI's predictive limitations have wide implications for science policy, AI development, and the philosophy of discovery. Paper 1 addresses an important but narrower technical problem (LLM reasoning oversight via behavior cues) with solid but domain-specific contributions. Paper 2's breadth, timeliness, and relevance to the rapidly growing AI-for-science community give it higher potential impact.
Paper 1 presents a highly actionable, novel methodology (GSS) that directly accelerates materials and molecular discovery by tenfold. This has profound real-world applications in drug discovery, materials science, and clean energy. In contrast, Paper 2 is a benchmarking study reporting primarily negative results about AI's current inability to forecast scientific progress. While Paper 2 is relevant to meta-science and AI evaluation, Paper 1 offers a concrete, computationally rigorous tool that immediately solves a major bottleneck in high-dimensional energy landscape exploration, providing more immediate and tangible scientific impact.
Paper 1 presents a novel theoretical framework unifying three major fields—Bayesian inference, game theory, and thermodynamics—through a collective variational principle. It offers formal proofs connecting free-energy minimisation to Nash equilibria, introduces new concepts (free-energy Harsanyi dividend), and makes falsifiable predictions validated across multiple systems. This kind of deep unifying theory has potential for broad, lasting impact across physics, neuroscience, biology, and AI. Paper 2, while timely and methodologically sound, is primarily a benchmark/evaluation study revealing AI limitations in forecasting, with more incremental impact on the AI evaluation community.
MIMIC presents a novel generative multimodal foundation model that unifies multiple biological data modalities (sequence, structure, regulation, evolution) into a single framework, achieving state-of-the-art results across multiple tasks and enabling constrained biomolecular design. Its direct applications to RNA editing and protein design (e.g., PD-L1 binding) demonstrate significant translational potential. Paper 2, while introducing a valuable benchmark (CUSP) for evaluating AI forecasting of scientific progress, primarily characterizes limitations of current models rather than providing a transformative tool. MIMIC's methodological innovation, breadth of biological applications, and practical design capabilities give it substantially higher potential for driving downstream research and real-world impact.
Paper 1 likely has higher scientific impact due to broader cross-disciplinary scope and foundational contribution: a temporally grounded, controlled-knowledge benchmark (CUSP) for forecasting scientific progress across thousands of events. It addresses a general, timely question about AI’s role in science and provides reusable evaluation infrastructure with implications for scientometrics, AI evaluation, and research policy. Paper 2 is novel and important for AI safety/clinical deployment, but its domain is narrower (clinical safety framing effects) and may have more constrained breadth despite strong pre-registration and methodology.