Kurate.org
Rankings
Back to Rankings

Evaluation-driven Scaling for Scientific Discovery

Haotian Ye, Haowei Lin, Jingyi Tang, Yizhen Luo, Caiyin Yang, Chang Su, Rahul Thapa, Rui Yang

Apr 21, 2026arXiv:2604.19341v1_frozen_v1
cs.LGcs.AI
v1v2
Frozen v1 — this version was superseded on arXiv. Stats reflect the state at freeze time.View latest (v2) →
Share
Scorecard· 5/16
8.0/10 impact

Abstract

Language models are increasingly used in scientific discovery to generate hypotheses, propose candidate solutions, implement systems, and iteratively refine them. At the core of these trial-and-error loops lies evaluation: the process of obtaining feedback on candidate solutions via verifiers, simulators, or task-specific scoring functions. While prior work has highlighted the importance of evaluation, it has not explicitly formulated the problem of how evaluation-driven discovery loops can be scaled up in a principled and effective manner to push the boundaries of scientific discovery, a problem this paper seeks to address. We introduce Simple Test-time Evaluation-driven Scaling (SimpleTES), a general framework that strategically combines parallel exploration, feedback-driven refinement, and local selection, revealing substantial gains unlocked by scaling evaluation-driven discovery loops along the right dimensions. Across 21 scientific problems spanning six domains, SimpleTES discovers state-of-the-art solutions using gpt-oss models, consistently outperforming both frontier-model baselines and sophisticated optimization pipelines. Particularly, we sped up the widely used LASSO algorithm by over 2x, designed quantum circuit routing policies that reduce gate overhead by 24.5%, and discovered new Erdos minimum overlap constructions that surpass the best-known results. Beyond novel discoveries, SimpleTES produces trajectory-level histories that naturally supervise feedback-driven learning. When post-trained on successful trajectories, models not only improve efficiency on seen problems but also generalize to unseen problems, discovering solutions that base models fail to uncover. Together, our results establish effective evaluation-driven loop scaling as a central axis for advancing LLM-driven scientific discovery, and provide a simple yet practical framework for realizing these gains.

AI Impact Assessments

(3 models)

Scientific Impact Assessment: "Evaluation-driven Scaling for Scientific Discovery"

1. Core Contribution

This paper introduces SimpleTES, a framework that formalizes and scales evaluation-driven discovery loops for LLM-based scientific discovery. The key insight is that the evaluation budget N can be decomposed into three orthogonal dimensions: global width C (parallel independent trajectories), refinement depth L (iterative feedback-driven improvement), and local sample size K (best-of-K selection at each step). The paper argues—and demonstrates empirically—that principled allocation across these dimensions is as important as scaling model capability itself.

The framework is complemented by RPUCG, a graph-based context construction method for selecting historical nodes to include in prompts, and a trajectory-level post-training approach that trains models on successful discovery trajectories rather than individual step rewards.

The central problem addressed is important: while many systems use evaluation-driven loops (AlphaEvolve, OpenEvolve, TTT-Discover, etc.), none had explicitly formulated the scaling question—how to allocate evaluation queries optimally across exploration, refinement, and selection dimensions.

2. Methodological Rigor

Strengths in experimental design:

  • Coverage across 21 problems spanning 6 domains (quantum compilation, GPU kernels, algorithm engineering, math extremal analysis, combinatorial construction, data science) provides strong evidence of generality.
  • Controlled comparisons: the authors use open-source gpt-oss models throughout, while baselines often use frontier models or ensembles, making the demonstrated superiority more impressive.
  • Comprehensive ablations (Figures 15-17) systematically vary C, L, K and total budget N, revealing task-dependent optimal allocations—mathematical tasks favor larger C, while kernel optimization favors deeper L.
  • The post-training experiments (Section 4.2) include both in-domain and out-of-domain evaluation, demonstrating genuine generalization rather than task-specific memorization.
  • The reward hacking analysis (Section 4.3) is unusually thorough and honest, cataloging four distinct hacking strategies in kernel optimization and tolerance exploitation in math tasks.

    • Concerns:

  • The theoretical model (Appendix B, Pólya Urn formalization) is somewhat stylized and doesn't capture the full complexity of real LLM refinement dynamics. The gap between the model's assumptions (e.g., single-dimension refinement per step) and reality limits the theorem's prescriptive power.
  • The paper uses a single best-solution restart, acknowledging that the mechanism is "underexplored." This ad-hoc augmentation somewhat undermines the claim of a principled framework.
  • Some comparisons are complicated by different evaluation environments (e.g., Triton version differences in TriMul), though the authors acknowledge this.

    • 3. Potential Impact

    Immediate practical impact: The results are genuinely impressive across domains:

  • 2.17× speedup over glmnet for LASSO (a decades-old, heavily optimized solver)
  • 24.5% CNOT reduction in quantum circuit routing vs. LightSABRE
  • New SOTA on Erdős minimum overlap (0.380856), beating both human and prior AI records
  • New SOTA on sum-difference problem (1.144887), surpassing AlphaEvolve V2

    • Broader influence: The paper's greatest contribution may be conceptual—establishing that evaluation-driven loop scaling is a "central axis" for LLM-driven discovery, distinct from model scaling or reasoning-token scaling. This reframing could redirect research effort toward better evaluator design and budget allocation strategies.

    Adjacent fields: The framework naturally extends to any domain with programmatic evaluators: drug discovery (docking scores), materials science (DFT calculations), compiler optimization, etc.

    4. Timeliness & Relevance

    This paper arrives at a critical inflection point. Multiple groups (Google DeepMind's AlphaEvolve, Together AI, TTT-Discover) have demonstrated impressive results with evaluation-driven discovery, but each uses different, often complex, system designs. SimpleTES's contribution is to distill these into a clean, analyzable framework and show that much of the complexity may be unnecessary—the three-dimensional budget allocation (C, L, K) captures most of the scaling behavior.

    The demonstration that open-source models with principled evaluation scaling can match or exceed frontier models with sophisticated pipelines is particularly timely given concerns about compute inequality in AI research.

    5. Strengths & Limitations

    Key strengths:

  • Exceptional breadth of evaluation across genuinely diverse scientific domains
  • Clean conceptual framework that unifies prior work under a common lens
  • Honest treatment of reward hacking and surrogate-metric limitations
  • Strong empirical evidence that evaluation scaling follows predictable laws (Figure 17)
  • Post-training results showing OOD generalization validate that trajectory-level learning captures transferable discovery skills
  • Minimalist prompt strategy (Figure 25) that moves adaptation into search rather than prompt engineering

    • Notable limitations:

  • The framework assumes fast, queryable surrogate evaluators, excluding wet-lab sciences, subjective domains, and formal verification with sparse rewards
  • Budget allocation (C, L, K) is fixed a priori rather than adaptive—the paper acknowledges this but doesn't address it
  • The post-training uses simple IRFT rather than exploring more sophisticated credit assignment, which the authors acknowledge
  • Some results benefit from strong initial programs (e.g., LightSABRE for quantum routing, MAGIC for denoising), making it unclear how much gain comes from the starting point versus the search
  • The 110-page length, while thorough, may limit accessibility

    • Reproducibility: The paper provides detailed hyperparameters, evaluation protocols, and prompts (Appendix E), supporting reproducibility. However, the reliance on gpt-oss models (specific OpenAI models) may limit exact replication.

    Summary

    This is a high-impact systems paper that makes a compelling case for evaluation-driven scaling as a first-class research direction. Its breadth of empirical validation is exceptional, and the conceptual clarity of the (C, L, K) decomposition provides a useful lens for the field. The main limitations—fixed budget allocation, dependence on good evaluators, and relatively simple theoretical grounding—represent natural directions for future work rather than fundamental flaws.

    Rating:8.2/ 10
    Significance 8.5Rigor 7.8Novelty 7.5Clarity 7.5

    Generated Apr 22, 2026

    Comparison History (0)

    No comparisons yet.