How Far Are We From True Auto-Research?

Paper 2 has higher impact potential due to a clear methodological contribution (new IC-SMDP formalism and an asynchronous decentralized neural Q-learning algorithm) paired with a novel finite-sample convergence bound under decentralized partial observability—likely broadly reusable across multi-agent RL, distributed systems, and LLM pipeline orchestration. It is timely for cross-org/vendor agent workflows and offers principled design guidance via decomposed error sources, supported by experiments that validate the theory. Paper 1 is timely and useful as an evaluation study of auto-research, but is more diagnostic/benchmarking-oriented with narrower methodological novelty and less generalizable theoretical output.

Paper 2 has higher impact potential: it introduces a controllable benchmark (ScaleLogic) enabling systematic, reproducible scaling studies of RL for long-horizon reasoning, and reports clear quantitative laws (power-law scaling with depth; expressiveness-dependent exponents) with strong fits and validation across RL methods plus curriculum effects. Its findings are timely for LLM post-training, with direct implications for compute planning and task design, and broad relevance across ML, reasoning, and alignment. Paper 1 is valuable as an audit/diagnostic of agentic auto-research, but is more observational and narrower in immediate methodological generalization.

Paper 1 is more likely to have higher impact: it proposes a novel self-improvement loop for long-horizon embodied learning using unlabeled internet video, demonstrating substantial performance gains over strong baselines on demanding multi-hour tasks. This advances core capability (scalable supervision and long-horizon planning) with clear applications to robotics/agents and broad relevance across RL, imitation learning, and foundation-model agents. Paper 2 is timely and valuable as an evaluation/audit study, but it is primarily diagnostic; it introduces a benchmark/scaffold and highlights failure modes rather than delivering a new capability breakthrough, limiting downstream transformative impact.

Paper 2 addresses a critical, life-altering flaw in current AI safety paradigms: 'safety' measures causing omission harm in medical crises. Its methodology is highly rigorous, utilizing pre-registered clinical scenarios and physician-validated scoring. While Paper 1 provides a valuable benchmark for AI research agents, Paper 2 has profound implications for global AI safety policy, medical AI deployment, and human health, giving it a significantly higher potential for broad, interdisciplinary real-world impact.

Paper 1 introduces a massive-scale foundation model for healthcare, demonstrating immediate and transformative real-world applications in disease prediction and health economics. Its population-scale validation offers profound, cross-disciplinary impact. Paper 2, while highly relevant to AI methodology, is primarily an evaluative benchmark highlighting current limitations in auto-research, making Paper 1's tangible contributions more impactful.

Paper 1 introduces a novel theoretical framework unifying Bayesian inference, game theory, and thermodynamics—three foundational fields—with mathematical proofs and falsifiable predictions validated across multiple domains. Its breadth of impact spans neuroscience, biology, physics, and AI, establishing deep structural connections. Paper 2, while timely and practically useful as a benchmark for AI-generated research, is more of an empirical evaluation study with findings likely to become outdated as AI systems rapidly improve. Paper 1's theoretical contributions have longer-lasting significance and broader cross-disciplinary reach.

Paper 1 offers deeper scientific insight by identifying fundamental epistemological failures in LLM-based scientific agents through 25,000+ runs across 8 domains, demonstrating that base models—not scaffolds—determine reasoning quality, and that evidence is ignored in 68% of traces. This mechanistic understanding of *why* AI scientists fail (lack of epistemic reasoning patterns) has broader implications across all scientific domains and provides actionable direction (reasoning as a training target). Paper 2, while valuable in benchmarking auto-research quality, is more descriptive and narrower in scope (CS papers only, 117 papers, 3 agents). Paper 1's findings are more fundamental and generalizable.

Paper 1 offers a novel, physically grounded unification of diffusion generative modeling and random structure search, delivering large efficiency gains and improved coverage of metastable minima, including out-of-distribution compositions. Its applications to molecular/crystal structure discovery are immediate and broadly valuable across chemistry, materials science, and computational physics, with clear methodological substance and measurable performance improvements. Paper 2 is timely and important for AI evaluation, but is primarily a benchmarking/diagnostic study with narrower direct scientific utility and less enduring cross-domain impact than a new structure-search paradigm for real-world materials discovery.

Paper 1 demonstrates a concrete, unprecedented achievement: an LLM-based agent autonomously discovering and experimentally validating a novel physical mechanism (optical bilinear interaction) on real hardware. This represents a paradigm shift in how scientific discoveries can be made, with direct implications for AI-driven science and optical computing. Paper 2, while valuable as a benchmarking study revealing limitations of auto-research systems, is primarily diagnostic and incremental. Paper 1's novelty—first end-to-end autonomous discovery with real experimental validation—has far broader impact across physics, AI, and hardware design, making it a landmark contribution.

Paper 2 presents a novel multimodal foundation model for biomolecules with immediate, high-impact applications in structural biology, genomics, and drug design. Its ability to perform constrained design and achieve state-of-the-art results across diverse biological modalities gives it immense potential for real-world scientific advancement. While Paper 1 is a timely and valuable benchmark exposing the limitations of current AI in automated research, Paper 2 introduces a foundational tool that actively drives new discoveries in a critical scientific domain.

Paper 1 offers a novel, technically substantive paradigm (multi-agent symbolic + metaheuristic equation discovery) with clear methodological claims (recovering governing equations across dynamics) and strong potential real-world impact across sciences (interpretable, extrapolatable models; large error reductions; parameter compression). Its breadth spans physics/engineering/biology and aligns with timely goals in scientific machine learning. Paper 2 is valuable as a meta-evaluation benchmark of agentic auto-research and exposes critical failure modes, but it is primarily diagnostic within AI/ML methodology and is less likely to yield direct cross-domain scientific advances than a general equation-discovery framework.

Paper 1 is more likely to have higher scientific impact: it introduces a novel generative “health world model” spanning 667 multimodal longitudinal measurements, demonstrates strong generalization to independent cohorts, and supports actionable clinical tasks (risk prediction and intervention-conditioned simulation) with quantitative validation against RCTs and established risk scores. Its real-world applications (digital twins, forecasting, trial simulation) are broad and timely for precision medicine. Paper 2 is valuable as an evaluation study of agentic auto-research and exposes important failure modes, but its primary impact is methodological/diagnostic within AI research workflows and is less immediately translational.

Paper 2 provides a more rigorous and critically needed empirical evaluation of auto-research systems. Its key contributions—identifying that manuscript-only review is misleading, quantifying failure modes (fabrication, underpowered experiments, plan/execution mismatch), and showing no agent-generated paper meets top-venue standards—offer foundational insights for the field. The 117-paper benchmark with multi-lens evaluation methodology is more likely to shape future research directions and evaluation standards. Paper 1, while technically interesting, presents an incremental system improvement, whereas Paper 2 challenges fundamental assumptions about auto-research progress, which has broader and more lasting impact.

Paper 2 has higher impact potential: it introduces a general, timely benchmark-style scaffold (ResearchArena) and a multi-lens evaluation methodology (manuscript-only vs artifact-aware vs human meta-review) that exposes systemic failure modes in auto-research. Its findings are broadly relevant across AI, HCI, ML evaluation, and scientific integrity, with immediate real-world implications for deploying research agents and for peer-review tooling. Paper 1 is novel within AV planning but is narrower in scope and reports limited metric gains, reducing likely cross-field uptake despite useful benchmarking.

Paper 2 addresses the highly timely and broadly impactful question of whether AI agents can autonomously conduct research, providing a large-scale systematic evaluation (117 papers, multiple agents, multiple review lenses). Its findings—that manuscript-only review overestimates quality and that critical failure modes like fabricated results persist—have immediate implications for the AI research community, funding agencies, and scientific integrity. While Paper 1 introduces a novel metacognitive evaluation framework with solid methodological rigor, its scope is narrower (resource allocation under token budgets). Paper 2's breadth of impact, timeliness given the auto-research hype cycle, and actionable taxonomy of failure modes give it higher potential scientific impact.

Paper 2 addresses the profound and highly relevant question of whether AI agents can conduct autonomous scientific research. By systematically evaluating AI-generated papers and exposing critical flaws in current evaluation methods and agent rigor, it has broad implications for AI safety, research integrity, and the future of science across all disciplines. While Paper 1 provides valuable systems-level insights for LLM serving, Paper 2's focus on the limits of AI capabilities gives it much wider potential scientific and societal impact.

Paper 2 has higher likely impact due to timeliness and breadth: it provides a systematic, multi-lens evaluation framework (manuscript-only vs artifact-aware vs human meta-review) over 117 agent-generated papers, revealing concrete failure modes (fabrication, underpowered studies, plan/execution mismatch) and large agent-dependent differences. This directly informs evaluation standards, benchmarking, and safety/quality controls for rapidly growing autonomous research systems across many fields. Paper 1 is innovative and rigorous within survey methodology and disaster preparedness, but its impact is narrower and more domain-specific.

Paper 2 addresses a critical, highly timely issue with broad implications across all scientific fields: the actual capabilities and limitations of AI-driven autonomous research. By systematically exposing the gap between manuscript quality and experimental rigor in AI-generated papers, it guides the future development of AI scientists. While Paper 1 presents an innovative approach to drug design with clear practical utility, Paper 2's fundamental critique of how AI conducts and reports research offers a broader and more transformative impact on the scientific method itself.

Paper 2 offers a rigorous, large-scale empirical evaluation of an highly relevant and hyped topic (auto-research AI). By introducing the ResearchArena benchmark and exposing critical failure modes like fabricated results and evaluation mismatches, it provides actionable, immediate value to the AI community. In contrast, Paper 1 is a conceptual vision paper lacking empirical validation, making Paper 2 significantly more scientifically impactful and methodologically rigorous.

Paper 2 has higher impact potential: it introduces a scalable evaluation scaffold (ResearchArena), produces a sizable empirical dataset (117 papers), and identifies concrete, generalizable failure modes in agentic research that are timely and broadly relevant across ML, HCI, research integrity, and benchmarking. Its artifact-aware evaluation improves methodological rigor compared to manuscript-only scoring and yields actionable insights for building reliable auto-research systems. Paper 1 is a valuable, reproducible case study for AI-assisted formalization, but its scope is narrower and largely diagnostic within theorem proving/Lean tooling.