How Far Are We From True Auto-Research?

Paper 2 tackles the highly relevant and hyped domain of autonomous AI research, providing a crucial reality check. By introducing a rigorous evaluation framework that contrasts manuscript-only reviews with artifact-aware peer reviews, it exposes critical experimental flaws and fabricated results in agent-generated papers. Its methodological rigor and broad implications for how the scientific community evaluates AI scientists give it a higher potential for widespread, paradigm-shifting impact across multiple fields compared to the narrower, albeit important, security focus of Paper 1.

Paper 2 likely has higher impact: it proposes a novel, optimizer-like, controllable method for training agent “skills” with validation-gated edits, addressing reproducibility and stability in text-space optimization. It demonstrates broad, rigorous empirical results (6 benchmarks, 7 models, 3 harnesses, 52 cells) with strong gains and transfer, plus released code—supporting real-world adoption in agent systems. Paper 1 is timely and valuable as an evaluation/diagnostic study of auto-research quality, but it is more observational and its contributions are narrower than a general, deployable optimization framework.

Paper 2 has higher impact potential due to a clear methodological contribution (new IC-SMDP formalism and an asynchronous decentralized neural Q-learning algorithm) paired with a novel finite-sample convergence bound under decentralized partial observability—likely broadly reusable across multi-agent RL, distributed systems, and LLM pipeline orchestration. It is timely for cross-org/vendor agent workflows and offers principled design guidance via decomposed error sources, supported by experiments that validate the theory. Paper 1 is timely and useful as an evaluation study of auto-research, but is more diagnostic/benchmarking-oriented with narrower methodological novelty and less generalizable theoretical output.

Paper 2 has higher impact potential: it introduces a controllable benchmark (ScaleLogic) enabling systematic, reproducible scaling studies of RL for long-horizon reasoning, and reports clear quantitative laws (power-law scaling with depth; expressiveness-dependent exponents) with strong fits and validation across RL methods plus curriculum effects. Its findings are timely for LLM post-training, with direct implications for compute planning and task design, and broad relevance across ML, reasoning, and alignment. Paper 1 is valuable as an audit/diagnostic of agentic auto-research, but is more observational and narrower in immediate methodological generalization.

Paper 1 is more likely to have higher impact: it proposes a novel self-improvement loop for long-horizon embodied learning using unlabeled internet video, demonstrating substantial performance gains over strong baselines on demanding multi-hour tasks. This advances core capability (scalable supervision and long-horizon planning) with clear applications to robotics/agents and broad relevance across RL, imitation learning, and foundation-model agents. Paper 2 is timely and valuable as an evaluation/audit study, but it is primarily diagnostic; it introduces a benchmark/scaffold and highlights failure modes rather than delivering a new capability breakthrough, limiting downstream transformative impact.

Paper 2 addresses a critical, life-altering flaw in current AI safety paradigms: 'safety' measures causing omission harm in medical crises. Its methodology is highly rigorous, utilizing pre-registered clinical scenarios and physician-validated scoring. While Paper 1 provides a valuable benchmark for AI research agents, Paper 2 has profound implications for global AI safety policy, medical AI deployment, and human health, giving it a significantly higher potential for broad, interdisciplinary real-world impact.

Paper 1 introduces a massive-scale foundation model for healthcare, demonstrating immediate and transformative real-world applications in disease prediction and health economics. Its population-scale validation offers profound, cross-disciplinary impact. Paper 2, while highly relevant to AI methodology, is primarily an evaluative benchmark highlighting current limitations in auto-research, making Paper 1's tangible contributions more impactful.

Paper 1 introduces a novel theoretical framework unifying Bayesian inference, game theory, and thermodynamics—three foundational fields—with mathematical proofs and falsifiable predictions validated across multiple domains. Its breadth of impact spans neuroscience, biology, physics, and AI, establishing deep structural connections. Paper 2, while timely and practically useful as a benchmark for AI-generated research, is more of an empirical evaluation study with findings likely to become outdated as AI systems rapidly improve. Paper 1's theoretical contributions have longer-lasting significance and broader cross-disciplinary reach.

Paper 1 offers deeper scientific insight by identifying fundamental epistemological failures in LLM-based scientific agents through 25,000+ runs across 8 domains, demonstrating that base models—not scaffolds—determine reasoning quality, and that evidence is ignored in 68% of traces. This mechanistic understanding of *why* AI scientists fail (lack of epistemic reasoning patterns) has broader implications across all scientific domains and provides actionable direction (reasoning as a training target). Paper 2, while valuable in benchmarking auto-research quality, is more descriptive and narrower in scope (CS papers only, 117 papers, 3 agents). Paper 1's findings are more fundamental and generalizable.

Paper 1 offers a novel, physically grounded unification of diffusion generative modeling and random structure search, delivering large efficiency gains and improved coverage of metastable minima, including out-of-distribution compositions. Its applications to molecular/crystal structure discovery are immediate and broadly valuable across chemistry, materials science, and computational physics, with clear methodological substance and measurable performance improvements. Paper 2 is timely and important for AI evaluation, but is primarily a benchmarking/diagnostic study with narrower direct scientific utility and less enduring cross-domain impact than a new structure-search paradigm for real-world materials discovery.