Unlocking LLM Creativity in Science through Analogical Reasoning
Andrew Shen, Shaul Druckmann, James Zou
Abstract
Autonomous science promises to augment scientific discovery, particularly in complex fields like biomedicine. However, this requires AI systems that can consistently generate novel and diverse solutions to open-ended problems. We evaluate LLMs on the task of open-ended solution generation and quantify their tendency to mode collapse into low-diversity generations. To mitigate this mode collapse, we introduce analogical reasoning (AR) as a new approach to solution generation. AR generates analogies to cross-domain problems based on shared relational structure, then uses those analogies to search for novel solutions. Compared to baselines, AR discovers significantly more diverse generations (improving solution diversity metrics by 90-173%), generates novel solutions over 50% of the time (compared to as little as 1.6% for baselines), and produces high-quality analogies. To validate the real-world feasibility of AR, we implement AR-generated solutions across four biomedical problems, yielding consistent quantitative gains. AR-generated approaches achieve a nearly 13-fold improvement on distributional metrics for perturbation effect prediction, outperform all baselines on AUPRC when predicting cell-cell communication, infer brain region interactions with a high Spearman correlation (=0.729) to published methods, and establish state-of-the-art performance on 2 datasets for oligonucleotide property prediction. The novel and diverse solutions produced by AR can be used to augment the search space of existing solution generation methods.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper identifies and quantifies a significant problem — mode collapse in LLM-generated solutions to open-ended scientific problems — and proposes analogical reasoning (AR) as a structured mitigation strategy. AR operates in two steps: (1) extracting relational structure from a problem and generating cross-domain analogies, and (2) using those analogies to search for solutions in distant domains. The key insight is that by forcing LLMs through the bottleneck of explicit structural analogy (object mappings and shared relations, grounded in Gentner's structure-mapping theory from cognitive science), the search space expands dramatically beyond the canonical solutions that LLMs typically converge toward.
The contribution is twofold: a diagnostic finding (LLMs exhibit severe mode collapse even when explicitly prompted for cross-domain solutions) and a prescriptive framework (AR as a "diversity engine"). The paper validates AR through both intrinsic evaluation (diversity, novelty, analogy quality metrics) and extrinsic validation (four biomedical case studies with quantitative gains).
Methodological Rigor
Strengths in evaluation design: The paper uses three complementary evaluation axes (diversity, novelty, analogy quality), each with multiple metrics. The Vendi Score is a principled diversity metric. The novelty evaluation pipeline — retrieval from Semantic Scholar, SPECTER-based re-ranking, then LLM-judged scoring — is well-designed and validated against human annotations from an external dataset (Si et al.'s AI Researcher dataset). The human preference study (78% preference for AR solutions, κ=0.445) and analogy quality validation (up to 88.6% human-LLM agreement) add credibility.
Concerns: Several methodological issues warrant attention:
1. Selection bias in case studies: The four biomedical case studies are cherry-picked demonstrations, not systematic evaluations. The paper generates one promising solution per problem — but doesn't report how many AR-generated solutions were tried and failed before finding a workable one. The success rate of AR solutions in practice remains unclear.
2. Fair comparison issues: AR uses Perplexity Deep Research for case study solution generation (replacing Claude/GPT/Gemini used in evaluations), introducing a confound. The baselines in case studies are existing published methods, not solutions generated by baseline LLM settings with equal implementation effort.
3. LLM-as-judge circularity: Novelty and analogy quality are primarily assessed by Claude Sonnet 4.5 — the same model family used for generation. While human validation is provided, the correlation values are moderate (Spearman ρ=0.36-0.43 for novelty), and the Structural Depth metric showed poor human-LLM agreement (accuracy=0.600, κ=-0.133).
4. Dataset construction: The AR Dataset of 266 papers was curated using an LLM (Perplexity) to find papers employing analogical reasoning, introducing potential systematic biases in problem selection. The evaluation set of 50 problems may favor AR by construction, as these are problems known to have cross-domain solutions.
5. Temperature and sampling: All methods use temperature=1.0, but AR's two-step process (extraction + search) may benefit disproportionately from the compounding stochasticity of two independent sampling steps versus one or two for baselines.
Potential Impact
The paper addresses a genuine need in autonomous science: LLM-driven systems need diverse, novel hypotheses to avoid the "streetlight effect" of only exploring familiar solution spaces. The framing of AR as a "diversity engine" that augments existing approaches is compelling and practically useful.
Near-term impact: AR could be integrated into frameworks like AI Scientist, AI Co-scientist, or Kosmos as a complementary generation module. The cost ($0.02/problem) makes it scalable. The AR Dataset of 266 cross-domain biomedical breakthroughs is itself a useful resource.
Broader impact: The paper formalizes a well-known cognitive strategy (analogical thinking) into a reproducible LLM pipeline. This could influence how researchers think about prompting strategies for creative tasks beyond science — in engineering design, policy formulation, or business strategy.
Case study results: Some results are genuinely impressive — the 13x improvement on MMD PCA for perturbation prediction, SOTA on AUPRC for cell-cell communication, and competitive PCMCI results for brain coupling. The chess-to-oligonucleotide analogy (piece-square tables) is particularly creative. However, these are proof-of-concept demonstrations; the solutions discovered are relatively straightforward implementations (GMMs, SNR analysis, PCMCI, linear features) that could plausibly have been found through systematic literature review.
Timeliness & Relevance
This paper is highly timely. The autonomous science field is accelerating, with multiple high-profile systems (AI Scientist v2, Kosmos, AI Co-scientist) published in 2024-2025. The mode collapse problem is widely recognized but rarely quantified for scientific ideation specifically. The paper fills an important gap by providing both the diagnosis and a structured remedy.
Key Strengths
Key Limitations
Overall Assessment
This is a well-executed paper that makes a genuine contribution to the autonomous science pipeline. The core idea — using structured analogical reasoning to overcome LLM mode collapse — is intuitive, well-motivated, and convincingly demonstrated to improve diversity and novelty. The case studies, while limited in number and subject to selection, demonstrate real feasibility. The main weakness is the gap between the impressive diversity metrics and the practical question of how often AR-generated solutions actually work when implemented. The paper would benefit from a systematic evaluation of solution viability rates.
Generated May 13, 2026
Comparison History (23)
Paper 1 introduces a fundamentally new paradigm (analogical reasoning) for LLM-driven scientific discovery with validated real-world biomedical applications achieving state-of-the-art results across multiple domains. Its breadth of impact spans AI methodology and multiple scientific fields. Paper 2, while offering elegant mechanistic insights about reasoning token sparsity and a practical inference-time intervention, addresses a more narrowly scoped technical problem within LLM reasoning. Paper 1's novelty in connecting cognitive science concepts to AI-driven science and its demonstrated cross-domain applicability give it higher potential for broad scientific impact.
Paper 2 likely has higher scientific impact due to stronger novelty-to-application linkage and broader cross-domain relevance: it proposes a concrete analogical reasoning framework to reduce mode collapse, reports large diversity/novelty gains, and validates with multiple implemented biomedical case studies showing substantial quantitative improvements and SOTA results. This combination of a general method for creative generation plus real-world scientific task performance suggests wide adoption potential across autonomous science and ML-for-science. Paper 1 is timely and important for safety in agentic systems, but its impact is more specialized and primarily diagnostic/mitigation-focused rather than enabling new capabilities across fields.
Paper 2 is likely higher impact: it introduces a broadly applicable method (analogical reasoning) targeting a central limitation of LLMs in scientific discovery (mode collapse/low diversity) and demonstrates real-world gains across multiple biomedical tasks with strong quantitative improvements, suggesting immediate utility and cross-domain relevance. Its methodological contribution is general and timely for autonomous science. Paper 1 is innovative and industrially relevant for CAD-CAE orchestration, but its impact is more domain-specific and depends on integration into particular engineering toolchains despite a valuable dataset and closed-loop RL framing.
Paper 1 presents a highly innovative approach to unlocking autonomous scientific discovery using Analogical Reasoning to overcome LLM mode collapse. Its potential impact is vast, supported by concrete, state-of-the-art breakthroughs across four distinct biomedical problems. While Paper 2 offers an important methodological critique of AI alignment evaluation, Paper 1 provides a directly applicable, cross-disciplinary tool that significantly advances the frontier of AI-driven scientific research, offering broader and more immediate transformative effects across STEM fields.
Paper 2 is more likely to have higher scientific impact due to its conceptual novelty (shifting safety guarantees from model behavior to formally verified agentic frameworks) and broad, timely relevance to AI safety. It introduces a general verification paradigm (havoc oracle semantics, boundary-enforceable properties) with mechanized proofs in Dafny and a concrete verified framework (PocketFlow), enabling rigorous, capability-invariant guarantees. This could influence multiple areas—formal methods, security, agent frameworks, and governance—whereas Paper 1, though strong and application-validated, is a narrower methodology improvement within LLM-assisted scientific ideation.
Paper 1 likely has higher scientific impact due to stronger cross-domain breadth and timeliness: it introduces an explicit analogical-reasoning framework to mitigate LLM mode collapse in open-ended scientific solution generation, with large diversity/novelty gains and multiple validated biomedical deployments (including SOTA results). This positions it as a generally applicable method for autonomous science and creative discovery. Paper 2 is methodologically solid and practically valuable for e-commerce simulation/personalization, but its impact is more domain-specific and hinges on access to large proprietary clickstream data, potentially limiting broad scientific reuse.
Paper 2 demonstrates significant real-world scientific impact by applying analogical reasoning in LLMs directly to complex biomedical problems. While Paper 1 introduces an innovative structural mechanism for LLM reasoning, Paper 2 bridges AI and autonomous scientific discovery, showing state-of-the-art results across multiple concrete biological applications. This cross-disciplinary utility and tangible acceleration of scientific research gives Paper 2 a broader and more profound potential impact.
Paper 1 tackles the profound challenge of autonomous scientific discovery, demonstrating substantial real-world validation across multiple biomedical domains. Its interdisciplinary approach and broad applicability offer immense potential to accelerate scientific workflows. In contrast, Paper 2 addresses a specific, albeit rigorous, technical issue within AI agent memory systems, making its impact narrower and primarily confined to the AI engineering community.
Paper 2 offers higher scientific impact because it addresses a fundamental limitation of LLMs in science (mode collapse) by introducing analogical reasoning. While Paper 1 provides a highly practical zero-code tool for data processing, Paper 2's approach enhances the core creative capabilities of autonomous AI scientists. Its demonstration of state-of-the-art results and massive diversity improvements across complex biomedical tasks suggests a broader, cross-disciplinary impact on how AI generates novel scientific hypotheses and solutions, extending beyond automated data processing into true scientific discovery.
Paper 1 likely has higher scientific impact due to a more foundational, broadly applicable systems contribution: a typed, compositional code-DAG representation that jointly addresses tool-library scaling, retrieval under context constraints, and continual skill abstraction. It combines algorithmic innovation with theoretical guarantees (sublinear retrieval, monotone co-evolution, well-formedness) and strong cross-domain benchmarks (math, tables, code), suggesting wide adoption potential for tool-augmented agents. Paper 2 is timely and shows strong biomedical case studies, but its contribution is more task-level and may generalize less than a unified agent/tooling framework.
While Paper 1 offers a highly efficient approach to LLM alignment, Paper 2 demonstrates a broader and more transformative scientific impact by successfully applying LLMs to autonomous scientific discovery. Paper 2's analogical reasoning method not only solves AI mode collapse but provides empirically validated, state-of-the-art results across four diverse biomedical problems. This bridging of AI and empirical science suggests a paradigm-shifting tool for cross-disciplinary research.
Paper 1 likely has higher scientific impact due to greater novelty (a concrete analogical-reasoning framework to mitigate LLM mode collapse in open-ended scientific solution generation), stronger demonstrated real-world applicability (implemented across four biomedical tasks with large quantitative gains, including SOTA results), and broader cross-field relevance (generative creativity, scientific discovery workflows, biomedicine). Paper 2 is timely and methodologically solid for improving long CoT via insight replay, but its gains are modest on average and primarily impact evaluation-time reasoning rather than directly enabling new scientific applications.
Paper 2 demonstrates broader and more profound scientific impact by introducing a method to enhance LLM creativity and scientific discovery across domains. While Paper 1 provides a valuable benchmark for spatial reasoning in AI, Paper 2 actively generates novel, real-world solutions to complex biomedical problems, achieving state-of-the-art results and massive improvements in diversity metrics. The ability to autonomously augment scientific discovery through analogical reasoning has transformative potential across multiple scientific disciplines, giving it a significantly higher ceiling for real-world application and cross-field impact compared to an AI benchmarking environment.
Paper 2 likely has higher scientific impact due to strong timeliness (LLMs for autonomous science), broad cross-field relevance (ML methods applicable beyond biomedicine), and clear real-world validation with substantial quantitative gains across multiple biomedical tasks. Its contribution—analogical reasoning to mitigate mode collapse in open-ended generation—targets a widely recognized limitation and could generalize to many discovery workflows. Paper 1 is methodologically rigorous and novel in stochastic multi-round allocation, but its application scope is narrower and the broader interdisciplinary reach is likely smaller than a generally applicable LLM reasoning framework.
Paper 1 offers a concrete, novel method (analogical reasoning) to address an observed failure mode (mode collapse) in open-ended LLM solution generation, with quantified gains and multiple real biomedical validations including strong predictive improvements and SOTA results—supporting methodological rigor and near-term real-world impact. Paper 2 proposes an important systems/data-model concept for agentic workflows, likely impactful for tooling and governance, but it appears more conceptual and less empirically validated in the abstract. Overall, Paper 1 has clearer evidence of immediate cross-domain scientific utility and measurable advances.
Paper 1 is more likely to have higher scientific impact due to its broader cross-domain relevance (LLM solution generation for autonomous science), clearer novelty (analogical reasoning to mitigate mode collapse), and stronger demonstrated real-world utility via multiple biomedical implementations with substantial quantitative gains. Its applications extend beyond biomedicine to general scientific discovery and creative problem-solving with LLMs, aligning with timely interest in AI-for-science. Paper 2 is solid and introduces a useful behavioral metric, but its scope is narrower (chess/player modeling) and the methodological contribution is more incremental (embeddings + limited MCTS) with less obvious downstream societal impact.
Paper 2 likely has higher impact: it introduces a broadly applicable method (analogical reasoning) to improve LLM-driven open-ended scientific solution generation, addressing a timely core limitation (mode collapse) with large gains in diversity/novelty and multiple real biomedical validations, including state-of-the-art results. This combination of methodological contribution plus cross-domain applicability positions it to influence autonomous science, ML, and biomedical ML. Paper 1 is innovative and well-engineered for UAV/HAPS networking, but its impact is more domain-specific and depends on deployment feasibility and LLM-in-control acceptance.
Paper 1 likely has higher scientific impact: it introduces a broadly applicable analogical-reasoning framework to mitigate LLM mode collapse in open-ended scientific solution generation, with large diversity/novelty gains and multiple concrete biomedical implementations showing strong quantitative improvements, including SOTA results. This combination of methodological novelty, real-world validation across several bio tasks, and relevance to autonomous science suggests wider cross-field uptake. Paper 2 is timely and rigorous for fact verification, but its application scope is narrower (MHFV) and impact may be more confined to NLP evaluation/verification benchmarks.
Paper 2 addresses a fundamental bottleneck in AI-driven scientific discovery (mode collapse in open-ended problem solving) and introduces a novel Analogical Reasoning approach. Its broad applicability across scientific domains, combined with rigorous empirical validation demonstrating state-of-the-art results across four distinct biomedical problems, suggests a much higher potential for transformative real-world impact compared to Paper 1's domain-specific educational dataset.
Paper 1 introduces a broadly applicable, timely method (analogical reasoning) to address LLM mode collapse in open-ended scientific solution generation, with large reported diversity/novelty gains and multiple concrete biomedical validations (including SOTA results), suggesting high real-world utility and cross-field impact (AI + scientific discovery). Paper 2 is a solid, incremental advance in temporal KG extrapolation by integrating ontology information to mitigate sparsity, but its impact is more domain-specific and primarily methodological within KG research, with less demonstrated downstream societal/scientific application breadth.