FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast

Paper 2 demonstrates higher potential impact through superior methodological rigor and advancements in autonomous agent memory. While Paper 1 offers an interesting proof-of-concept for KG-based metacognition, its evaluation is limited to a small sample size (90 queries). Paper 2 tackles the critical challenge of test-time compute and agent memory evolution without weight updates, utilizing comprehensive ablations across four LLM families in a complex POMDP environment. Its insights into population broadcast mechanics provide foundational knowledge for scaling self-improving, multi-agent AI systems.

Paper 1 (FORGE) introduces a more novel and broadly applicable framework—population-based memory evolution for LLM agents without weight updates—addressing a fundamental challenge in agent learning. It demonstrates rigorous evaluation across 4 LLM families with ablations confirming mechanism contributions. Paper 2 (PPR-GDE) addresses diversity collapse in RL for open-ended generation, which is relevant but more incremental, limited to role-playing tasks, and builds on well-established RLHF/preference optimization paradigms. FORGE's approach to emergent agent memory has broader potential impact across agentic AI applications.

Paper 2 (LMAC) addresses a fundamental challenge in multi-agent reinforcement learning—communication under partial observability—with broader applicability across diverse MARL benchmarks. Its contribution of using LLMs to design communication protocols is novel and generalizable, potentially impacting robotics, autonomous systems, and distributed AI. Paper 1 (FORGE), while methodologically interesting, is evaluated only on a single environment (CAGE-2 B-line) with acknowledged limited generalizability. Paper 2's broader experimental validation and wider applicability to the large MARL community give it higher potential impact.

Paper 2 is likely higher impact: it introduces a large, released, standardized benchmark targeting a major industrial domain with proprietary-documentation grounding and end-to-end workflow evaluation, enabling broad, reproducible comparisons and serving as shared infrastructure for many follow-on studies. Its “Execution Wall” finding is a timely, actionable diagnostic for deploying LLM agents in real systems and is likely to influence evaluation methodology across domains. Paper 1 is novel and shows strong gains, but evidence is confined to a single environment/attacker setting, limiting generality and cross-field uptake.

Paper 1 presents a concrete, novel algorithmic protocol (population-broadcast self-evolving memory with no weight updates) with substantial empirical gains on a challenging, stochastic long-horizon cyber-defense benchmark, plus ablations identifying key mechanisms—supporting methodological rigor and near-term applicability. Its approach is timely for agent reliability and can plausibly transfer across agentic tasks where prompt-memory is used. Paper 2 is timely and potentially broadly influential conceptually, but as a position paper it offers limited empirical validation and actionable implementation detail, making near-term scientific/engineering impact less certain.

FORGE introduces a novel, broadly applicable framework for improving LLM agent performance through self-evolving memory without weight updates—a paradigm with wide applicability across agentic AI tasks. Its population-based broadcast mechanism is a genuinely new contribution with clear ablation evidence. Paper 1, while addressing an important gap (logicality in scientific reasoning), is more incremental, focused narrowly on physics QA benchmarks, and primarily contributes a data curation methodology rather than a fundamentally new technique. FORGE's cross-model generality and practical implications for resource-efficient agent improvement give it broader potential impact.

Paper 1 focuses on automatic equation discovery (symbolic regression), a fundamental challenge in 'AI for Science' with broad applicability across physics, biology, and chemistry. Its ability to reliably extract symbolic laws from data promises direct impact on scientific discovery. While Paper 2 presents an interesting population-based memory evolution for agents, its empirical validation is strictly confined to a single cybersecurity POMDP environment, significantly limiting its proven generalizability and immediate cross-disciplinary scientific impact compared to Paper 1.

Paper 2 (FORGE) presents a concrete, empirically validated method for improving LLM agent performance without weight updates, demonstrating clear quantitative gains across multiple model families on a challenging benchmark. Its contributions—population-based memory evolution, broadcast mechanisms, and representation comparisons—are immediately actionable and broadly applicable to the growing field of LLM agents. Paper 1 (SEED) introduces a conceptual framework for experimental design representation that, while intellectually interesting, is more niche, relies on a lightweight feasibility test rather than rigorous empirical validation, and addresses a narrower audience primarily concerned with experimental methodology for AI systems.

Paper 2 has higher potential impact due to a clearer, broadly applicable framing (personalization failures as commitment/constraint management, not recall) and a method that is model-agnostic and evaluable via verifiable guarantees (“zero failures within validator scope”) with explicit trade-offs (availability vs safety, recall limits). The approach generalizes across many personalized LLM products (assistants, agents, long-context, memory tools) and is timely amid deployment concerns. Paper 1 is useful and novel for agent self-improvement, but evidence is confined to a single benchmark/attacker setting and may generalize less beyond agent RL-style tasks.

Paper 1 introduces a training-free, plug-and-play memory module applicable to a wide range of LLMs without requiring gradient updates. Its broad evaluation across various model sizes, modalities, and general benchmarks (code, QA) suggests high versatility and ease of adoption. In contrast, Paper 2 presents a prompt-based memory evolution framework evaluated on a single, highly specific cybersecurity environment. Consequently, Paper 1 possesses significantly higher breadth of impact and potential for widespread integration into existing LLM architectures.

Paper 2 likely has higher scientific impact: it proposes a unified, intervention-aware framework for clinical trajectory prediction that connects forecasting, counterfactual estimation, and policy evaluation while explicitly treating treatment/observation feedback—central limitations in current clinical AI. Its potential real-world applications (decision-grade evidence, treatment policy stress-testing, safer learning health systems) are broad and timely, spanning medicine, causal inference, time-series modeling, and health policy. Paper 1 is novel and empirically solid within LLM agent training-without-updates, but its evidence is confined to a single benchmark setting, limiting breadth and external validity.

Paper 2 is a comprehensive survey covering AI for inverse PDE problems—a foundational topic spanning medical imaging, geophysics, materials science, and aerodynamics. Its breadth of impact across multiple scientific and engineering fields, combined with its role as the first unified systematic review of this area, gives it high citation potential and broad relevance. Paper 1, while methodologically interesting, addresses a narrow domain (a single cybersecurity benchmark) with explicitly acknowledged limited generalizability, restricting its broader scientific impact.

Paper 1 introduces a novel, generalizable framework (FORGE) for evolving LLM agent memory without weight updates, demonstrating systematic methodology across multiple LLM families with ablation studies. Its contributions to population-based prompt evolution and memory propagation have broader applicability beyond the specific benchmark. Paper 2, while practically impactful at Baidu Maps scale, is more of an engineering contribution with incremental improvements to an existing production system. Paper 1's methodological novelty, cross-model generalizability, and potential to influence the growing field of LLM agent self-improvement give it higher scientific impact potential.

Paper 1 addresses a core challenge in AI—enhancing LLM agent decision-making and memory without gradient updates—which has broad applicability across complex environments. Its rigorous evaluation across multiple LLM families and substantial performance gains over strong baselines indicate high methodological rigor and significant potential impact. Paper 2, while novel in combining LLMs with Fuzzy Cognitive Maps for political science modeling, focuses on a more niche methodology and application, likely limiting its broader scientific influence compared to the foundational agent improvements in Paper 1.

Paper 1 introduces a novel, weight-free self-improvement protocol (population broadcast + graduation) for LLM agents and demonstrates sizable, consistent gains across multiple model families in a challenging, stochastic long-horizon cyber-defense POMDP. If robust, this could generalize broadly to agent training/operation where finetuning is infeasible, impacting RL/agents, security automation, and prompt-based learning methods. Paper 2 is a valuable benchmark with clear applications, but its impact is more domain-scoped (Chinese gaming vertical) and primarily evaluative rather than proposing a broadly reusable learning mechanism.

Paper 1 presents a highly novel, generalized approach for LLM agent self-improvement without weight updates, demonstrating significant quantitative gains across multiple state-of-the-art models. Its population-based memory broadcast technique has broad implications across the rapidly growing field of AI agents. In contrast, Paper 2, while highly useful for the EDA community, focuses on benchmark maintenance for RTL generation, giving it a much narrower scope and applicability compared to the fundamental algorithmic advancements proposed in Paper 1.

Paper 2 (FORGE) has higher potential scientific impact due to a more novel algorithmic contribution (population-broadcast, self-evolving memory without weight updates) and broader relevance to agent learning, continual adaptation, and robustness across model families. Its results target a challenging, stochastic, long-horizon POMDP domain (cyber defense) with clear performance gains and ablations identifying key mechanisms, suggesting methodological rigor. Paper 1 (PRISM) is highly applicable and timely for enterprise reliability, but is more of a systems/engineering framework with narrower scientific generality and heavier dependence on platform-specific evaluation and LLM-as-judge metrics.

Paper 1 presents a highly impactful approach to automated neural architecture search, demonstrating progress toward recursive self-improvement where AI designs better AI. Its discovery of architectures that outperform strong baselines like Llama 3.2 at the 1B scale has broad implications for foundation model development across all domains. In contrast, Paper 2 offers a valuable but narrower contribution regarding agent memory, explicitly noting its evidence is confined to a single specific benchmark (CAGE-2 B-line). Paper 1's generalizability and potential to shift the paradigm of foundation model design give it significantly higher scientific impact.

Paper 2 targets competitive programming, a highly rigorous and widely recognized benchmark for complex LLM reasoning. By establishing a new state-of-the-art across multiple established datasets using a novel graph-structured knowledge network for continuous learning, it demonstrates broader applicability and stronger methodological validation than Paper 1, which evaluates its approach on a single specific network-defense environment.

Paper 2 is a position paper that sets a broad research agenda for 'Metacognitive AI', addressing fundamental challenges like efficiency, accuracy, and security across multiple AI domains. While Paper 1 offers a strong, specific algorithmic improvement for LLM agents, Paper 2's conceptual framework, backed by a community software tool, has a higher potential for widespread adoption and paradigm-shifting impact across the broader AI landscape.