Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Ziyan Liu, Zhezheng Hao, Yeqiu Chen, Hong Wang, Jingren Hou, Ruiyi Ding, Yongkang Yang, Wence Ji
Abstract
Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.
AI Impact Assessments
(1 models)Scientific Impact Assessment: Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
1. Core Contribution
The paper identifies a specific failure mode in memory-augmented LLM agents: recursive summarization accumulates semantic noise that progressively degrades the agent's internal belief about the latent task state, and outcome-based RL training cannot localize which intermediate summary caused this degradation. The core contribution is twofold:
Belief Entropy (BE): A self-supervised proxy signal that estimates the agent's epistemic uncertainty about the latent task state given its current compressed memory. This is operationalized via an "anchor question" that probes both task progress and remaining information gaps, with uncertainty measured through token-level predictive entropy of the model's response.
MMPO Framework: A training procedure that augments outcome-based RL with dense, per-turn rewards derived from Belief Entropy. Sub-trajectory rewards combine normalized BE (via sigmoid transformation) with terminal outcome rewards, and group-relative advantage estimation provides credit assignment at the turn level.
The formulation is grounded in POMDPs, connecting summary-induced belief to the conditional entropy H(s_t|m_t), and arguing that minimizing this quantity is equivalent to maximizing mutual information between summaries and latent task states.
2. Methodological Rigor
Theoretical grounding: The POMDP formulation is clean and well-motivated. The chain st → ht → mt and the data processing inequality argument elegantly justify why summarization quality matters. The information-theoretic decomposition H(y|m_t, q) = H(y|m_t, q, s_t) + I(y; s_t|m_t, q) provides a principled justification for the anchor question design.
However, the gap between the theoretical objective (minimizing H(s_t|m_t)) and the practical proxy (H(y|m_t, q)) relies on assumptions A1 (relevance) and A2 (memory grounding) that are stated but not rigorously verified. The approximation in Eq. 23-24 assumes state-conditioned response uncertainty is "approximately stable" across memories—a non-trivial assumption that is not empirically validated.
Empirical validation of BE: The three-pronged empirical validation (trajectory dynamics, outcome correlation, inference-time selection) is convincing. The Pearson correlation of r = -0.684 between ΔH_BE and task accuracy, and the Best-of-N selection improvement without training, provide strong evidence that BE captures meaningful signal before being used for optimization.
Experimental design: The paper evaluates on two distinct memory-agent frameworks (MemAgent and MEM1) across multiple tasks (RULER-HotpotQA, multi-objective QA, WebShop), which strengthens generalizability claims. The improvements are modest but consistent: +3.14% average at long contexts for 7B, and meaningful gains on MEM1 benchmarks. The 97.1% performance retention at 1.75M tokens is notable.
Weaknesses in rigor: The ablation study (Table 4) is limited to a single context length (56K). The full robustness study in Table 5 extends this but only covers Qwen2.5-7B. The comparison with alternative proxies (Table 6) is informative but also single-setting. Statistical significance is not reported for any result. The claim of "97.1% performance" at 1.75M tokens appears to be computed relative to 7K performance (78.91/82.38 ≈ 95.8% for 7B), and the exact computation is unclear.
3. Potential Impact
Direct applications: The framework addresses a genuine bottleneck in deploying LLM agents for long-horizon tasks—document analysis, multi-step research, extended dialogue, and interactive tool use. The ability to maintain performance at million-token scales is practically valuable.
Broader influence: The concept of using metacognitive probes to generate dense intermediate rewards for memory policies could generalize beyond the specific instantiation here. The anchor question mechanism is a simple, transferable idea. The connection between belief preservation and memory quality could influence how the community thinks about memory architecture design.
Limitations of impact: The improvements, while consistent, are incremental (typically 2-5%). The method adds training complexity (12% overhead) and requires task-specific anchor question design. The paper's claim that the anchor question can be straightforwardly adapted across tasks is demonstrated only for QA and simple tool-use settings.
4. Timeliness & Relevance
This paper is highly timely. Memory-augmented LLM agents are an active frontier (MemGPT, MEM1, MemAgent all from 2025), and the credit assignment problem in training memory policies is a recognized bottleneck. The work on RLVR for reasoning (DeepSeek-R1, GRPO) has created a natural question about how to extend these ideas to memory optimization. MMPO provides a clean answer by defining what "good intermediate memory" means through the belief-preservation lens.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
6. Additional Observations
The paper's positioning as "metacognitive" is inspired by cognitive science literature (Flavell, Nelson & Narens), but the connection is more metaphorical than substantive—the anchor question is a simple prompt-based uncertainty probe, not a genuine metacognitive mechanism. The framing is somewhat oversold relative to the technical contribution, which is essentially: use entropy of anchor-question responses as a dense reward for memory policy training.
The sub-trajectory advantage aggregation (Eq. 8) and the connection to GRPO are technically sound and represent a clean contribution to credit assignment for memory agents.
Generated May 29, 2026
Comparison History (16)
Paper 1 introduces a more fundamental and broadly applicable contribution: a novel self-supervised proxy (Belief Entropy) for optimizing memory policies in long-horizon LLM agents, with strong theoretical grounding in metacognition and information theory. Its ability to maintain 97.1% performance at 1.75M-token contexts addresses a critical scalability bottleneck. While Paper 2's Map-then-Act paradigm is clever and practical, it is more of an engineering framework. Paper 1's fine-grained memory supervision mechanism has broader methodological implications for RL-based agent training and could influence multiple research directions in long-context reasoning.
Paper 2 has higher likely impact because it establishes a formalized, broadly useful evaluation target (EnvSim Ability) plus a sizeable, verifiable benchmark and a concrete mitigation pipeline. Benchmarks that expose a universal failure mode (“state change cliff”) can redirect an entire subfield, are widely reusable across labs/models, and directly support scalable agent training with strong real-world relevance and timeliness. Paper 1 is novel and valuable for long-horizon agent memory optimization, but its impact is narrower and more method-specific, while Paper 2 provides infrastructure and diagnostics with cross-domain reach.
Paper 2 likely has higher scientific impact because it delivers an immediate, field-defining real-world artifact: the largest integrated marine Pb database plus a public visualization platform, enabling new oceanographic and pollution studies. Its expert-guided, verifiable extraction workflow is methodologically grounded with large-scale deployment (230k papers) and manual validation (92% accuracy), and the approach is transferable to other high-stakes scientific domains. Paper 1 is novel for LLM-agent memory optimization, but its impact is more methodological within AI and depends on broader adoption.
Paper 1 introduces a foundational, architecture-level innovation that directly addresses the high computational costs of expanding LLM context windows. By coupling a highly compact online memory state with the attention mechanism without requiring full fine-tuning, it offers significant improvements in serving efficiency and scalability. While Paper 2 presents a strong optimization technique for agent memory, Paper 1's plug-and-play efficiency solution has broader applicability across all long-context LLM deployments, promising a more widespread impact on how models are practically served and utilized.
Seirênes introduces a novel adversarial self-play framework that addresses a fundamental vulnerability in LLM reasoning—fragility to contextual distractions. Its approach is more broadly applicable across model scales (4B-30B), demonstrates strong empirical gains (+7-10 points across seven benchmarks), and reveals vulnerabilities even in top-tier closed-source models. The self-play co-evolutionary curriculum is a creative methodological contribution with broad implications for robust AI training. While MMPO addresses an important memory optimization problem for long-horizon tasks, Seirênes tackles a more fundamental and widespread issue in LLM reasoning with broader potential impact across the field.
Paper 2 addresses a fundamental and pervasive bottleneck in LLMs—memory degradation in long-horizon reasoning—by introducing a novel self-supervised metric (Belief Entropy). Its ability to scale effectively to 1.75M-token contexts offers broader theoretical implications and impact across all LLM applications. In contrast, Paper 1 presents a highly practical but more application-specific framework tailored to data-scarce domains.
Paper 1 proposes a novel, generally applicable training signal (Belief Entropy) for memory policy learning in long-horizon LLM agents, addressing a core failure mode (compounding summary noise) with finer-grained supervision than outcome-only RL. If validated, it can directly improve real-world agent reliability and scaling to very long contexts, with broad impact across agentic systems, memory, and RL/LLM alignment. Paper 2 is timely and useful infrastructure, but as a domain-specific benchmark its impact is more incremental and narrower unless it becomes a widely adopted standard.
Paper 2 likely has higher scientific impact due to a broadly applicable, technically novel training objective (Belief Entropy) for optimizing memory in long-horizon LLM agents, addressing a central bottleneck in agent reliability and scalability. It proposes a general method with clear practical utility across many agentic systems and demonstrates strong empirical results at very long contexts (1.75M tokens), indicating timeliness and engineering relevance. Paper 1 is important for AI safety/social dynamics, but is primarily diagnostic within simulations and may have narrower immediate downstream methodological adoption.
Paper 2 identifies a fundamental and counterintuitive inverse scaling law—larger LLMs become less robust to distractor instructions—which challenges core assumptions about scaling benefits. This finding has broad implications for RAG systems, agentic deployments, and LLM safety. The mechanistic insight via perplexity analysis and the practical GRPO-based mitigation make it highly actionable. Paper 1 addresses an important but more niche problem (memory optimization for long-horizon agents) with strong results, but Paper 2's discovery of a systematic vulnerability affecting all scaled LLMs has wider cross-field impact and timeliness.
Paper 2 addresses a foundational challenge in LLM agents—long-horizon memory and reasoning—with a novel self-supervised metric (Belief Entropy). This methodological advancement has broad implications across virtually all fields utilizing autonomous agents. In contrast, while Paper 1 presents a highly innovative and practical solution for real-time industrial scheduling, its impact is largely constrained to operations research and manufacturing, making Paper 2's potential scientific impact significantly broader and more cross-disciplinary.
Paper 2 is likely higher impact: it introduces a broadly applicable, conceptually novel training signal (Belief Entropy) for diagnosing and optimizing memory quality in long-horizon LLM agents, addressing a central bottleneck for agentic systems. The approach has clear real-world applications (autonomous assistants, tool-using agents) and strong timeliness given rapid growth of long-context and agent research. Its proxy-based, fine-grained supervision may generalize across tasks and architectures. Paper 1 is valuable and novel optimizer-centrically, but its impact is narrower (safety robustness under perturbations) and may be more incremental to existing alignment/robustness lines.
Paper 2 addresses a critical bottleneck in the highly active field of LLM agents: long-horizon memory degradation. By introducing 'Belief Entropy' to optimize memory policies, it demonstrates impressive scalability (maintaining performance up to 1.75M tokens). This offers massive potential for real-world agentic applications, giving it a higher estimated impact compared to Paper 1's focus on dense retrieval interpretability.
Paper 2 likely has higher impact due to a more novel, actionable method (MMPO) with a clear training signal (Belief Entropy) that addresses a central bottleneck in long-horizon LLM agents: memory degradation. It demonstrates strong scalability (up to 1.75M tokens) and broad applicability to agentic systems across domains, making it timely for current industry and research needs. Paper 1 is a rigorous, valuable diagnostic benchmark with important findings, but its primary contribution is evaluative/characterization rather than a generally deployable algorithmic advance.
Paper 1 introduces a novel theoretical concept (Belief Entropy) and a concrete optimization method (MMPO) addressing a fundamental challenge in LLM agents—memory degradation over long horizons. Its strong empirical results at 1.75M-token contexts demonstrate significant technical advancement with broad applicability across diverse tasks. Paper 2 proposes a practical multi-agent medical AI framework, but its contribution is more architectural/engineering-oriented, combining existing components (generalist LLMs + specialist models) in a relatively intuitive way. Paper 1's methodological novelty and generalizability across domains give it higher potential for broad scientific impact.
Paper 2 introduces a highly novel concept (Belief Entropy) to solve memory degradation in long-horizon agents, a critical bottleneck for autonomous AI. While Paper 1 offers a practical deployment optimization for quantization, Paper 2's metacognitive approach scales to massive context lengths and opens new avenues for agentic reasoning, promising broader theoretical and applied impact across AI research.
Paper 1 (MMPO) addresses a fundamental and widely encountered challenge in LLM agents—memory degradation over long horizons—with a practical, well-motivated solution (Belief Entropy as a self-supervised proxy). Its strong empirical results (97.1% at 1.75M tokens) demonstrate clear practical value for the rapidly growing field of LLM agents. Paper 2, while theoretically rigorous in formalizing compositional incoherence, addresses a more niche problem with less immediate practical applicability, and its finding that intuitive mitigations fail limits near-term impact. Paper 1's broader applicability across diverse long-horizon tasks gives it higher potential impact.