MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models
Jason Z Wang
Abstract
We introduce MIRROR, a benchmark comprising eight experiments across four metacognitive levels that evaluates whether large language models can use self-knowledge to make better decisions. We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances using five independent behavioral measurement channels. Core experiments are run across the full model roster; experiments with specialized infrastructure requirements report explicitly marked model subsets. We find two phenomena with direct implications for agentic deployment: (1) compositional self-prediction fails universally -- the Compositional Calibration Error ranges from 0.500 to 0.943 on the original 15-model Exp3-v1 set (and 0.434 to 0.758 on the balanced 16-model Exp3-v2 expansion), indicating that models cannot predict their own performance on multi-domain tasks, and (2) models exhibit above-chance but imperfect domain-specific self-knowledge yet systematically fail to translate even this partial awareness into appropriate agentic action-selection -- external metacognitive control reduces the Confident Failure Rate from 0.600 to 0.143 (76% reduction at temperature 0; mean 70% at temperature 0.7 across 5 models from 4 labs). Providing models with their own calibration scores produces no significant improvement (p > 0.05); only architectural constraint is effective. This suggests that external metacognitive scaffolding -- not improved self-knowledge -- is the path to safer autonomous AI systems. Code, data, and Croissant metadata will be released publicly with the benchmark.
AI Impact Assessments
(1 models)Scientific Impact Assessment: MIRROR Benchmark
1. Core Contribution
MIRROR introduces a hierarchical benchmark that systematically tests whether LLMs can translate self-knowledge into appropriate action — the gap between metacognitive *monitoring* (knowing what you know) and metacognitive *control* (acting on that knowledge). The benchmark spans four metacognitive levels (atomic self-knowledge, cross-domain transfer, compositional prediction, and adaptive self-regulation) with eight experiments and five independent behavioral measurement channels. The core novelty is not in measuring calibration per se, but in measuring the full pipeline from calibration to action-selection, operationalized through a four-condition escalation curve in Experiment 9.
The two headline findings — universal compositional self-prediction failure (CCE 0.434–0.943) and the knowing-doing gap (76% CFR reduction only through external constraint) — are well-defined and practically consequential. The demonstration that providing models their own calibration scores produces no significant improvement (C1→C2, p=0.90) while architectural constraint produces large effects (C3→C4, d=1.21) is a clean and actionable result.
2. Methodological Rigor
Strengths in design: The five independent behavioral channels (wagering, opt-out, difficulty selection, tool delegation, natural language signals) provide convergent measurement that reduces the risk of prompt-specific artifacts. The four-condition escalation design (uninformed → self-informed → instructed → constrained) is well-structured for isolating the monitoring-control dissociation.
Statistical rigor: Bootstrap BCa confidence intervals (10,000 iterations) are used throughout. Effect sizes are reported. The paper includes extensive sensitivity analyses: temperature robustness (t=0.7 across 5 models from 4 labs), weak-domain threshold sensitivity (three definitions), parse/API missingness diagnostics (complete-case, conservative, IPW), format-matched controls, and wager-independent sensitivity checks.
Concerns:
3. Potential Impact
Immediate applications: The finding that domain-level routing outperforms instance-level confidence thresholding at matched escalation budgets (Appendix O) is directly deployable. The practical implication — build external scaffolding rather than trusting model self-assessment — is simple and actionable for agentic system designers.
Benchmark utility: MIRROR fills a genuine gap between calibration benchmarks (which measure monitoring only) and agentic benchmarks (which measure execution only). The ~3-hour API-only evaluation makes it accessible. The 5-channel design is a methodological contribution that others can adopt.
Broader influence: The monitoring-to-control framework borrowed from Nelson & Narens (1990) provides useful conceptual vocabulary for the field. The evidence that self-knowledge doesn't compose across domains (TII 0.019–0.175) has implications for multi-agent systems and tool-use architectures.
4. Timeliness & Relevance
This paper addresses a pressing need as LLMs are increasingly deployed as autonomous agents. The assumption that models can self-monitor and appropriately defer is embedded in many agentic architectures (Reflexion, Constitutional AI, confidence-based routing). Demonstrating that this assumption fails systematically, and quantifying the cost (~562 vs ~214 failures per 1,000 tasks), is highly timely.
The concurrent work landscape (Schmied et al. 2025, Qiao et al. 2025, Barkan et al. 2025) validates the timeliness but also suggests MIRROR may be one of several parallel efforts converging on similar conclusions.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
Reproducibility: High for the analysis pipeline; moderate for data generation (LLM-dependent question generation). The planned public release of all artifacts is commendable.
Summary
MIRROR is an ambitious and well-executed benchmark paper that addresses a real gap in LLM evaluation. The escalation curve finding is clean, large, and robust across multiple sensitivity analyses. The practical implications are clear. The main limitations are the low baseline accuracy raising interpretive questions, limited frontier model coverage, and some framing choices that overstate certain claims. The extensive appendix material and planned artifact release demonstrate thoroughness.
Generated May 5, 2026
Comparison History (30)
Paper 2 introduces a comprehensive benchmark revealing fundamental limitations in LLMs' metacognitive capabilities and self-prediction. Identifying these universal failures and concluding that external metacognitive scaffolding is required for safer autonomous AI systems will likely have a broader, more profound impact on AI safety, alignment, and agentic design than Paper 1's domain-specific algorithmic improvements in multi-hop QA.
Paper 1 likely has higher scientific impact due to broader relevance and timeliness: metacognitive calibration and safe agentic deployment are central, cross-cutting problems in modern AI. MIRROR contributes a large, multi-level benchmark with extensive model coverage and multiple measurement channels, supporting methodological rigor and reproducibility. Its core findings (systematic compositional self-prediction failure; effectiveness of external scaffolding over self-score feedback) are actionable for safety and evaluation research and could influence standards across labs. Paper 2 targets an important but narrower RAG preprocessing issue with promising results, yet appears more domain-specific and potentially less generalizable.
Paper 1 introduces a large-scale benchmark addressing a fundamental limitation in LLMs (metacognitive calibration), providing critical insights for the safety and deployment of autonomous agents. Its discovery that models universally fail at compositional self-prediction and require external scaffolding challenges current assumptions, likely influencing future AI architectures. Paper 2 offers a valuable but more incremental framework for retrieval-augmented generation, entering a highly crowded field with a narrower scope of impact.
Paper 1 likely has higher scientific impact due to broader relevance and timeliness: metacognitive calibration is central to safe, agentic LLM deployment across many tasks and domains. MIRROR offers a large, multi-lab, multi-model benchmark with substantial evaluation scale and multiple measurement channels, supporting methodological rigor and reproducibility. Its negative/diagnostic findings (universal compositional self-prediction failure; limited benefit of providing calibration scores; effectiveness of architectural constraints) are actionable for safety and agent design. Paper 2 is practically valuable for RAG, but appears narrower and more application-specific.
Paper 2 introduces a comprehensive benchmark revealing fundamental limitations in LLMs' metacognitive capabilities and self-calibration. Its large-scale empirical findings directly challenge current assumptions about autonomous AI safety, suggesting a paradigm shift towards external scaffolding. This will likely have a broader and more immediate impact across AI safety and deployment than the specialized methodological advancements in uncertainty quantification presented in Paper 1.
Paper 2 introduces a comprehensive benchmark revealing fundamental limitations in LLM metacognition and self-prediction. By demonstrating that models systematically fail to translate self-knowledge into appropriate action-selection, it provides actionable, paradigm-shifting insights for AI safety and agentic deployment—specifically, the necessity of external scaffolding over internal self-knowledge. This broad empirical evaluation across multiple models will likely drive widespread future research in autonomous AI, giving it a broader and more immediate scientific impact than the methodological advancements of Paper 1.
OLLM introduces a novel architectural contribution (options-based next-token prediction with discrete latent variables) that addresses fundamental limitations in LLM generation diversity, controllability, and alignment. Its lightweight plug-in design applicable to any pretrained LLM, significant performance gains (51% to ~70% on math reasoning), and the structural approach to alignment without KL penalties represent broadly applicable innovations. While MIRROR provides valuable empirical insights about metacognitive failures in LLMs, it is primarily a benchmark/evaluation contribution. OLLM's architectural innovation has greater potential to influence future model design, RL-based alignment, and controllable generation across many domains.
OLLM introduces a novel architectural innovation (options-based next-token prediction with discrete latent variables) that addresses fundamental limitations of standard LLM generation. It offers a general-purpose, lightweight plug-in applicable to any pretrained LLM, with strong empirical gains (51%→70% on math reasoning) and a principled approach to controllability and alignment through structure rather than heuristics. This has broad implications for RL-based LLM training, diverse generation, and alignment. While MIRROR provides valuable empirical insights about metacognitive calibration, it is primarily a benchmark/evaluation contribution with less architectural novelty and narrower methodological impact.
Paper 2 likely has higher scientific impact: it introduces a large, hierarchical benchmark for metacognitive calibration with broad, timely relevance to safe agentic LLM deployment across many domains. Its methodology is extensive (16 models, ~250k instances, multiple measurement channels) and yields generalizable negative results plus a clear actionable implication (external scaffolding/architectural constraints outperform self-knowledge). Paper 1 is innovative and application-driven, but its impact is narrower (human-vehicle collaboration) and more dependent on domain-specific validation and deployment constraints.
Paper 1 likely has higher scientific impact due to its broad, timely relevance to LLM safety/agent deployment and its benchmark nature: MIRROR provides a large-scale, multi-level evaluation (16 models, ~250k instances, multiple measurement channels) with clear, generalizable findings about metacognitive calibration and control. Benchmarks often become community standards, shaping future research across AI safety, alignment, evaluation, and agent design. Paper 2 is application-important for human-vehicle collaboration, but appears more domain-specific and less likely to become a cross-field reference point than a widely adopted LLM metacognition benchmark.
RePAIR introduces a novel paradigm (Interactive Machine Unlearning) with a concrete technical contribution (STAMP) that enables user-driven, training-free knowledge removal from LLMs. It addresses critical practical needs (privacy, harmful content removal) with strong empirical results and clear computational advantages. While MIRROR provides valuable benchmarking insights about metacognitive calibration, its contributions are primarily diagnostic rather than constructive. RePAIR's actionable framework with immediate real-world applications in privacy compliance, safety, and on-device deployment gives it broader and more transformative impact potential across multiple research communities.
MIRROR addresses a fundamental question about LLM self-knowledge and metacognition with broad implications across all agentic AI deployment, not just one domain. Its finding that external metacognitive scaffolding—not improved self-knowledge—is necessary for safer autonomous AI is a high-impact insight applicable across fields. The benchmark's scale (16 models, 250K instances, 8 experiments) and the universality of its findings (compositional self-prediction fails universally) make it likely to influence AI safety research broadly. FinGround, while rigorous and practically valuable, is more narrowly scoped to financial document QA.
Paper 2 introduces a comprehensive, large-scale benchmark (MIRROR) addressing a critical bottleneck in autonomous AI: metacognitive calibration. By empirically demonstrating that LLMs systematically fail at compositional self-prediction and cannot leverage self-knowledge for safe action-selection, it fundamentally shifts the focus of agentic AI safety toward external metacognitive scaffolding. While Paper 1 offers a valuable, counter-intuitive insight into prompting dynamics, Paper 2 provides a reusable evaluation tool and broad architectural recommendations that will likely steer future research in AI safety and autonomous agent design.
Paper 2 likely has higher impact due to broader novelty and scope: a large, hierarchical benchmark for metacognitive calibration with extensive multi-model, multi-channel evaluation and clear quantitative findings that generalize beyond one domain. Its results directly inform agentic safety design (external scaffolding vs. self-knowledge), with strong methodological rigor (250k instances, many labs, statistical testing) and broad relevance across AI alignment, evaluation, and deployment. Paper 1 is timely and useful for finance safety, but is narrower in domain and contribution size (task suite + mitigation benchmarking).
Paper 1 likely has higher impact: it introduces a field-defining framework for agent accountability via auditability (clear concepts, dimensions, mechanism taxonomy) and backs it with multi-pronged empirical evidence plus actionable artifacts (Auditability Card, open problems). Its real-world applicability to deployed agent systems (security, compliance, incident response) is immediate and cross-cutting across AI, security, and governance. Paper 2 is rigorous and useful as a benchmark, but its impact is narrower (metacognitive calibration evaluation) and more incremental relative to the broader socio-technical need for auditable autonomous systems.
MIRROR addresses a fundamental question about LLM metacognition and self-knowledge with a large-scale benchmark (16 models, ~250K instances, 5 measurement channels). Its finding that external metacognitive scaffolding—not improved self-knowledge—is needed for safer autonomous AI has broad implications for the entire agentic AI field. Paper 2 makes a solid engineering contribution to AI memory systems with strong empirical results, but its scope is narrower (memory architecture design) and more incremental. MIRROR's systematic evaluation of metacognitive calibration failure is more likely to influence safety research, alignment, and agentic system design across the field.
Paper 1 presents a fundamentally novel finding about internal emotion representations in LLMs that causally influence alignment-critical behaviors like reward hacking, blackmail, and sycophancy. This mechanistic interpretability work opens new research directions for understanding and controlling LLM behavior at the representation level. Paper 2 contributes a useful benchmark for metacognitive calibration, but benchmarks have more incremental impact. Paper 1's discovery of 'functional emotions' as causal mediators of misaligned behavior has broader implications for AI safety, interpretability, and cognitive science, likely generating more follow-up research and practical applications.
Paper 1 is likely higher impact due to a more novel and broadly relevant benchmark for metacognitive calibration, large-scale multi-lab evaluation (16 models, ~250k instances), multiple measurement channels, and clear, actionable safety implications (external scaffolding vs. self-knowledge). Its findings generalize across agentic deployments beyond a single domain. Paper 2 is timely and practical (tool-use planning + uncertainty-guided search) but appears more domain-specific (e-commerce toolkit) and the algorithmic contribution (entropy-guided branching) is less fundamentally new than Paper 1’s comprehensive metacognition evaluation and negative/diagnostic results.
Paper 1 introduces a comprehensive benchmark for LLM metacognition, addressing a critical and timely issue for autonomous AI safety. Benchmarks in the LLM domain typically drive significant empirical research and accrue high citations. While Paper 2 offers exceptional methodological rigor through mechanized proofs, its highly theoretical focus on formal structural governance will likely impact a narrower subset of the formal verification and AI safety communities compared to the broad, immediate applicability of Paper 1.
Paper 1 introduces a large-scale benchmark uncovering fundamental limitations in LLM metacognition, offering critical insights for AI safety and autonomous agent deployment. Its conclusion that external scaffolding is required over improved self-knowledge shifts current paradigms. Paper 2, while methodologically rigorous, offers a more incremental algorithmic improvement to RL training. Paper 1's broader implications for understanding LLM capabilities and safety give it a higher potential scientific impact.