AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
Parsa Mazaheri, Kasra Mazaheri
Abstract
Large language model agents now act on codebases, browsers, operating systems, calendars, files, and tool ecosystems, but the benchmarks used to evaluate them are fragmented: each emphasizes a different unit of measurement (final task success, tool-call validity, repeated-pass consistency, trajectory safety, or attack robustness). A line of 2024-2025 work has converged on the diagnosis that a single accuracy column is no longer the right unit of comparison for deployable agents. AgentAtlas extends this line of work with four components: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a nine-category trajectory-failure taxonomy with two orthogonal hierarchical labels (primary_error_source, impact); (iii) a taxonomy-aware vs. taxonomy-blind methodology that measures how much of a model's apparent capability comes from the supervision in the prompt; and (iv) a benchmark-coverage audit mapping fifteen agent benchmarks against six behavioral axes. To demonstrate the methodology we run a small fixed eight-model set (1,342 generated items, four frontier closed and four open-weight) under both prompt modes. Removing the explicit label menu drops every model's trajectory accuracy by 14-40 pp to a tight 0.54-0.62 floor regardless of family, and no single model wins on all three of control accuracy, trajectory diagnosis, and tool-context utility retention. We treat the synthetic run as a measurement-protocol demonstration, not a benchmark release.
AI Impact Assessments
(1 models)Scientific Impact Assessment: AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
1. Core Contribution
AgentAtlas proposes a taxonomic and measurement framework for evaluating LLM agents beyond single-number outcome metrics. It contributes four components: (i) a six-state control-decision taxonomy (Act/Ask/Refuse/Stop/Confirm/Recover), (ii) a nine-category trajectory-failure taxonomy with two orthogonal hierarchical labels (primary_error_source and impact), (iii) a taxonomy-aware vs. taxonomy-blind prompting methodology that quantifies how much of an agent's apparent capability derives from explicit supervision in the prompt, and (iv) a benchmark-coverage audit mapping 15 agent benchmarks against six behavioral axes.
The central thesis — that final task success is an insufficient unit of measurement for deployable LLM agents — is not novel in itself, as the paper openly acknowledges. The contribution lies in operationalizing this insight through a unified vocabulary and demonstrating, via an empirical protocol, that prompt format and evaluation axis meaningfully alter model rankings.
2. Methodological Rigor
The paper has a mixed methodological profile. The benchmark coverage audit (§6) is a useful descriptive contribution, applying a simple 0/1/2 rubric across six axes to 15 benchmarks. While the rubric is somewhat subjective, the resulting coverage map (Fig. 3) clearly exposes gaps — particularly in efficiency (no benchmark scores "strong") and memory/state (one benchmark).
The empirical demonstration (§7) is more problematic. The 1,342-item synthetic dataset was generated entirely by Claude Opus 4.7, and all gold labels come from the same model. This creates a fundamental circularity: "model X agrees with gold" conflates with "model X agrees with Opus's preferences." The authors acknowledge this (caveat (i) in §8) but it substantially undermines the interpretive value of absolute accuracy numbers.
The taxonomy-aware vs. taxonomy-blind comparison is the most interesting methodological contribution. The finding that removing the explicit label menu drops trajectory accuracy by 14–40 pp to a compressed 0.54–0.62 floor is striking and well-documented. However, several confounds are acknowledged: the blind-to-closed-set mapping uses a deterministic substring rule with a Haiku fallback estimated to misroute ~3–5% of outputs, potentially inflating the drop. The cross-axis incoherence finding (Fig. 4) is more robust since it relies on aware-mode metrics only.
The sample sizes are adequate for the control split (684) and trajectory split (400) but thin in parts of the security split (n=3 refuse-gold, n=7 stop-gold), limiting reliability of security findings.
3. Potential Impact
Positive aspects of potential impact:
Limitations on impact:
4. Timeliness & Relevance
The paper is highly timely. The rapid proliferation of agent benchmarks (OSWorld, WebArena, τ-bench, AgentDojo, etc.) and the growing gap between benchmark scores and deployment readiness create genuine need for unifying evaluation frameworks. The observation that OSWorld now has above-human submissions while agents still fail on basic control decisions is a compelling motivating example.
The inclusion of MCP security concerns and tool-poisoning attacks is forward-looking and addresses an emerging deployment risk. The efficiency gap identified (no benchmark with strong coverage) points to a genuine blind spot.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Overall Assessment
AgentAtlas makes a reasonable contribution to the growing literature on multi-axis agent evaluation. Its primary value lies in the benchmark coverage audit and the taxonomy-aware vs. blind methodology, both of which highlight real measurement issues. However, the paper falls short of providing a validated, releasable evaluation resource, and its taxonomies are modest extensions of existing work. The empirical findings, while suggestive, are undermined by the single-generator design and absence of human calibration. This is a useful position-and-framework paper, but its impact will depend on whether the community adopts the vocabulary and whether a properly validated benchmark follows.
Generated May 21, 2026
Comparison History (21)
Paper 2 introduces a novel task (ToM-PD), a large-scale annotated dataset (ToM-BPD), and a concrete reasoning framework (TTBYS) that demonstrates strong empirical results—outperforming GPT-5 on key metrics. It contributes reusable resources (dataset, framework) with clear applications in persuasive dialogue, negotiation, and human-AI interaction. Paper 1 provides useful taxonomies and a measurement protocol for agent evaluation but explicitly disclaims being a benchmark release and presents only a small demonstration study. Paper 2's combination of dataset, method, and strong results gives it broader and more immediate scientific impact.
Paper 2 addresses a critical bottleneck in AI research: evaluating LLM agents beyond simplistic outcome leaderboards. By introducing comprehensive taxonomies for control decisions and trajectory failures, it provides a foundational methodology applicable across all agentic AI research. While Paper 1 offers valuable multi-disciplinary scientific benchmarks, Paper 2's framework has a broader impact on the fundamental development, evaluation, and deployment of AI systems, giving it a higher potential for widespread scientific influence.
Paper 2 presents a novel and concrete technical contribution—asymmetric-view policy optimization with a cognitive user simulator for proactive dialogue—that introduces new training methodologies (privileged self-distillation, state-transition refinement) with clear real-world applications in sales and persuasion. It advances the frontier of RL-based LLM fine-tuning with a principled approach to a well-defined problem. Paper 1, while valuable as a meta-evaluation framework with useful taxonomies, is explicitly positioned as a 'measurement-protocol demonstration' rather than a benchmark release, limiting its immediate actionable impact. Paper 2's methodological innovations are more likely to inspire follow-up work.
Paper 1 presents a concrete technical contribution (Meta-Soft) addressing a critical bottleneck in LLM deployment—KV cache compression for long contexts—with a novel dynamic meta-token framework and attention-flow integration mechanism. This has immediate practical applicability to improving LLM efficiency, a high-demand area. Paper 2 proposes evaluation taxonomies for LLM agents, which is useful but more incremental; it explicitly states it is a 'measurement-protocol demonstration, not a benchmark release,' limiting its immediate impact. Paper 1's methodological innovation and broad applicability to the efficiency problem give it higher potential impact.
AgentAtlas addresses a broader, more fundamental problem—how to evaluate LLM agents across diverse environments—with a generalizable taxonomy and methodology applicable across many domains and benchmarks. Its multi-dimensional evaluation framework (control-decision taxonomy, failure taxonomy, taxonomy-aware vs. blind methodology, and benchmark-coverage audit) offers tools the entire agent evaluation community can adopt. While WorkstreamBench is rigorous and practically relevant, its scope is narrower (finance spreadsheets), limiting its cross-field impact. AgentAtlas's findings about prompt-dependency and the absence of a single dominant model have broader implications for agent deployment and evaluation methodology.
Paper 1 investigates a profound, cross-disciplinary question—whether AI can predict future scientific progress—and introduces a large-scale benchmark (CUSP). Its findings on AI's limitations in scientific forecasting have broader implications for meta-science and AI-driven discovery compared to Paper 2's narrower, methodological focus on LLM agent evaluation taxonomies.
Paper 1 likely has higher impact due to a clearer, deployable artifact: a unified, released benchmark (360 tasks) plus an agentic evaluator with strong validation (0.79 Spearman vs experts) and broad experimental coverage (humans vs MLLMs, multiple T2I backends). This supports reproducible progress and immediate real-world application in T2I prompting. Paper 2 offers valuable taxonomies and evaluation protocol insights for agents, but is positioned as a methodological demonstration without releasing a benchmark/dataset, which may reduce uptake and downstream standardization impact despite high relevance.
SimGym presents a novel, end-to-end framework addressing a significant practical problem in e-commerce A/B testing, combining VLM agents with real traffic data and validated against real-world outcomes (77% directional alignment). It offers clear real-world applications—reducing A/B test cycles from weeks to under an hour—with strong methodological rigor through empirical validation on a major platform. Paper 2 (AgentAtlas) contributes useful taxonomies and evaluation methodology for LLM agents, but is explicitly positioned as a 'measurement-protocol demonstration, not a benchmark release,' limiting its immediate impact. SimGym's concrete, validated system with direct industry applicability gives it higher potential impact.
Paper 1 provides novel empirical insights into a fundamental question about LLM-based code optimization: whether LLMs truly leverage feedback and search or primarily rely on pretrained priors. Its controlled experiments reveal surprising findings (greedy optimization behavior, insensitivity to input specifications, degradation with low-density languages) that have broad implications for the growing field of LLM-driven optimization and discovery systems. Paper 2 proposes useful evaluation taxonomies for LLM agents but is more incremental in nature—extending existing diagnostic frameworks—and explicitly positions itself as a methodology demonstration rather than a benchmark contribution.
Paper 2 (PRISM) is likely to have higher scientific impact because it delivers a large, reusable benchmark (10,372 pairs, bilingual) with concrete, automatable metrics and a clear empirical finding (Execution–Spatial Gap) that can drive model and method development. Its applications span programmatic video generation, spatial-temporal reasoning, code generation, and multimodal evaluation, making it broadly useful and timely. Paper 1 offers valuable evaluation taxonomies and insights, but is positioned as a protocol demonstration with a small-scale run and no benchmark release, limiting immediate adoption and downstream impact.
Paper 1 addresses a critical bottleneck (hallucinations) in deploying VLMs for robotics using an innovative pseudocode-guided approach. By achieving SOTA results that surpass GPT-4V, it offers immediate, measurable utility and strong real-world applicability. While Paper 2 provides a valuable evaluation taxonomy, its self-admitted status as a 'demonstration' rather than a full benchmark release may limit its immediate widespread adoption compared to Paper 1's concrete algorithmic advancements.
Paper 2 addresses a critical and highly timely bottleneck in AI: the evaluation of LLM agents beyond simplistic outcome leaderboards. By proposing comprehensive taxonomies for control decisions and trajectory failures, it provides a much-needed methodological foundation for a rapidly growing field. While Paper 1 offers a strong, novel approach to Vehicle Routing Problems, its impact is largely confined to operations research and combinatorial optimization. Paper 2's focus on deployable AI agents gives it a significantly broader potential impact across multiple domains in artificial intelligence and software engineering.
Paper 1 addresses a fundamental and broadly applicable problem in LLM agent evaluation—fragmented benchmarking—by proposing comprehensive taxonomies and a methodology applicable across the entire agent ecosystem. Its contributions (control-decision taxonomy, failure taxonomy, taxonomy-aware vs. blind methodology, benchmark audit) have potential to influence how the rapidly growing field of LLM agents is evaluated. Paper 2 solves a narrower domain-specific problem (EV battery fault diagnosis using LLMs), which, while practically useful, has more limited breadth of impact and less methodological novelty for the broader AI research community.
Paper 1 presents a highly innovative approach to test-time compute scaling, allowing a 7M parameter model to significantly outperform frontier LLMs on complex reasoning tasks at a fraction of the cost. This breakthrough in efficient, stochastic recursive reasoning offers immense real-world potential for deploying capable models in resource-constrained environments, likely driving broader algorithmic impact than the evaluation taxonomies proposed in Paper 2.
Paper 1 presents a highly innovative algorithmic advancement (Probabilistic TRM) that enables a 7M-parameter model to outperform frontier LLMs on complex reasoning tasks through stochastic test-time compute scaling. Its potential to drastically reduce compute costs while improving reasoning capabilities gives it broader, more transformative real-world applications and higher scientific impact than the evaluation taxonomy proposed in Paper 2.
Paper 2 has higher likely impact because it introduces a scalable, actionable method for corpus-level agent trace diagnostics with demonstrated downstream improvements (e.g., 30.4pp scaffold gains) and an end-to-end system practitioners can apply in production. Its methodology connects diagnostics to measurable performance changes and supports long, real-world traces, increasing applicability and timeliness. Paper 1 offers valuable evaluation taxonomies and an important critique of leaderboards, but it is positioned mainly as a measurement-protocol demonstration on a small fixed model set and may translate less directly into deployed gains.
Paper 2 is likely higher impact due to broader, more generalizable contributions: unified taxonomies for agent control decisions and failures, a methodology to disentangle real capability from prompt-provided supervision, and a cross-benchmark coverage audit. These elements can standardize evaluation across many agent domains and influence how the community reports results beyond single-number leaderboards—highly timely for 2024–2026. Paper 1 is valuable and applied, but its impact is more tool-/workflow-specific to trace diagnostics and may be narrower in cross-field standardization and evaluation protocol influence.
Paper 2 likely has higher impact: it addresses a widely shared, timely bottleneck—how to evaluate deployable LLM agents beyond single-score leaderboards—via concrete taxonomies, prompt-supervision controls, and benchmark-coverage auditing that can be adopted across many domains and benchmarks. Its contribution is broadly applicable to agent research, safety, and product evaluation. Paper 1 is novel and practically valuable, but its scope is narrower (data-system composition) and evidence is limited to a proof-of-life workload, making near-term impact more specialized despite strong applied relevance.
Paper 2 likely has higher scientific impact: it proposes broadly applicable, timely evaluation frameworks (decision taxonomy, failure taxonomy, prompt-supervision sensitivity, and benchmark coverage auditing) that can standardize how agent behavior is measured across many tasks and domains. This can influence subsequent research agendas and benchmarking practices across academia/industry. Paper 1 is highly practical and valuable for enterprise deployment, but it is more of a systems/demo contribution with impact concentrated in governance engineering rather than field-wide measurement methodology. Paper 2’s rigor and cross-field applicability suggest wider uptake.
Paper 1 addresses a concrete, timely gap in LLM agent evaluation with novel taxonomies and a rigorous methodology demonstrating that prompt supervision inflates apparent model capability. Its contributions (control-decision taxonomy, trajectory-failure taxonomy, taxonomy-aware vs. blind comparison, benchmark-coverage audit) are immediately actionable by the large and growing agent evaluation community. Paper 2 presents a high-level vision for AI-native 6G without concrete technical contributions or empirical validation—it is speculative and lacks methodological rigor. Paper 1's empirical findings and reusable evaluation framework give it broader and more immediate scientific impact.