Agents' Last Exam
Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu
Abstract
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long-horizon, economically valuable, real-world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 subfields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is 2.6%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP-relevant impact.
AI Impact Assessments
(1 models)Scientific Impact Assessment: Agents' Last Exam (ALE)
1. Core Contribution
ALE introduces a benchmark of 1,490 task instances across 55 subfields and 13 industry clusters, designed to evaluate Generalist Computer-Use Agents (GCUAs) on long-horizon, economically valuable professional workflows. The key distinguishing features are: (a) tasks sourced from 250+ domain experts who contribute actual professional projects they have completed, (b) grounding in the O*NET/SOC 2018 occupational taxonomy for systematic industry coverage, (c) deterministic automated verification replacing human judges, and (d) a living benchmark design with rolling task rotation to combat contamination.
The paper's central thesis—that the gap between benchmark success and economic impact is fundamentally an evaluation problem—is compelling and well-articulated. The benchmark fills a genuine void: prior work either covers narrow domains (SWE-bench for software, OSWorld for GUI), relies on human evaluation (GDPval, RLI), or tests knowledge rather than execution capability (MMLU, HLE).
2. Methodological Rigor
Task construction pipeline. The five-gate protocol (expert sourcing → submission → first-pass review → implementation → QC) is well-designed and mirrors academic peer review standards. The conference-style review decisions and iterative feedback loops add credibility. The provenance breakdown (Figure 5) showing 960 external and 530 commissioned tasks with review distributions is transparent.
Evaluation architecture. The decoupled three-component design (task specification, agent, environment) is clean and extensible. The evaluation mode taxonomy is thoughtfully constructed: 93.2% of tasks use code-based deterministic scoring, with LLM judges reserved only for inherently perceptual outputs. The gate-and-score pattern prevents reward hacking effectively.
Experimental design. The paper evaluates a comprehensive matrix of 14+ foundation models × 6+ harnesses, providing both model-fixed and harness-fixed comparisons. The finding that model choice accounts for ~3× the performance spread of harness choice (18 pp vs. 5-6 pp) is methodologically valuable. The three-tier difficulty structure (Near-Term, Full-Spectrum, Last-Exam) is pragmatic given the high per-task evaluation cost ($3-10).
Potential concerns. The public/private split (150/1,340) means community-accessible evaluation covers only ~10% of tasks—a necessary contamination defense but one that limits independent verification. The paper acknowledges but doesn't fully resolve the representativeness question, though the r=0.89 correlation (Figure 11) is reassuring. Some variance estimates come from only three runs due to compute constraints. The taxonomy derivation using "LLM-assisted" classification of occupations into subdomains introduces a potential circularity that could have been more carefully validated.
3. Potential Impact
Near-term influence. ALE could become the primary benchmark for evaluating agent systems targeting professional deployment—analogous to what SWE-bench became for coding agents. The 2.6% average pass rate on the hardest tier ensures years of headroom. The explicit connection to economic value (GDP-relevant impact) reframes the evaluation conversation in a way that resonates with both researchers and industry stakeholders.
Broader implications. The GCUA agent taxonomy (Brain, Eyes, Body, Hands, Feet) provides a useful conceptual framework that clarifies the gap between CLI-only and GUI-only agents. The failure analysis finding that ~78% of failures stem from understanding and approach errors (not execution bugs) has important implications for model development priorities.
Industry adoption. The benchmark's grounding in real professional software (SolidWorks, DaVinci Resolve, Moldex3D, Blender, MicroDicom) makes results directly interpretable for industry stakeholders evaluating AI adoption decisions. The task cards in the appendix are exemplary in making the benchmark concrete.
4. Timeliness & Relevance
This paper addresses perhaps the most pressing question in AI: why haven't impressive benchmark gains translated into economic transformation? The timing is excellent—agent frameworks (Claude Code, Codex, etc.) have proliferated rapidly, but evaluation infrastructure has lagged. The living benchmark design with rolling task replacement acknowledges the contamination arms race that has plagued static benchmarks.
The explicit framing around economic impact and the SOC/O*NET grounding is novel and timely, connecting AI evaluation to labor economics in a way that could influence policy discussions about AI automation.
5. Strengths & Limitations
Key strengths:
Notable limitations:
Overall Assessment
ALE represents a significant contribution to AI evaluation methodology. It is the most comprehensive attempt to date to benchmark AI agents on authentic professional workflows with automated verification. The scale of expert involvement, systematic taxonomy grounding, and careful evaluation design set a new standard for agent benchmarks. While limitations exist in scope and verification completeness, the paper's framing, execution, and documentation are strong. The benchmark's lasting impact will depend on community adoption and whether the living benchmark promise materializes.
Generated Jun 5, 2026
Comparison History (25)
Paper 2 introduces a comprehensive, expertly-crafted benchmark that addresses a universal bottleneck in AI: the gap between standard benchmark performance and real-world economic utility. By providing a standardized evaluation framework for long-horizon agentic tasks, it is likely to become a foundational metric driving both research and industry applications. While Paper 1 addresses a critical AI safety concern regarding internalized reasoning, Paper 2's broad, field-defining utility gives it a wider potential scientific and economic impact.
OpenSkill introduces a novel framework for open-world self-evolution of LLM agents without target-task supervision—a fundamental methodological contribution addressing a core limitation of existing approaches. It proposes concrete mechanisms (bootstrapping verification from open-world resources, self-built virtual tasks) that advance the field's understanding of autonomous agent learning. While ALE is a valuable benchmark contribution with practical relevance, benchmarks typically have more transient impact compared to new methodological frameworks. OpenSkill's transferable skills and supervision-free learning paradigm have broader potential to influence future agent development research.
Agents' Last Exam (ALE) addresses a fundamental gap between AI benchmark performance and real-world economic impact, introducing a comprehensive, living benchmark with 250+ industry experts across 55 subfields. Its breadth of impact spans virtually all non-physical industries, and it tackles the timely, high-stakes question of AI deployment relevance. GITCO, while methodologically sound, is narrowly focused on improving time series foundation models via inference-time context optimization—a useful but incremental contribution within a specific subfield. ALE's potential to reshape how AI systems are evaluated for economic value gives it substantially broader impact.
Paper 2 (Agents' Last Exam) has higher potential impact due to its broad, timely relevance to evaluating economically meaningful agent performance, a major bottleneck in applied AI. Its large-scale, expert-curated, verifiable, long-horizon benchmark could shape research directions across LLM agents, evaluation, alignment, and human-computer interaction, and drive real-world deployment standards. Paper 1 is a solid methodological contribution to multimodal time-series robustness, but its impact is narrower to specific modalities/tasks and likely incremental relative to the sweeping cross-domain influence a widely adopted benchmark can have.
Agents' Last Exam (ALE) addresses a critical gap between AI benchmark performance and real-world economic impact, introducing a large-scale, living benchmark with 1K+ tasks across 55 subfields developed with 250+ industry experts. Its breadth of impact across industries, timeliness given rapid AI agent development, and practical relevance to deployment decisions give it substantially higher potential impact. Paper 2 makes a narrower contribution applying affinity-based RL to a board game environment, extending prior work incrementally with limited real-world applicability and a smaller scope of influence.
Agents' Last Exam (ALE) introduces a novel, large-scale benchmark addressing a fundamental gap between AI benchmark performance and real-world economic impact. Its collaboration with 250+ industry experts, comprehensive taxonomy covering 55 subfields, and living benchmark design give it broad cross-field relevance and lasting utility. Paper 2, while useful, is primarily an empirical comparison of existing LLMs on existing datasets for a narrow task (Lean 4 proof generation), offering incremental practical guidance but limited novelty or methodological contribution. ALE's scope, novelty, and potential to reshape AI evaluation give it substantially higher impact potential.
Agents' Last Exam (ALE) has higher potential scientific impact due to its breadth and timeliness. It introduces a comprehensive, living benchmark spanning 55 subfields and 13 industry clusters with 1K+ tasks, developed with 250+ industry experts, addressing a fundamental gap between AI benchmark performance and real-world economic deployment. This addresses a critical, widely-recognized problem in AI evaluation and could influence the entire field's research direction. Brick-Composer, while innovative in its specific domain of physical assembly with MLLMs, addresses a narrower problem with more limited cross-field applicability.
Paper 1 likely has higher scientific impact because it introduces a large, extensible, industry-collaborative benchmark targeting long-horizon, economically relevant agent workflows with verifiable outcomes—addressing a central, timely bottleneck in AI evaluation and deployment. As a “living” benchmark spanning many industries/tasks, it can become shared infrastructure used across labs and fields, shaping research directions and enabling standardized progress measurement. Paper 2 is methodologically strong and highly relevant to AI safety, but its scope is narrower (coding sabotage/human oversight) and may influence a more specialized community than a broadly adopted benchmark.
Paper 1 addresses a critical bottleneck in AI: the disconnect between benchmark success and real-world economic utility. By collaborating with 250+ experts to create a living evaluation framework of 1,000+ tasks, it provides a highly rigorous, timely tool that will drive AI agent research across multiple industries. In contrast, Paper 2 is a perspective paper on hybrid models in neurology. While medically significant, Paper 1's massive empirical effort, cross-disciplinary applicability, and direct alignment with current AI development hurdles give it a significantly higher potential for broad, foundational scientific impact.
Agents' Last Exam (ALE) addresses a fundamental gap between AI benchmark performance and real-world economic impact, offering a large-scale, living benchmark spanning 55 subfields and 13 industry clusters with 1K+ tasks developed with 250+ industry experts. Its breadth, focus on economically meaningful evaluation, and finding that top models achieve only 2.6% on hard tasks make it highly impactful for the AI agents community. DragOn addresses an important but narrower problem (drag-based GUI interactions), contributing a useful dataset but with more limited scope and breadth of impact across fields.
Paper 2 (Agents' Last Exam) likely has higher scientific impact due to broad, timely relevance: it reframes evaluation toward long-horizon, economically meaningful, verifiable tasks across 13 industry clusters and 1K+ tasks, developed with 250+ experts and designed to evolve as a living benchmark. This can influence research directions across agent design, evaluation methodology, robustness, and alignment, and provide a widely adopted standard. Paper 1 is novel and useful for LLM serving efficiency, but its impact is more specialized to systems infrastructure and may affect a narrower slice of the community.
Agents' Last Exam (ALE) addresses a critical and timely gap between AI benchmark performance and real-world economic impact, with a large-scale collaborative effort (250+ experts), rigorous taxonomy aligned with federal occupational standards, and a living benchmark design. It has broad practical relevance across industries and is likely to influence how AI agents are evaluated and deployed. Paper 2 proposes a theoretical motivational architecture for conversational AGI, but remains largely conceptual, lacks empirical validation, and targets a narrower, more speculative research niche with less immediate practical impact.
Paper 1 makes fundamental theoretical contributions to reinforcement learning by formally defining model exploitation, proving its near-inevitability, and establishing connections to reward hacking. These results address a core challenge in AI safety—safe planning with imperfect world models—with broad implications for the field. Paper 2 introduces a useful benchmark but is incremental in nature; benchmarks tend to have shorter shelf lives and narrower theoretical contributions. Paper 1's formal framework and impossibility results are likely to have lasting influence on RL theory and AI safety research.
Paper 1 (Agents' Last Exam) likely has higher impact due to its broad, GDP-relevant scope and direct alignment with real-world deployment gaps. Its large taxonomy (1K+ tasks across 13 industry clusters), collaboration with 250+ experts, and “living benchmark” design can shape evaluation norms across many applied domains, influencing both academia and industry. Paper 2 is novel and timely for autonomous agent development and safety, but is narrower (five domains) and more specialized; its impact may concentrate within agent research rather than across professional workflows.
Agents' Last Exam (ALE) introduces a large-scale, living benchmark covering 1K+ economically valuable tasks across 55 subfields, developed with 250+ industry experts. It addresses a fundamental evaluation gap between benchmark performance and real-world deployment. Its breadth of impact across fields, timeliness (agents are rapidly advancing but lack rigorous real-world benchmarks), and potential to become a standard evaluation instrument give it higher scientific impact. While RHO presents a clever self-supervised optimization method with strong results, it is more narrowly focused on agent harness improvement methodology.
Agents' Last Exam (ALE) introduces a large-scale, living benchmark for evaluating AI agents on economically valuable real-world tasks, developed with 250+ industry experts across 55 subfields. It addresses a fundamental gap between benchmark performance and real-world deployment—a highly timely and broadly impactful problem. Paper 2 presents GuardNet, a useful but incremental contribution to LLM safety using shallow neural network ensembles, with modest results that don't surpass larger models. ALE's breadth of impact, novelty in evaluation methodology, and relevance to the entire AI deployment ecosystem give it substantially higher potential scientific impact.
Paper 2 has higher potential impact due to its massive scale and direct focus on closing the gap between AI benchmarks and real-world economic value. While Paper 1 offers a valuable technical benchmark for memory, Paper 2 involves over 250 industry experts, covers 55 subfields across diverse industries, and directly addresses a critical bottleneck in AI deployment. This gives it broader cross-disciplinary relevance and immediate real-world applicability.
Agents' Last Exam (ALE) addresses a fundamental gap between AI benchmark performance and real-world economic impact, introducing a large-scale, living benchmark with 1K+ tasks across 55 subfields developed with 250+ industry experts. Its breadth of impact across professional domains, timeliness given the agent deployment wave, and practical relevance to GDP-level economic outcomes give it higher potential impact. LC-ERD, while technically interesting in improving LLM reasoning via reward decomposition, addresses a narrower methodological problem. ALE has potential to reshape how the entire field evaluates and deploys AI agents.
Paper 1 provides novel, granular facility-level data on a critically important and timely topic—the environmental footprint of US hyperscale data centers driven by AI growth. Its methodology linking 403 specific facilities to EPA eGRID data yields a surprising finding (48% higher carbon intensity than grid average) with immediate policy relevance for energy planning, climate policy, and corporate sustainability. Paper 2 introduces a useful AI benchmark but enters a crowded benchmark landscape, and its impact depends on adoption. Paper 1 addresses a concrete, urgent societal concern with quantitative evidence that will be widely cited across energy, environmental, and policy fields.
Paper 2 (Agents’ Last Exam) likely has higher scientific impact due to its broad, timely contribution: a large-scale, industry-grounded benchmark for long-horizon agentic tasks with verifiable outcomes, built with extensive expert input and designed to evolve. Such evaluation infrastructure can reshape research agendas across LLM agents, alignment, tooling, and economics of deployment. Paper 1 is a solid systems contribution for RAG serving efficiency with clear practical value, but its impact is narrower (RAG/KV-cache optimization) and more incremental relative to the field-wide effect a widely adopted benchmark can create.