Open-World Evaluations for Measuring Frontier AI Capabilities
Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, J. J. Allaire, Rishi Bommasani, Harry Coppock, Magda Dubois
Abstract
Benchmark-based evaluation remains important for tracking frontier AI progress. But it can both overstate and understate deployed capability because it privileges tasks that can be precisely specified, automatically graded, easy to optimize for, and run with low budgets and short time horizons. We advocate for a complementary class of evaluations, which we term open-world evaluations: long-horizon, messy, real-world tasks assessed through small-sample qualitative analysis rather than benchmark-scale automation. In this paper we survey recent open-world evaluations, identify their strengths and limitations, and introduce CRUX (Collaborative Research for Updating AI eXpectations), a project for conducting such evaluations regularly. As a first instance, we task an AI agent with developing and publishing a simple iOS application to the Apple App Store. The agent completed the task with only a single avoidable manual intervention, suggesting that open-world evaluations can provide early warning of capabilities that may soon become widespread. We conclude with recommendations for designing and reporting open-world evals.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Open-World Evaluations for Measuring Frontier AI Capabilities
1. Core Contribution
The paper makes two intertwined contributions: a conceptual framework for "open-world evaluations" — long-horizon, real-world tasks assessed through qualitative log analysis rather than benchmark-scale automation — and a concrete instantiation (CRUX #1) in which an AI agent autonomously develops and publishes an iOS app to the Apple App Store. The conceptual contribution is to name, taxonomize, and legitimize an evaluation methodology that has been emerging organically across labs (Anthropic's C compiler, Project Vend, Claude Plays Pokemon, etc.) but lacked a unifying framework or shared methodological standards.
The five-dimensional taxonomy (openness, complexity/duration, number of tasks, human intervention, method of evaluation) provides a useful vocabulary for positioning evaluations along a gradient from simple Q&A benchmarks to messy real-world deployments. The six methodological recommendations (specify construct, document interventions, analyze/release logs, real-time monitoring, dry runs, cost reporting) are practical and well-motivated.
2. Methodological Rigor
The paper is transparent about its own methodological limitations, which is both a strength and a challenge. The CRUX #1 experiment has n=1, is not reproducible in the traditional sense, and the authors explicitly acknowledge this. The honesty is commendable, but it means the empirical contribution is thin by conventional standards.
The experimental design is reasonable for what it is: dry runs were conducted, interventions were classified and documented, logs are released, and costs are reported — the paper practices what it preaches. The distinction between avoidable and unavoidable interventions (e.g., Apple-mandated 2FA vs. the agent losing track of credentials) is clearly drawn. However, some analytical judgments are acknowledged as subjective (e.g., classifying the OpenClaw daemon crash as infrastructure rather than agent failure).
The survey of prior open-world evaluations (Section 2.3, Appendix C, Table 1) is useful but necessarily incomplete and somewhat informal. The paper does not offer quantitative meta-analysis — it couldn't, given the heterogeneity of the evaluations surveyed.
One concern: the paper's core claim that open-world evaluations provide "early warning of capabilities" rests on a single demonstration. The iOS app task was deliberately chosen to be simple (a breathing exercise app), and the non-coding aspects (navigating Apple's submission process) are well-documented online. Whether this generalizes to genuinely novel or harder deployment scenarios remains untested.
3. Potential Impact
Methodological influence. The paper could meaningfully shape how the AI evaluation community thinks about complementing benchmarks. The gradient framework (Figure 2) is pedagogically effective and could become a standard reference. If CRUX becomes an ongoing project with regular iterations, it could establish norms for a currently ad hoc practice.
Policy relevance. The framing around early warning for policymakers and app store operators is timely. The responsible disclosure to Apple and the observation about potential agent-driven spam submissions are practically relevant. The paper explicitly connects evaluation methodology to governance decisions, which broadens its audience.
For AI developers. The recommendations around log analysis, intervention documentation, and cost reporting could improve the quality of capability claims from labs. Currently, many open-world evaluations (as the paper documents) lack these basic methodological commitments.
Limitations of impact. The paper does not introduce new technical methods for building or improving AI agents. It is primarily a position/methods paper with a single case study. Its influence will depend heavily on whether CRUX continues and whether the community adopts its recommendations.
4. Timeliness & Relevance
The paper is exceptionally timely. The rapid saturation of traditional benchmarks (well-illustrated by Figure 1), the proliferation of ad hoc open-world experiments from labs, and the growing policy stakes of AI capability assessment all create demand for the kind of methodological reflection this paper provides. The observation that Anthropic's Mythos Preview system card acknowledges saturating "most concrete, objectively-scored evaluations" underscores the urgency.
The paper addresses a genuine methodological vacuum: there was no systematic framework or set of norms for the growing class of real-world agent evaluations, and this paper provides one.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The paper's positioning is strategic: by framing open-world evaluations as complementary rather than competitive with benchmarks, it avoids antagonizing the benchmarking community while still making a case for methodological pluralism. The cost analysis (975 for monitoring) is a useful practical insight. The discussion of evaluation awareness (Section 3.1.1) is thoughtful and reveals a genuine dilemma for capability evaluations.
The paper reads more as a manifesto and inaugural case study for an ongoing research program than as a self-contained contribution. Its ultimate impact will depend on execution of future CRUX iterations and community adoption.
Generated May 21, 2026
Comparison History (20)
Paper 2 has higher potential impact because it reframes how frontier AI capability is measured, proposing a broadly applicable evaluation paradigm (open-world evals) that can influence research, policy, and deployment practices across many subfields. Its real-world, long-horizon tasks address a timely gap in benchmark-centric evaluation and can become a standard complement to existing methods (via CRUX). Paper 1 is a solid, method-level contribution to agentic test-time scaling with measurable gains, but its impact is narrower to LLM-agent workflow optimization and may be more incremental.
Paper 1 has higher potential impact due to its broader, timelier framing of frontier AI evaluation: it challenges benchmark-centric paradigms and proposes open-world evaluations applicable across domains (agents, safety, governance, deployment readiness). The CRUX proposal and real deployment case study (shipping an iOS app) are likely to influence evaluation practice and policy beyond a single subfield. Paper 2 is methodologically strong with a valuable dataset/metrics for grounded personality perception in MLLMs, but its scope is narrower (social cognition/personality) and impacts fewer adjacent fields.
While Paper 1 provides a rigorous and valuable benchmark for a specific domain (personality perception), Paper 2 addresses a fundamental limitation in how frontier AI capabilities are currently measured across all domains. By advocating for and demonstrating 'open-world evaluations,' Paper 2 has a broader potential impact on AI safety, policy, and general capability tracking, shaping future methodologies for evaluating autonomous agents in real-world scenarios.
Paper 2 likely has higher impact: it contributes a large, standardized, multi-turn, long-horizon benchmark (502 solvable instances, 102 targets) with a public leaderboard, enabling reproducible comparison and sustained community progress in an economically and societally critical domain (drug discovery). Its methodological rigor and concrete task specification make it broadly usable for model training/evaluation and tool-augmented agent research. Paper 1 is timely and conceptually important, but its evidence base is smaller-sample and more qualitative, with less immediately scalable infrastructure for broad adoption.
Paper 1 introduces a highly practical and novel methodological framework for adapting LLM agents without retraining. Its rigorous empirical validation across 18 models and demonstration of high transferability and significant performance gains (88.5% average improvement) offer immediate, broad utility for agent development. Paper 2, while conceptually valuable for AI evaluation and policy, acts more as a position paper with a single qualitative case study, lacking the algorithmic rigor and broad, direct applicability of Paper 1.
Paper 1 introduces a broadly applicable evaluation paradigm ('open-world evaluations') for frontier AI systems, addressing a fundamental gap in how the AI community measures capabilities. Its conceptual contribution—complementing benchmarks with long-horizon, real-world tasks—has wide relevance across all of AI safety, policy, and capability research. The CRUX project provides an institutional framework for ongoing impact. Paper 2, while technically strong, addresses a narrower domain (EDA/Verilog agents) with a specialized test-time scaling method. Paper 1's breadth of influence across AI evaluation, governance, and multiple application domains gives it higher potential impact.
Paper 1 proposes a concrete, novel system (a persistent, budgeted “context map” cache with explicit modules) and reports substantial efficiency and accuracy gains across tasks, models, and even a production coding agent—suggesting strong real-world applicability and methodological rigor with measurable benchmarks. Its ideas could broadly influence long-context agent design, memory/caching strategies, and tooling for recurring corpora. Paper 2 is timely and valuable conceptually for evaluation practice, but is largely a survey/framework with limited empirical depth (small-sample qualitative case study), making near-term scientific impact less certain.
Paper 2 offers a higher potential scientific impact due to its fundamental algorithmic innovation (MAP framework) and broad applicability across interactive LLM agents. While Paper 1 raises a critical point about AI evaluation, its approach relies on qualitative, small-sample case studies. In contrast, Paper 2 provides a rigorous methodology, introduces a new dataset (MAP-2K), and demonstrates generalizable empirical performance gains on complex benchmarks like ARC-AGI-3. The release of a concrete, plug-and-play paradigm and training dataset will likely spur more direct follow-up technical research and downstream applications.
Paper 2 addresses a fundamental challenge in AI evaluation methodology that applies broadly across the entire frontier AI ecosystem, not just one domain. Its proposal of 'open-world evaluations' as a complement to benchmarks is timely given rapid AI progress, has broad cross-field relevance, and introduces a reusable framework (CRUX) for ongoing capability assessment. While Paper 1 makes a solid technical contribution to autonomous driving testing, its scope is narrower. Paper 2's potential to reshape how the community evaluates and anticipates AI capabilities gives it higher estimated impact.
Paper 1 likely has higher scientific impact due to a concrete, novel systems contribution (treating GPU power caps as a control variable jointly optimized with batching) with demonstrated, reproducible gains (up to 26.3% energy efficiency, fewer QoS violations) and immediate applicability to widespread LLM serving stacks (integration into vLLM, no retraining). It is timely given datacenter energy constraints and could influence both systems research and production deployments. Paper 2 is important conceptually but is more of a survey/proposal with limited methodological rigor and harder-to-standardize, lower-scalability evaluation evidence.