STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios
Sirui Liang, Bohan Yu, Peiyu Wang, Shiguang Guo, Wenxing Hu, Pengfei Cao, Jian Zhao, Cao Liu
Abstract
Large language models are increasingly used to power personal agents for everyday applications, but evaluating these agents remains a challenge. Existing benchmarks still rely on sandboxed artifacts, static task design, and coarse scoring, which hinder scalability and limit progress toward reliable personal-agent evaluation. This paper introduces STAGE-Claw, an automated framework for building and evaluating realistic personal-agent scenarios in state-based personal-computing environments. Given a task hint, STAGE-Claw automatically creates and validates a realistic benchmark task with its environment, task prompts, ground truth, and related verification programs. Agents are then evaluated in realistic operating environments, where performance is measured by the correctness of the final system state rather than only the textual response. Using STAGE-Claw, this paper creates a benchmark with 40 challenging real scenario agent tasks, evaluates 11 frontier models, and analyzes their task scores, costs, tool-call reliability, and common failure patterns. Overall, STAGE-Claw offers a scalable, state-based way to evaluate agents in realistic user scenarios.
AI Impact Assessments
(1 models)Scientific Impact Assessment: STAGE-Claw
1. Core Contribution
STAGE-Claw proposes an automated framework for constructing and evaluating LLM-based personal agent benchmarks grounded in realistic, state-based computing environments. The key idea is that agent evaluation should verify whether persistent system states (calendar events, emails, notes, files) have been correctly modified, rather than checking only textual outputs or sandboxed artifacts. The framework has four stages: (1) automated benchmark authoring from task hints, (2) validation of task reproducibility and difficulty, (3) agent execution in real environments, and (4) state-based evaluation via executable verifiers. Using STAGE-Claw, the authors create 40 tasks and evaluate 11 frontier models.
The three claimed advances over prior work are: replacing sandboxed artifact evaluation with real tool-state verification, automating benchmark construction (reducing manual curation), and providing process-aware diagnostics for failure analysis. The state-based evaluation concept is the most distinctive contribution—Table 3 shows that virtual (text-output-only) evaluation overestimates scores by 5–7 points compared to state-based evaluation, with execution failures and real-state gaps accounting for the discrepancy.
2. Methodological Rigor
The methodology has several concerning limitations:
Small scale with single-run evaluation. Only 40 tasks are created, and each model is evaluated on a single valid run per task. The authors acknowledge this limitation, but it severely constrains the statistical reliability of model comparisons. The limited three-run diagnostic (Appendix A.6) on only 3 of 11 models reveals substantial score variability across runs (visible in Figure 5), undermining confidence in the reported rankings.
Authoring agent circularity. Claude-Sonnet-4.6 is used as the benchmark authoring agent, and Claude models are among the evaluated systems. While the authoring agent doesn't solve tasks, it designs them—potential biases in task formulation toward Claude's strengths cannot be ruled out.
Validation concerns. The checker agent validates tasks automatically, but the quality criteria are somewhat vague. The 80-point threshold for acceptance and the average of 2.67 repair iterations suggest non-trivial failure rates, yet the paper doesn't analyze what properties failed tasks exhibited or whether the repair process introduces systematic biases.
Cost prohibitive. At 0.17–6.55 per model per task, the framework is expensive. This limits both the benchmark size and the feasibility of repeated evaluations, which contradicts the "scalable" framing.
Confounds in real environments. The authors note that scores may reflect OS configuration, tool wrappers, and runtime stability rather than model capability alone. This is a fundamental tension: realism is gained at the cost of controlled measurement.
3. Potential Impact
The paper addresses a genuine gap: as LLM agents move from text generation to persistent-state manipulation in real computing environments, evaluation must follow. The state-based evaluation paradigm is conceptually sound and timely. However, the practical impact is limited by several factors:
The failure analysis (Section 4.1) and tool-use analysis (Section 4.2) provide useful diagnostic insights—particularly that tool failures dominate non-passing runs (95.4%) and that more tool calls don't correlate with better performance. The memory perturbation analysis (Section 4.4) is interesting but extremely small-scale (4 tasks, 1 model).
4. Timeliness & Relevance
This paper is highly timely. The rapid deployment of LLM-powered personal agents (Claude Code, OpenClaw, etc.) has created urgent demand for evaluation frameworks that go beyond text-based benchmarks. The paper correctly identifies three real limitations of existing benchmarks: sandboxed artifacts, static task design, and coarse scoring. The comparison table (Table 1) positions STAGE-Claw relative to recent concurrent work, though the field is moving so rapidly that the referenced benchmarks span only 2024–2026.
5. Strengths & Limitations
Strengths:
Limitations:
Additional Observations
The paper reads more as a system/benchmark description than a research contribution with deep technical novelty. The automated pipeline uses standard LLM-based generation and validation patterns. The evaluation insights, while useful, are descriptive rather than analytical—they catalog failure modes without proposing solutions. The contribution is primarily engineering and benchmarking rather than methodological innovation.
The writing is generally clear but the paper could benefit from tighter focus. The appendices contain substantial detail about prompts and task descriptions, which aids reproducibility but suggests the core framework may be thinner than presented.
Generated Jun 10, 2026
Comparison History (19)
Paper 2 (STAGE-Claw) likely has higher scientific impact: it introduces a scalable, automated benchmarking framework for realistic, state-based personal-computing environments with programmatic verification, addressing a major evaluation bottleneck for agent research. Its applicability spans many agent types, tools, and domains, enabling reproducible comparisons and guiding progress across the field. Paper 1 (TreeSeeker) is a solid inference-time control contribution for deep search, but is narrower in scope and may be more quickly subsumed by broader agent frameworks; benchmarks/infrastructure tend to have wider, longer-lasting cross-field influence.
Paper 2 addresses a fundamental efficiency bottleneck in LLM inference (quadratic attention scaling) with a novel insight that RL can bridge the performance gap between sliding-window and self-attention architectures. This finding has broad implications for efficient LLM deployment across many applications, not just math reasoning. The architecture-aware RL approach is a generalizable methodological contribution. Paper 1, while useful, is an incremental improvement in agent benchmarking—a crowded space—and its impact is more narrowly scoped to evaluation methodology rather than enabling new capabilities.
Paper 1 addresses a major, widespread bottleneck in agent research: the need for realistic, state-based evaluation benchmarks rather than static or purely text-based ones. Its automated framework for generating and validating state-based tasks has broad applicability across the field of LLM agents. Paper 2 focuses on a more specific, niche problem regarding how the structural organization of 'Skills' affects agent behavior, which offers narrower methodological impact compared to a new foundational benchmarking paradigm.
Paper 2 (STAGE-Claw) likely has higher impact: it delivers a concrete, scalable benchmarking framework plus an actual benchmark (40 realistic state-based tasks) and broad evaluation across 11 frontier models, addressing a timely bottleneck in agent reliability and measurement. Its method (state-based verification, automatic task/environment/validator generation) is broadly reusable across personal-computing, HCI, and AI evaluation, enabling standardized comparisons and iterative progress. Paper 1 is novel and important conceptually (aggregation inversion + external audit loop) but appears demonstrated on a narrower case and may need broader empirical validation to match Paper 2’s immediate applicability.
STAGE-Claw addresses a more fundamental and broadly impactful challenge: automated benchmark generation and state-based evaluation for LLM-powered personal agents, a rapidly growing area. Its scalable, automated framework for creating and validating benchmarks tackles key limitations (static tasks, coarse scoring) affecting the entire agent ecosystem. While EngVQA provides valuable domain-specific evaluation for engineering reasoning with a strong process-oriented framework, its scope is narrower. STAGE-Claw's contribution to agent evaluation infrastructure has broader applicability across the AI community and addresses more timely scalability concerns.
Paper 1 likely has higher impact due to broader applicability and timeliness: scalable, automated benchmarking for realistic agent tasks in state-based OS-like environments addresses a central bottleneck in deploying LLM agents. Its framework can be adopted across many tasks, models, and research groups, influencing evaluation standards and accelerating progress. Paper 2 is methodologically rigorous (pre-registration, strong statistics) and offers clear insights for spatial-memory systems, but its scope is narrower (occlusion/visibility in spatial recall) and closer to a specialized diagnostic/ablation than a general infrastructure contribution.
ActiveMem addresses a fundamental architectural limitation in LLM reasoning—centralized memory causing context overload vs. information loss tradeoffs. Its biologically-inspired distributed memory framework with demonstrated SOTA results on established benchmarks represents a more novel and broadly applicable contribution. Paper 2 introduces a useful evaluation framework, but benchmarking tools generally have narrower impact than architectural innovations that can be adopted across many systems. ActiveMem's approach could influence how future LLM agents are designed for long-horizon tasks across diverse applications.
Paper 1 introduces an automated framework for generating state-based benchmarks, addressing a critical bottleneck in the scalability and realism of agent evaluation. This methodological innovation has broader applicability across various personal computing environments compared to Paper 2, which, while highly relevant to AI research automation, relies on a more static benchmark approach. Paper 1's focus on automated creation and state-based verification offers a more robust and scalable tool for the rapidly growing field of LLM agent development.
Paper 1 has higher likely scientific impact: it introduces a concrete, scalable benchmarking framework for evaluating LLM-powered agents in realistic stateful environments with automated task generation/validation and state-based scoring—addressing a timely, widely felt bottleneck in agent research. The contribution is methodological infrastructure that can be adopted across many domains and models, enabling reproducible progress. Paper 2 reads as an over-broad integration of many established techniques with large claimed gains; novelty is unclear, and such “unified framework” claims often lack rigorous, isolatable contributions and generalizable evaluation.
STAGE-Claw addresses the highly timely and broadly impactful problem of evaluating LLM-powered agents in realistic scenarios, proposing an automated benchmarking framework with state-based evaluation. Given the explosive growth in LLM agent development, this work has significant potential to shape evaluation standards across the field. Paper 2, while technically sound, focuses on computational optimizations for a specific neurosymbolic framework (NeurASP), which has a narrower audience and more incremental contribution. The breadth of impact and timeliness strongly favor Paper 1.