TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Paper 1 addresses a more fundamental and underexplored research question—whether in-context RL can handle multi-agent coordination—revealing important negative results that challenge assumptions from single-agent ICRL success. This has broader theoretical implications for foundation agents and multi-agent systems. Paper 2, while practically useful, is primarily an engineering contribution (a benchmark for terminal agents) with narrower scope. Paper 1's findings are more likely to redirect research efforts in both ICRL and ad-hoc teamwork communities, giving it higher scientific impact despite Paper 2's immediate practical utility.

Paper 2 has higher likely impact because it proposes a broadly applicable, efficiency-focused agent framework (online skill distillation with measurable token savings) on a widely used multimodal benchmark, addressing a timely bottleneck: rising inference cost. Its components and new efficiency metrics can transfer across web agents and other interactive settings, enabling real-world deployment improvements. Paper 1 is valuable infrastructure (a scalable, authentic terminal benchmark) but is primarily evaluative; its impact depends on adoption and may be narrower than a method that directly improves agent efficiency and performance.

Paper 1 likely has higher impact: it targets the broader and more consequential setting of always-on personal assistants spanning multi-device GUI/CLI, long-horizon user histories, and interdependent services, enabling evaluation of proactive assistance—capabilities central to near-term agent deployment. Its simulated months-long, noisy, conflicting user-state setup is a novel benchmark axis beyond terminal-only tasks, and the released scalable data-generation pipeline with demonstrated training gains increases practical adoption and research leverage. Paper 2 is rigorous and valuable, but its scope is narrower (terminal workflows) and likely affects a smaller slice of agent applications.

Paper 1 likely has higher scientific impact due to its novel, scalable benchmark-construction engine from real-world terminal recordings, producing a large, reusable dataset and evaluation suite with clear evidence of unmet capability and weak correlation with prior benchmarks. This directly enables broad, timely progress in agent evaluation, tool-use, and alignment across many domains. Paper 2 is potentially high-impact clinically, but the abstract suggests retrospective evaluation and a narrower domain; deployment and rigorous validation barriers in healthcare may limit near-term field-wide impact.

Paper 2 is likely higher impact: it tackles a timely, high-stakes security problem for memory-augmented LLM agents (poisoned persistent memory) with a novel post-hoc auditing angle, combining causal attribution with structural anomaly detection. The approach has clear real-world applicability for deploying safer agents and could influence both ML security and agent design. While Paper 1 provides a valuable, scalable benchmark for terminal agents, benchmarks are more incremental and domain-specific; MemAudit’s methods and framing generalize across agent systems and align with urgent concerns about agent robustness and adversarial misuse.

Paper 2 (TerminalWorld) likely has higher scientific impact due to its broadly reusable, scalable benchmark generation engine built from large-scale real terminal recordings, yielding a substantial public resource (1,530 tasks) directly applicable to evaluating and improving agentic systems. Its real-world relevance and timeliness (agent reliability, software automation) plus weak correlation with existing benchmarks suggests it measures a distinct, important capability, potentially reshaping evaluation practice across academia and industry. Paper 1 is novel and important for forecasting calibration, but is narrower in scope and primarily influences evaluation methodology within forecasting/LLM reliability.

EVE-Agent introduces a novel principled framework for trustworthy self-evolution in AI agents by requiring evidence verifiability—a conceptually significant contribution with broad implications for self-improving systems. The evidence verification mechanism (measuring marginal accuracy gain) is methodologically innovative and addresses a fundamental problem of hallucination/unreliability in self-training loops. While TerminalWorld is a solid benchmark contribution, benchmarks have more incremental impact and can become outdated. EVE-Agent's approach to auditable self-evolution has broader applicability across domains and addresses a more fundamental research challenge.

TerminalWorld introduces a novel scalable benchmarking methodology that addresses a significant gap in evaluating AI agents on real-world terminal tasks. Its automated data engine processing 80K+ recordings, coverage of 1,280 unique commands across 18 categories, and demonstration that current frontier models achieve only 62.5% pass rate provide high-impact insights for the rapidly growing AI agent community. The benchmark fills a distinct niche (weak correlation with existing benchmarks) and is designed to scale with evolving practices. While LACO presents solid technical contributions to collaborative driving, its scope is narrower, constrained to a specific simulation environment, and builds incrementally on existing paradigms.

Paper 1 addresses a critical, high-stakes domain (healthcare) by revealing significant flaws in how medical LLMs are currently evaluated. Demonstrating that static benchmarks overestimate interactive diagnostic performance has profound implications for patient safety and AI deployment in clinical settings. While Paper 2 provides a valuable benchmark for software agents, Paper 1's focus on clinical reasoning under uncertainty directly impacts the safe integration of AI in life-critical applications, giving it a broader and more urgent scientific and societal impact.

TerminalWorld introduces a novel, scalable benchmark with concrete data (1,530 tasks from 80,870 recordings), open-source resources, and empirical findings showing current agents achieve only 62.5% on real terminal tasks. It fills a clear gap in agent evaluation with a reproducible methodology. Paper 2 is a position/survey paper arguing for a shift in LLM-based planning approaches, which, while insightful, offers less concrete novelty and empirical contribution. Benchmarks tend to have outsized impact by enabling standardized evaluation across the community.