Interactive Evaluation Requires a Design Science

Scientific Impact Assessment

1. Core Contribution

This position paper argues that interactive evaluation of AI systems—where agents act through tools, environments, users, and other agents over time—should be treated as a principled design science rather than a loosely connected family of agent benchmarks. The authors formalize evaluation as an autonomous mapping E: X → Y, where X is admissible evidence and Y is the space of evaluative judgments. They show that interactive evaluation changes both sides: X expands from single responses to interaction-generated trajectories, while E must assess process quality, recoverability, coordination, robustness, and safety beyond final-answer correctness.

The paper's main intellectual contribution is a two-axis taxonomy organizing interactive evaluations by (1) what interaction artifacts enter as evidence (tools/environments, users, other agents, hybrid/dynamic systems) and (2) what evaluation programs map trajectories to judgments (task success, process quality, recoverability, safety/alignment). By mapping existing benchmarks into this 2D space, the authors identify systematic gaps—particularly the dominance of outcome-only scoring even when rich trajectory data is available, and the sparsity of hybrid/dynamic system evaluations.

2. Methodological Rigor

As a position paper, methodological rigor manifests differently than in empirical work. The paper's conceptual framework is clean and well-motivated. The E: X → Y formalism is simple but effective—it clarifies exactly where interactive evaluation departs from response-centered evaluation without overcomplicating the argument.

The taxonomy is derived systematically from the definition rather than imposed ad hoc. The boundary cases (multi-turn without action-dependence, tool calls without state change, chain-of-thought without external loops) are well-chosen and sharpen the definition's scope. The paper also includes a semi-automated benchmark collection methodology with quality filters (citation velocity, GitHub stars, venue) and LLM-based classification validated against manual labels (>90% agreement), lending some empirical grounding to the landscape analysis.

However, the empirical analysis remains largely descriptive. The χ² test comparing industry vs. academic evaluation-stage distributions (p=0.029) is suggestive but based on a relatively small industry sample (43 benchmark families from 4 companies). The 2D taxonomy mapping in Figure 3 relies on the authors' categorization of benchmarks, which, while reasonable, involves subjective judgment calls. The paper would benefit from inter-annotator agreement metrics on the taxonomy placement.

3. Potential Impact

The paper's impact potential is substantial but depends heavily on community adoption. Its contributions operate at three levels:

Conceptual clarity: The distinction between "recording trajectories" and "evaluating at the trajectory level" is genuinely important. Many current benchmarks collect rich interaction data but collapse it into pass/fail scores, discarding evidence about process quality, safety, and robustness. Making this explicit could shift how benchmark designers think about scoring.

Practical guidance: The design principles (specify the system and trajectory evidence, specify interaction protocols, design for perturbation and repair, separate outcome/process/risk) are actionable. The call for protocol documentation as "the interactive analogue of dataset documentation" is apt and could lead to standardized reporting practices.

Community organization: By providing a shared vocabulary and taxonomy, the paper could help reduce fragmentation across web agents, coding agents, multi-agent systems, and tool-use benchmarks that currently develop evaluation practices independently.

The two illustrative scenarios (coding agents, multi-agent social systems) in Appendix D effectively demonstrate how the framework applies in practice, though they remain at the conceptual level without implementing the proposed evaluation programs.

4. Timeliness & Relevance

The paper is highly timely. The rapid proliferation of agent benchmarks (WebArena, SWE-bench, τ-bench, SOTOPIA, etc.) has created exactly the kind of fragmentation the authors diagnose. The field is at an inflection point where agent evaluation is growing quickly but without shared conceptual infrastructure. The observation that many benchmarks admit trajectory evidence but score only outcomes captures a real and consequential gap.

The industry-vs-academic divergence highlighted in Figure 1 is also relevant: frontier labs are increasingly evaluating interactive capabilities while academic benchmarks retain a response-centered center of gravity. This creates a practical need for the kind of principled framework proposed here.

5. Strengths & Limitations

Strengths:

Limitations:

6. Additional Observations

The paper's framing as "design science" (citing Simon, Hevner, Wieringa) is interesting but underexplored. Design science methodology has specific prescriptions (design artifacts, evaluation cycles, communication protocols) that could have been more directly applied. The invocation feels more metaphorical than methodological.

The GitHub repository is referenced but its contents and utility for the community are unclear from the paper. Shared infrastructure—logging schemas, trajectory viewers, evaluation harnesses—is called for but not provided.