This assessment is based on version 1 of this paper. Version 3 is now available on arXiv — the authors may have revised their methods, results, or conclusions.
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
Kai Tzu-iunn Ong, Minseok Kang, Dongwook Choi, Junhee Cho, Seungju Kim, Seungwon Lim, Geunha Jang, Minwoo Oh
Abstract
Harness optimization enables automated agent creation by having an optimizer agent iteratively update the harness of target agents. Despite its success, current studies evaluate optimizers solely by observing target agents' performance gains. This indirect end-improvement evaluation neglects optimizers' actions at intermediate steps, which are often erroneous and hinder agent performance. Therefore, it is unclear whether harness optimization is driven by optimizers' informed update actions or simply trial-and-error. This necessitates direct evaluation of harness optimizers. However, evaluating harness optimizers directly is non-trivial and costly due to the lack of oracle harnesses. To address this, we present a simple, low-cost design to directly evaluate them, namely priority ranking. By asking harness optimizers to rank components (e.g., tools) in a given harness by their potential to improve/hinder agent performance when updated, our design quantifies optimizer ability at the step level without expensive rollouts or manual examination. More importantly, optimizers' ranking performance correlates with their ability to improve agents in actual multi-step harness optimization, establishing priority ranking as a reliable predictor of optimization ability. Priority ranking is enabled by Shor, a collection of 182 human-verified optimization scenarios spanning across domains, designs, and time stages. Codes and data can be found at https://github.com/k59118/Harness_Optimizer_Evaluation.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper identifies a significant gap in how harness optimizers (agents that iteratively improve other agents' surrounding infrastructure) are evaluated. Currently, optimizers are assessed only via end-to-end performance gains of target agents — a black-box approach that ignores whether intermediate optimization steps are informed or merely trial-and-error. The authors propose priority ranking, a task where optimizers must rank harness components (prompt, tool, memory, workflow) by their potential to improve agent performance when updated. This is accompanied by SHOR, a dataset of 182 human-verified optimization scenarios with ground-truth priority labels derived from actual SR changes across component-targeted updates.
The key insight is elegant: rather than requiring oracle next-harnesses (which don't exist) or expensive multi-step rollouts, the evaluation reduces direct optimizer assessment to a single-step ranking problem over relative component priorities. This sidesteps both the cost and the epistemic bottleneck of direct evaluation.
Methodological Rigor
Strengths in methodology:
Concerns:
Potential Impact
Direct impact on harness optimization research: This work provides the first systematic framework for understanding *why* optimizers succeed or fail, not just *whether* they do. The 8× cost reduction and 17× speed improvement over end-improvement observation make optimizer comparison practically feasible for resource-constrained labs.
Actionable insights for optimizer design: Table 4's finding that providing oracle component-location information improves error resolution by 17-51 percentage points is highly consequential. It suggests a clear architectural recommendation: decouple diagnosis (which component?) from execution (how to fix it?), potentially leading to two-stage optimizer designs.
Broader implications: The finding that "good agent harnesses are not necessarily good optimizer harnesses" (Finding 2) challenges the implicit assumption in the field that strong agentic frameworks are universally transferable. This has implications for how the community benchmarks and develops optimization infrastructure.
Industrial relevance: The discussion of CI/CD integration for priority ranking as a quality gate is forward-looking, though the current SOTA performance levels (best Acc@1 of 0.305) suggest this is aspirational rather than immediately deployable.
Timeliness & Relevance
This paper arrives at an inflection point. Harness optimization has rapidly gained traction (ADAS, AFlow, ReCreate, Meta-Harness, HyperAgents), yet evaluation methodology has not kept pace — all prior work uses end-improvement observation exclusively. The paper fills this gap at a moment when the community is scaling up automated agent creation, making the question of optimizer reliability increasingly urgent. The concurrent work by Ursekar et al. [15] validates this need independently, though their approach (manual commit history analysis) is far less scalable.
Strengths & Limitations
Key Strengths:
1. The problem formulation is incisive — converting an intractable direct evaluation problem into a tractable ranking task while maintaining predictive validity.
2. The empirical correlation between ranking and optimization ability (validated in-domain and OOD) provides the scientific foundation needed to justify this proxy evaluation.
3. The finding about mid-stage harnesses yielding strongest correlation (Figure 4) adds nuance and practical guidance for when to apply the evaluation.
4. The dataset construction is thorough, with multiple filtering stages and human verification.
Notable Limitations:
1. The correlation, while statistically significant, explains ~36% of variance (ρ²≈0.36), leaving substantial unexplained variation — as the Spider 2.0-lite exception illustrates. The authors appropriately characterize ranking as necessary but not sufficient.
2. The component taxonomy is static and predefined. Real optimization often involves cross-component interactions that ranking over four categories cannot capture.
3. Reproducibility concerns: the dataset relies on specific API model versions (GPT-5.x, Claude Sonnet 4.6) that may be deprecated, though the methodology is replicable.
4. The paper could benefit from ablation studies on annotation design choices (e.g., number of annotators, filtering thresholds).
5. OOD correlation for GPQA (ρ=0.282, not significant) suggests that generalization claims should be tempered.
Overall Assessment
This is a well-motivated, methodologically sound contribution that opens a new evaluation dimension for an increasingly important problem. The priority ranking framework is simple enough to be widely adopted while being empirically grounded enough to be trustworthy. The dataset and actionable insights for optimizer development add substantial practical value. The main limitations — coarse granularity, modest correlation sample size, and single target agent — are reasonable for a "first step" paper and define clear directions for follow-up work.
Generated May 22, 2026
Comparison History (18)
Paper 2 introduces a novel, low-cost evaluation methodology and a verified dataset (Shor) for automated agent optimizers. By solving a critical evaluation bottleneck, it offers immediate, practical utility for researchers. While Paper 1 provides a timely security survey, Paper 2's methodological contribution and open-source benchmark are more likely to directly drive and measure future empirical advancements, yielding higher fundamental scientific impact.
PANDO addresses a practical and timely problem—making multimodal web agents more efficient rather than more expensive—with concrete, reproducible results (58.3% success rate with 58-61% fewer tokens on VisualWebArena). It introduces a complete framework with actionable techniques and novel efficiency metrics that the community can adopt broadly. Paper 1 proposes a useful evaluation methodology for harness optimizers, but its scope is narrower, serving primarily as a diagnostic benchmark rather than enabling new capabilities. Paper 2's combination of strong empirical results, practical efficiency gains, and new evaluation metrics gives it broader impact potential.
Paper 1 addresses a fundamental evaluation gap in the rapidly growing field of agentic AI and harness optimization, proposing a novel, principled evaluation methodology (priority ranking) with a curated benchmark (Shor). It has broader scientific impact across AI agent research, offering methodological rigor with human-verified scenarios and demonstrated correlation with actual optimization outcomes. Paper 2, while timely and well-executed, addresses a narrower applied question (citation behavior in AI answer engines) with more limited generalizability and primarily practitioner-oriented contributions rather than advancing fundamental scientific understanding.
Paper 2 addresses a practical and broadly relevant problem in LLM agent serving—managing growing context windows efficiently—with clear engineering contributions (parallel compaction) validated across multiple model architectures and benchmarks. It has immediate real-world applicability to any long-horizon LLM agent deployment, affecting latency, throughput, and reliability. Paper 1 introduces a useful evaluation methodology for harness optimizers, but it targets a narrower sub-community focused on automated agent creation. Paper 2's broader applicability across the rapidly growing LLM agent ecosystem gives it higher potential impact.
Paper 1 addresses a fundamental methodological gap in the rapidly growing field of autonomous agents by introducing a direct evaluation metric and benchmark dataset. Foundational evaluation tools typically have a broader and more systemic scientific impact across the AI community compared to domain-specific applications, like the topology optimization pipeline presented in Paper 2.
Paper 1 likely has higher impact due to its broadly applicable, timely contribution to evaluating agent/harness optimizers: a direct, low-cost proxy metric (priority ranking) validated to correlate with downstream optimization outcomes, plus a human-verified benchmark (Shor). This addresses a core methodological gap in agent optimization research and can influence evaluation practices across tool-use, prompt/harness engineering, and automated agent design. Paper 2 is valuable (new ToM persuasion task/dataset and framework), but is more domain-specific, and claims (e.g., outperforming GPT-5) may be harder to generalize or verify without strong baselines and auditing.
Paper 2 likely has higher impact due to broader applicability and timeliness: it delivers an end-to-end multi-agent engineering design framework plus a benchmark spanning workflow reasoning, RAG isolation, and real HPC/SLURM orchestration with manufacturing integration. This supports real-world adoption in engineering pipelines and provides a standardized evaluation suite across models and tasks. Paper 1 is novel and methodologically useful for direct optimizer evaluation, but is narrower in scope (harness optimization) and its benchmark size/domain breadth is more limited, reducing cross-field impact.
Paper 2 introduces a novel approach to metacognitive AI, enabling LLMs to autonomously identify and repair knowledge gaps using graph-theoretic metrics. This addresses critical limitations in current AI systems, such as hallucinations and static knowledge. Its broad applicability, demonstrated across standard QA datasets, and integration of trending technologies like GraphRAG suggest significant potential for widespread adoption and real-world impact. Paper 1, while methodologically rigorous, focuses on a narrower niche of evaluating harness optimizers, which has a smaller target audience and lower expected breadth of impact.
Paper 1 offers a timely, practically useful evaluation paradigm for LLM/agent “harness” optimization, addressing a clear gap (direct, step-level optimizer assessment) with a low-cost metric and an accompanying human-verified benchmark dataset (Shor). The reported correlation with real multi-step optimization suggests actionable predictive validity, enabling broader, reproducible progress across agent optimization research. Paper 2’s causal+argumentation XAI integration is interesting but closer to incremental synthesis of existing causal discovery and argumentation formalisms, demonstrated on limited benchmarks, with likely narrower immediate adoption compared to agent evaluation infrastructure.
Paper 2 likely has higher impact due to broader relevance and timeliness: it addresses concept alignment and AI safety, comparing model judgments to human participants and linking misalignment to downstream problematic behavior. The “implausible category members” probe is a simple, innovative evaluation idea that can generalize across models, benchmarks, and cognitive-science/AI alignment communities. Paper 1 is valuable for agent/harness optimization evaluation and provides a dataset, but its scope is narrower (optimizer assessment within a specific agent-building paradigm) and may affect a smaller set of researchers and applications.