Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Paper 1 likely has higher impact because it targets a broadly relevant, under-measured variable—agent “harness” effects—across many models and realistic tool-using workflows, providing a sizable, instrumented benchmark with traces to diagnose failure modes. This can influence evaluation standards, reporting practices, and system design across the fast-growing LLM agent ecosystem (software engineering, productivity, robotics-like tool use). Paper 2 is timely and important for clinical NLP, but its scope is narrower and more domain-specific, with real-world deployment constrained by regulation and data access.

Paper 1 is likely higher impact due to a concrete, broadly useful benchmark and dataset enabling systematic study of harness (execution-layer) effects across models—an under-measured, timely factor in real-world agent performance. It offers methodological rigor (sandboxed tasks, oracle-checkable validators, large trajectory set, rich traces) and clear applications for evaluating/debugging agent stacks, improving reliability, efficiency, and auditability. Paper 2 is important conceptually and timely for safety, but as a position piece its impact depends more on subsequent adoption and empirical depth; its benchmark scope appears narrower and tied to a specific agent setup.

Harness-Bench addresses a critical and timely gap in LLM agent evaluation—isolating the effect of the execution harness from the model itself—which has broad practical implications for the rapidly growing agent deployment ecosystem. It introduces a comprehensive benchmark with 5,194 trajectories and identifies a novel concept (execution-alignment failures) relevant across many agent applications. Paper 2 makes a solid methodological contribution (ADR metric, paired-formula protocol) for evaluating LLM reasoning on SAT, but targets a narrower, more theoretical question with less immediate breadth of impact across the field.

Paper 1 introduces a paradigm shift in evaluating LLM agents by highlighting the critical role of the system 'harness', rather than just the base model. This addresses a major gap in current methodologies. While Paper 2 offers valuable insights into memory systems, Paper 1's broader focus on the entire execution stack and its potential to redefine standard reporting practices for agent capabilities gives it a higher potential for widespread methodological impact across the field of AI agents.

Paper 2 addresses a timely and critical problem in AI safety—how to handle potentially adversarial behavior in deployed AI coding agents. It provides concrete, actionable findings (retrying leaks exploitable information, resampling strategies, contradictions with prior work) that directly inform the design of safer AI systems. Its findings challenge established assumptions in the AI control literature, which amplifies its impact. Paper 1 introduces a useful benchmark for studying harness effects on agent performance, which is valuable but more incremental. Paper 2's safety implications give it broader urgency and cross-field relevance.

Paper 2 presents a high-impact, interdisciplinary application of LLM agents to neuroimaging, addressing significant bottlenecks in medical research. By automating complex preprocessing and achieving high accuracy in Alzheimer's Disease classification, it offers immediate, tangible benefits to clinical and neuroscience research. While Paper 1 provides a useful AI benchmark, Paper 2's direct real-world healthcare application and ability to accelerate reproducible scientific analysis give it a broader and more profound potential scientific impact.

Paper 1 addresses a critical and foundational challenge in the rapidly expanding field of autonomous LLM agents: standardized evaluation. By demonstrating that the system 'harness' significantly affects agent performance, it challenges current evaluation paradigms and provides a necessary benchmark for the broader AI research community. Paper 2, while highly valuable for commercial applications, proposes a more domain-specific application of contrastive learning (e-commerce product images), giving it a narrower scope of scientific impact compared to the fundamental methodological shift proposed in Paper 1.

Paper 1 offers deeper mechanistic insights into how LLMs perform reasoning, identifying specific attention heads responsible for deductive reasoning steps through causal mediation analysis. This contributes fundamental understanding of LLM internals with broad implications for interpretability and model improvement. Paper 2 introduces a useful benchmark for evaluating harness effects on agent workflows, but benchmarks tend to have more incremental impact and shorter relevance windows. Paper 1's contributions to mechanistic interpretability address a more foundational question with broader applicability across the field.

Paper 2 introduces a novel conceptual framework (GEM) that redefines long-term agent memory as a fundamentally new data-management workload, with formal correctness conditions and proof that existing paradigms are insufficient. This has broader theoretical and practical impact across database systems, AI agents, and memory management. Paper 1, while useful, is primarily a benchmarking contribution for harness-level evaluation—important but more incremental. Paper 2 opens new research directions at the intersection of databases and AI agents, likely inspiring more follow-on work across multiple communities.

Paper 1 likely has higher impact: it introduces a new, reproducible benchmark targeting an under-measured but practically critical variable (agent harness configuration) with substantial empirical scale (106 tasks, 5,194 trajectories) and rich logged artifacts/traces enabling broad downstream research (reliability, tool use, eval methodology, systems). Its applications are immediate for deployed agent stacks and could standardize reporting at the model–harness level. Paper 2 is timely and conceptually important as a critique with stronger controls, but its scope is narrower (specific introspection paradigms) and is less directly enabling for engineering and cross-field benchmarking.

Paper 1 introduces a critical benchmark for evaluating the system-level 'harness' of LLM agents, addressing a major gap in real-world agent evaluation. Benchmarks that redefine how capabilities are measured (model-harness pairing vs. base model alone) typically have broader, longer-lasting impact across the field than specific training optimizations like the one presented in Paper 2.

Paper 1 likely has higher impact: it introduces a broadly applicable benchmark isolating “harness” effects in LLM agent execution, a timely and under-measured source of variance as agents move into real deployments. Its methodology (sandboxed offline tasks, oracle-checkable outcomes, traces, large trajectory count) enables rigorous, reproducible diagnosis across many systems and research groups. The resulting framing—reporting capability at the model–harness configuration level—could shift evaluation practice across agent tooling, alignment, reliability, and systems engineering. Paper 2 is useful and practical, but more incremental and narrower to claim–citation verification.

Paper 1 addresses a critical, highly practical gap in the rapidly expanding field of LLM agents by isolating the impact of the system 'harness' from the base model. This insight fundamentally shifts how agent capabilities should be evaluated and reported. While Paper 2 offers a valuable cognitive benchmark for Theory of Mind, Paper 1 has broader real-world applicability for building, diagnosing, and deploying reliable autonomous systems, giving it a higher potential for immediate and widespread scientific and engineering impact.