Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz
Abstract
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Insights Generator
1. Core Contribution
The paper formalizes a genuinely underexplored problem: corpus-level trace diagnostics for LLM agents. Rather than debugging individual trajectories (the dominant paradigm), IG seeks to discover population-level behavioral patterns across entire trace corpora. The key architectural innovation is a scout-investigator decomposition where scout agents propose hypotheses from sampled traces (breadth) and investigator agents validate them at corpus scale with statistical evidence (depth), coordinated by an orchestrator in an iterative loop. All trace access is mediated through a structured Python data processing layer, preventing raw trace content from flooding model context windows.
This is a meaningful reframing. The distinction between hypothesis generation and corpus-scale validation addresses a real bottleneck: manual trace inspection doesn't scale, and single-trajectory debugging tools miss cohort-level patterns. The motivating example — "84% of incorrect Python-using traces exhibit silent computation failures" — illustrates the kind of insight that per-trace methods fundamentally cannot surface.
2. Methodological Rigor
The paper's four-setting evaluation framework (automated vs. human evaluator × report quality vs. downstream impact) is well-designed and addresses the inherent difficulty of evaluating free-form diagnostic outputs. The evaluation includes:
Strengths in rigor: The statistical analysis (Appendix A.9) is thorough — mixed-effects models, permutation tests, and leave-one-out sensitivity analyses all support the central claim. The ablation study isolating typed role decomposition from generic subagent dispatch is informative.
Weaknesses in rigor: The human intervention study has n=6 per arm — adequate for detecting the large observed effect (Cohen's d=1.89) but insufficient for generalizable conclusions. The study is conducted on a single benchmark (SpreadsheetBench). The coverage metric shows all multi-agent systems cluster between 87-98%, suggesting it's not very discriminating. The LLM-as-a-judge results and human expert ratings diverge (IG and CC reports rated similarly by humans despite clear LLM-judge separation), which the authors attribute to human cognitive limitations rather than examining whether the LLM judge might be miscalibrated — a somewhat one-sided interpretation.
3. Potential Impact
Practical impact is potentially high. Agent debugging is genuinely a pain point in production LLM systems. A system that can surface "silent computation failures" or "cross-checking that verifies internal consistency within the wrong framework" from hundreds of traces addresses a real need. The 30.4pp improvement demonstrates concrete practitioner value.
Research impact: The formalization of corpus-level trace diagnostics as a distinct problem (separate from single-trajectory debugging and taxonomy classification) could spawn a subfield. The four-setting evaluation framework provides a template for future work. The union-gold clustering approach for evaluating open-ended diagnostic outputs is a useful methodological contribution.
Limitations to impact: The system costs ~23-38 for alternatives) and takes ~48 minutes. This is manageable for production debugging but limits iteration speed. The approach is fundamentally tied to the quality of the underlying LLM (Claude Opus 4.6 throughout), making it hard to separate system design contributions from model capability.
4. Timeliness & Relevance
This paper arrives at a critical moment. LLM agents are rapidly moving into production (coding assistants, research agents, workflow automation), and the gap between building agents and debugging them at scale is widening. The paper directly addresses this bottleneck. The comparison against recent systems (Trace2Skill, HALO/RLM, AgentRx) demonstrates awareness of the fast-moving landscape. The connection to agent optimization frameworks (VeRO, AFlow, Meta-Harness) positions IG as a complementary component in the emerging agent development stack.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
Additional Observations
The paper is well-written with extensive appendices providing reproducibility-critical details (prompts, protocols, statistical analyses). The honest reporting of the human-expert-as-judge near-parity and the edit-size non-significance is commendable. However, the framing sometimes oversells: "nearly doubling the 16.2pp gain" is technically correct but the CC Subagents baseline is itself strong, and the human judges couldn't distinguish the reports.
The contribution sits at the intersection of systems engineering and evaluation methodology rather than fundamental algorithmic innovation. The scout-investigator pattern is sensible but not deeply novel — it mirrors standard exploratory data analysis workflows operationalized through multi-agent dispatch.
Generated May 22, 2026
Comparison History (25)
AutoMedBench introduces a novel workflow-aware benchmark for autonomous medical-AI research with stage-level diagnostics across diverse medical imaging tasks. Its focus on the high-impact medical AI domain, combined with systematic evaluation of agent behavior through a five-stage workflow framework, addresses a critical gap in benchmarking autonomous research agents. While Paper 2 (Insights Generator) offers valuable corpus-level trace diagnostics, AutoMedBench has broader impact potential due to its domain specificity in healthcare, practical relevance for advancing autonomous medical research, and comprehensive benchmark design that can shape future agent development across multiple medical tasks.
Paper 2 (AGENTCL) has higher estimated impact because it proposes a rigorous, controlled evaluation framework and metrics for continual learning in language agents—an enabling contribution likely to be reused broadly across labs and agent architectures. Its controlled task streams address a core methodological gap (confounded/naive streams), improving rigor and comparability, and it spans multiple domains (coding, research, reasoning). Paper 1 is practically valuable for debugging and shows strong applied gains, but is more tool/system-specific and likely narrower in long-term, cross-field adoption than a benchmark+methodology that can standardize future work.
Paper 1 addresses a critical and fundamental bottleneck in LLMs—reliability and confidence estimation—by innovatively applying interactive proof theory to selective prediction. This approach has broad implications for AI safety, hallucination mitigation, and reasoning. While Paper 2 offers a valuable tool for agent diagnostics, its contribution is primarily in software engineering workflows, whereas Paper 1 advances the fundamental understanding of model deliberation, yielding higher potential scientific impact across the broader AI research community.
Paper 1 addresses a highly timely and fundamental challenge in Large Reasoning Models: test-time compute and reasoning trajectory control. By systematically analyzing and calibrating reflection markers during decoding, it offers a novel, training-free method that improves reasoning efficiency. While Paper 2 presents a highly practical tool for agent diagnostics, Paper 1 provides deeper algorithmic insights into the mechanics of LLM reasoning, which is likely to spur broader fundamental research in inference optimization and test-time scaling.
Paper 1 is more novel and potentially higher-impact: it moves agent adaptation from prompt/text artifacts to source-level self-rewriting with verification, promotion, and rollback—addressing structural failures unreachable by existing self-improvement methods. This could materially change how production agents are maintained, enabling continuous, deterministic evolution with safety gates. While Paper 2 is timely and useful for scalable diagnostics, corpus-level trace analysis is a more incremental extension of existing observability/analysis paradigms and depends on humans/agents to apply insights, whereas Paper 1 directly closes the loop to autonomous, deployable fixes.
While Paper 1 offers a valuable practical tool for LLM debugging, Paper 2 presents a highly counterintuitive and paradigm-challenging scientific finding: that higher observation fidelity hurts embodied LLM problem-solving. By demonstrating that perceptual noise disrupts LLM reasoning failures like repetitive loops, Paper 2 fundamentally challenges current evaluation assumptions in embodied AI. This conceptual disruption is likely to spark significant theoretical debate, broader follow-up research across robotics and LLM reasoning, and a reevaluation of how cognitive architectures are tested, giving it a higher potential scientific impact.
Paper 1 addresses a fundamental and broadly applicable problem—systematic diagnosis of LLM agent failures at scale—which is relevant across virtually all LLM agent deployments. It formalizes a new problem (corpus-level trace diagnostics), proposes a multi-agent architecture, and demonstrates concrete downstream improvements (30.4pp). This has high practical impact for the rapidly growing LLM agent ecosystem. Paper 2, while novel in benchmarking prompter proficiency for T2I systems, addresses a narrower problem space with more limited cross-field applicability. Paper 1's infrastructure-level contribution has broader and more lasting impact potential.
Paper 2 addresses a universal and critical bottleneck in LLM agent development: corpus-level trace diagnostics and debugging. While Paper 1 provides a valuable domain-specific benchmark for terminal tasks, Paper 2 introduces a novel methodology applicable across any LLM agent domain, offering scalable, evidence-backed diagnostics that lead to significant downstream performance improvements. This broader applicability and methodological innovation give Paper 2 a higher potential scientific impact.
Paper 2 has higher estimated impact: it addresses a timely, widely felt bottleneck (scalable diagnosis of LLM agent failures) with a more general methodology applicable across domains and agent frameworks. Its corpus-level formalization plus evidence-grounded insight generation can influence evaluation, debugging, and MLOps practices broadly, and the reported downstream gains (e.g., +30.4pp scaffold improvement) suggest strong real-world value. Paper 1 is innovative for modular specialization/efficiency, but its impact is more concentrated in deployment/parameter-efficient adaptation, with somewhat narrower cross-field methodological implications.
Paper 2 addresses a critical bottleneck in the rapidly growing field of LLM agent development by introducing a scalable, automated diagnostic system. Its methodological rigor, demonstrated by significant objective performance improvements (30.4pp gain), gives it a stronger potential for broad and immediate technical impact compared to Paper 1's qualitative, interview-based exploration of organizational culture, which, while valuable, has a narrower methodological scope and less quantifiable impact.