Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
Akshay Manglik, Apaar Shanker, Kaustubh Deshpande, Jason Qin, Yash Maurya, Veronica Chatrath, Vijay S. Kalmath, Levi Lentz
Abstract
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
AI Impact Assessments
(1 model)Scientific Impact Assessment: Insights Generator
1. Core Contribution
The paper formalizes "corpus-level trace diagnostics" as a distinct problem: given a corpus of LLM agent execution traces, produce grounded natural-language insights characterizing systematic behavioral patterns, each linked to supporting evidence. The key architectural contribution is the scout-investigator decomposition — scouts propose hypotheses from sampled traces (breadth), while investigators validate them at corpus scale with statistical evidence (depth). All trace access is mediated through a stateful Python data processing layer with pre-injected analysis primitives, avoiding raw trace content flooding the LLM context.
This addresses a genuine gap: prior work focuses on single-trajectory debugging (AgentDebug, DoVer, AgentRX), fixed taxonomy classification (MAST, AgentFail), or treats traces as secondary to scaffold optimization (Meta-Harness). The IG system is the first to explicitly frame and address iterative, open-ended pattern discovery across trace populations as a first-class problem.
2. Methodological Rigor
The evaluation framework is thoughtfully designed along two axes (who evaluates × what is measured), yielding four complementary settings. This is a genuine methodological contribution.
LLM-as-a-Judge (Section 4.1): The three-layer evaluation (coverage against union-gold clusters, pairwise win rates with position-swap debiasing, per-dimension rubric scoring) is thorough. The union-gold construction — pooling findings across all systems to create a shared reference — is a reasonable approach for evaluating open-ended outputs. IG achieves 77.9% average pairwise win rate versus 62.4% for the next-best system, with quality advantages concentrated in mechanism explanation and specificity.
Human Expert Intervention (Section 4.2): This is the most compelling evaluation. The 30.4pp improvement over baseline (vs. 16.2pp for CC Subagents) with p≈0.016 is notable, and the robustness checks (leave-one-out, permutation test, mixed-effects model) are appropriate for the small sample. However, n=6 per arm is genuinely small, and the authors acknowledge this. The high ICC (0.86) indicates participant skill dominates variance, raising questions about generalizability across practitioner populations.
Human Expert-as-a-Judge (Section 4.3): This result is less decisive — aggregate ratings are nearly identical (4.25 vs 4.22), with IG leading only on depth (+0.17). The authors' explanation that humans struggle to evaluate claims across large trace sets is plausible but somewhat convenient.
Iterative Patcher Loop (Section 4.4): All report-equipped systems converge to similar terminal performance (0.81-0.84), but the pure-patcher regression to 0.58 demonstrates the value of grounded analysis input. The detailed regression analysis (12/13 regressions traced to a specific prompt edit) is convincing.
Weaknesses in rigor: The comparison systems, while spanning useful architectural variations, all use the same backbone model (Claude Opus 4.6). The study is conducted on only two benchmarks. The LLM-as-a-judge evaluation introduces circular concerns when using Claude to evaluate reports generated by Claude-based systems. Cost is substantially higher (23-38 for alternatives), which is acknowledged but not deeply analyzed in terms of cost-effectiveness tradeoffs.
3. Potential Impact
Practical impact: For organizations running LLM agents at scale, IG addresses a real workflow pain point. The insight that report depth and evidentiary grounding (not just coverage) drive practitioner effectiveness is actionable. The 14.2pp gap between report conditions suggests that investment in diagnostic quality has measurable ROI.
Methodological impact: The formalization of corpus-level trace diagnostics as a problem, and the four-setting evaluation framework, provide useful scaffolding for future work. The union-gold clustering approach for evaluating open-ended analytical outputs could transfer to other domains.
Integration potential: IG is explicitly designed to be complementary to agent optimization frameworks (VeRO, AFlow, ADAS), potentially serving as the diagnostic front-end to automated improvement loops. The patcher loop experiments demonstrate this integration pathway.
4. Timeliness & Relevance
This is highly timely. As LLM agents proliferate in production environments (coding agents, research assistants, tool-using systems), the debugging bottleneck is increasingly acute. The paper correctly identifies that evaluation dashboards showing aggregate metrics miss the cohort-level patterns that drive actionable improvements. The specific example of "silent computation failures" (code runs without errors but implements the wrong mathematical model) illustrates a failure class that is genuinely difficult to surface without corpus-level analysis.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Overall Assessment: This is a solid systems paper that formalizes an important problem, proposes a reasonable architecture, and provides multi-faceted evaluation. The human intervention result is the standout finding. The work would benefit from larger-scale validation across diverse benchmarks and practitioner populations, and from deeper analysis of when the additional cost of IG is justified over simpler alternatives.
Generated May 21, 2026
Comparison History (28)
Paper 1 (GRADE) addresses fundamental efficiency challenges in multi-agent LLM reasoning with a comprehensive framework combining gated routing, adaptive depth, and a novel training algorithm (CoGRPO). It demonstrates strong empirical results across multiple benchmarks while halving compute costs, and introduces practical innovations like hot-swappable expert registries. Its contributions span architecture design, training methodology, and inference efficiency—areas of broad interest. Paper 2 tackles the important but narrower problem of trace diagnostics, offering useful but more specialized tooling. GRADE's architectural innovations have broader applicability and potential to influence future multi-agent system design.
Paper 1 formalizes a new problem (corpus-level trace diagnostics for LLM agents) that addresses a critical gap in LLM agent development workflows. Its practical impact is significant: 30.4pp performance improvements from implementing its insights. As LLM agents proliferate in production, systematic debugging tools become essential infrastructure. Paper 2, while technically solid with its gated routing and adaptive depth approach, offers incremental improvements (4.8 points on MMLUPro) in the well-explored space of multi-agent reasoning efficiency. Paper 1 opens a new research direction with broader applicability across all agent systems.
Paper 1 investigates fundamental mechanisms of training multi-agent LLM systems with reinforcement learning, uncovering critical gradient dynamics and policy-sharing tradeoffs. This provides foundational theoretical and empirical knowledge for a rapidly growing field. In contrast, Paper 2 offers a practical diagnostic tool for LLM agent failures. While highly useful for engineering and debugging, Paper 1's insights into training stability and model architecture will likely drive broader, more foundational advances in multi-agent AI research.
Paper 2 likely has higher impact due to timeliness and broad applicability: corpus-level diagnostics for LLM agents addresses an urgent, widespread production problem and shows measurable downstream gains (e.g., +30.4pp). Its methodology includes explicit problem formalization, a scalable multi-agent hypothesis-testing system, and multi-faceted evaluation (human rubric + objective improvements), increasing adoption potential across ML engineering, agent research, and software tooling. Paper 1 is mathematically novel and rigorous, but its immediate real-world uptake may be narrower and slower despite strong theoretical contributions.
Paper 1 addresses a critical and highly timely bottleneck in AI development: diagnosing and debugging LLM agents at scale. Its automated, corpus-level approach offers broad applicability and significant real-world utility for AI researchers and practitioners, demonstrated by substantial performance gains. In contrast, Paper 2 focuses on computational gastronomy and ingredient embeddings, which, while methodologically rigorous, has a much narrower domain of application and lower potential for widespread cross-disciplinary impact.
Paper 2 addresses a more fundamental and broadly applicable problem—systematic diagnostics for LLM agents across any domain—while Paper 1 focuses on a specific application (spreadsheet automation). Paper 2's formalization of corpus-level trace diagnostics introduces a novel framework applicable to all LLM agent development, with strong empirical validation (30.4pp improvements). Its multi-agent diagnostic architecture has broader methodological impact across the rapidly growing LLM agent ecosystem, whereas Paper 1's contributions, while practical, are more incremental and domain-specific.
Paper 1 addresses a critical bottleneck in LLM agent development—diagnosing systematic failures at scale. By formalizing corpus-level trace diagnostics and providing a framework that yields actionable insights, it significantly enhances the interpretability, reliability, and real-world deployability of agentic systems. Paper 2 offers a valuable, simplified training methodology, but Paper 1's focus on diagnostics and evaluation provides broader, field-wide utility for understanding and improving complex LLM interactions.
Paper 2 is likely to have higher impact due to timeliness and broad applicability: systematic diagnostics for LLM agents addresses a rapidly expanding, cross-domain need (software engineering, ML ops, HCI, safety). The proposed formalization of corpus-level trace diagnostics and an evaluated multi-agent hypothesis-testing system with quantified downstream gains (e.g., +30.4pp) suggests strong real-world utility and measurable effect. Paper 1 is solid and relevant to circular manufacturing, but its impact is more domain-specific and architecture-centric, with narrower breadth beyond industrial informatics.
Paper 1 addresses a fundamental and broadly applicable problem—systematic diagnosis of LLM agent failures at scale—which is highly relevant given the rapid deployment of LLM agents across industries. It formalizes a new problem (corpus-level trace diagnostics), introduces a principled multi-agent architecture, and demonstrates strong empirical results (30.4pp improvement). The breadth of impact is larger since it applies to any LLM agent system, not just autonomous driving. Paper 2, while technically sound, addresses a narrower domain (collaborative driving) with incremental advances over existing communication paradigms.
Paper 2 has higher estimated impact due to strong timeliness and broad applicability: corpus-level diagnostics for LLM agents addresses a rapidly growing, cross-domain deployment need. Its proposed formalization plus a scalable multi-agent methodology can influence research in agent evaluation, observability, debugging, and human-AI tooling. The evaluation claims concrete downstream gains (e.g., 30.4pp improvements) and comparative coverage, suggesting practical utility and methodological rigor. Paper 1 is valuable for safety assurance, but its niche focus and less standard updating rule (not Bayesian) likely limits breadth and adoption relative to LLM agent diagnostics.