When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning
Chirag Parmar, Akshat Mehta, Henglin Wu, Jagadish Ramamurthy, Shweta Medhekar
Abstract
When does multi-agent debate help data cleaning, and when does it hurt? Across three benchmarks, four model families, and over 6,000 task-condition pairs, we find debate's effect reverses sign: it degrades generation across all four models (-1.6 to -15.5pp) through critique-induced confusion (CIC), hallucinated Critic feedback that the Generator accepts uncritically, yet improves error detection (+27.4pp F1, d=1.0). We derive a debate benefit condition: debate helps when the probability of rescuing a wrong output (Critic verification odds weighted by fixability) exceeds the probability of destroying a correct one. A factorial experiment proves adversarial separation is essential: self-verification with identical tools fails, while a separate Critic with code-execution grounding and evidence-gated generation produces the first debate configuration to significantly exceed single-agent on a generative task (+5.3pp, p<0.05). The condition correctly predicts all nine task types and generalizes with zero false positives across 19 published comparisons in seven domains.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution
This paper addresses a critical and timely question: under what conditions does multi-agent debate improve versus degrade LLM performance on data cleaning tasks? The paper makes three interrelated contributions: (1) identification and characterization of critique-induced confusion (CIC), where hallucinated Critic feedback is uncritically accepted by the Generator, degrading output quality on generative tasks; (2) a formal debate benefit condition — debate helps when the Critic's verification odds weighted by fixability exceed the Generator's baseline accuracy odds (pc/(1−pc)×pr > pg/(1−pg)); and (3) an empirical demonstration that code-execution grounding combined with evidence-gated generation produces the first debate configuration to significantly exceed single-agent performance on a generative data cleaning task (+5.3pp, p<0.05).
The conceptual framing is the paper's strongest intellectual contribution. By decomposing debate effectiveness into three estimable quantities (pg, pc, pr), the authors transform an empirical observation (debate sometimes helps, sometimes hurts) into a principled, deployable decision rule. The connection to the classical data management distinction between error detection and error repair is elegant and well-motivated.
Methodological Rigor
The experimental design is notably thorough. The study spans three benchmarks (AutoDCWorkflow, MMTU, MaTElDa), four model families (Claude 4 Sonnet, Gemini 3.1 Pro, Qwen3 235B, DeepSeek R1), and over 6,000 task-condition pairs. Statistical methods are appropriate: paired bootstrap CIs with Holm-Bonferroni correction, Cohen's d effect sizes, Hartigan's dip test for bimodality, and explicit power analysis.
The factorial design crossing adversarial separation × tool augmentation (Section 7.3) is particularly well-constructed, revealing an epistatic interaction: neither adversarial separation alone nor tools alone improve quality, but their combination produces a significant +12.3pp swing. The self-consistency control (k=5 majority vote performing *worse* than single-agent) effectively rules out the multi-sample explanation for debate's detection benefits.
However, several methodological concerns temper the findings. The main experiments use small tables (5–100 rows), far from production scale. The scale validation (n=15 tables at 10K rows) is preliminary and limited to one model. The deterministic executor covers ~20 operations with fuzzy matching, potentially missing semantic quality differences. Cell accuracy is reported only for Claude, limiting cross-model comparisons on generative tasks. The Gemini results are somewhat confounded by output formatting issues (87% zero-F1 from unparseable JSON), and n=20 for Gemini's AutoDCWorkflow experiments provides limited statistical power.
The debate benefit condition's parameters (pg, pc, pr) are estimated qualitatively (e.g., pc = "high" or "low") rather than measured precisely, which somewhat undermines the formal apparatus. The condition correctly predicts all nine task types, but this is in-sample — the cross-domain validation against 19 published comparisons uses post-hoc estimation of pc and pr from task descriptions, not measured values.
Potential Impact
Practical impact is substantial. The paper provides an actionable decision rule for practitioners deploying multi-agent systems in data pipelines: use debate for detection tasks with low baseline accuracy and high verifiability; avoid it for open-ended generation where it wastes 4–7× compute while degrading quality. The evidence-gated generation mechanism (G2) offers a concrete architectural pattern for managing Critic compliance.
Theoretical impact is moderate but meaningful. The debate benefit condition provides a bridge between the scalable oversight literature (Irving et al., 2018) and practical deployment, operationalizing when "verifier efficiency" holds. The identification of CIC as a structural phenomenon (not addressable through prompt engineering alone) is a useful negative result for the multi-agent community.
Broader influence: The finding that per-item verifiability, not task domain, determines debate effectiveness could influence multi-agent system design across NLP, code generation, and scientific reasoning. The PIV (per-item verifiability) scoring rubric (Appendix M) provides a lightweight assessment tool.
Timeliness & Relevance
This work addresses an acute need. Multi-agent LLM architectures are being deployed in production data pipelines without principled guidance on when they help. The concurrent work by Zhang et al. (2025) reaches similar conclusions about task structure determining debate effectiveness, but this paper adds the mechanism (CIC), the formal condition, and the fix — a more complete contribution. The paper also arrives at a moment when data quality automation is a high-priority enterprise concern.
Strengths
1. Comprehensive experimental coverage: Four models, three benchmarks, 6,000+ task-condition pairs, with appropriate statistical controls.
2. The sign reversal finding is striking and well-documented: debate improves detection (+27.4pp F1, d=1.0) while degrading generation across all four models.
3. The factorial design cleanly demonstrates the epistatic interaction between adversarial separation and tool grounding.
4. The CIC mechanism is convincingly characterized through transcript analysis (95.3% compliance rate, 51.9% hallucinated operations).
5. Practical actionability: The three-question decision rule is immediately deployable.
Limitations
1. Scale gap: Production data cleaning involves tables with millions of rows, multi-table joins, and streaming updates — none tested here.
2. Qualitative parameter estimation: The debate benefit condition's predictive power relies on qualitative assessment of pc and pr, limiting its precision as a quantitative tool.
3. Topology scope: Only two-agent Generator-Critic topology tested; multi-agent panels may behave differently.
4. Cross-domain validation is retrospective: The 19 published comparisons are assessed post-hoc, not prospectively. Zero false positives in 19 comparisons is suggestive but not definitive.
5. Generative task fix is limited: The +5.3pp improvement from D-Code+G2 is significant but modest, and demonstrated only for Claude on one benchmark.
6. No fine-tuning or training-based approaches are compared; all results use prompting only.
Additional Observations
The bimodality analysis (debate as a variance amplifier) is an underappreciated insight with practical implications: even when mean performance is unchanged, debate increases tail risk. The compliance mechanism analysis (95.3% agreement rate) reveals a fundamental limitation of current LLMs in multi-agent settings — they are too deferential to adversarial feedback, a finding that connects to the broader sycophancy literature.
The paper is well-written and clearly structured, though dense. The appendices are extensive and support reproducibility. The work would benefit from a cleaner separation between confirmed findings (MaTElDa detection, p<0.001) and directional evidence (MMTU results, most AutoDCWorkflow comparisons).
Generated Jun 3, 2026
Comparison History (21)
Paper 1 addresses a fundamental bottleneck in LLM reasoning—the scarcity of high-quality process supervision data—with a novel theoretical framework (LC-ERD) combining variational logic potentials and multi-agent value decomposition. This tackles a core challenge in the rapidly growing field of LLM self-improvement, with broad implications for reasoning capabilities across domains. Paper 2 provides valuable empirical insights into multi-agent debate for data cleaning, including a useful debate benefit condition, but its scope is narrower (data cleaning) and more empirical than foundational. Paper 1's methodological contributions have greater potential to influence the broader trajectory of LLM reasoning research.
Paper 2 likely has higher scientific impact due to stronger generality and rigor: it studies multi-agent debate across many task-condition pairs, multiple model families, and benchmarks, identifies a concrete failure mechanism (critique-induced confusion), and derives a predictive “debate benefit condition” validated experimentally and via meta-generalization to 19 published comparisons. This yields a broadly applicable theory and design principle for multi-agent systems beyond data cleaning. Paper 1 is timely and useful for agentic RAG reliability, but is more domain-specific (agentic RAG pipelines) and appears primarily architectural/empirical rather than offering a widely predictive, cross-domain condition.
Paper 2 is more scientifically impactful: it identifies a counterintuitive failure mode (debate hurting generation), explains the mechanism (critique-induced confusion), and provides a predictive, generalizable condition validated across benchmarks and external literature (19 comparisons, zero false positives). It also proposes and experimentally verifies a concrete fix (adversarial separation with grounding/evidence gating) that improves a practical, widely relevant task (data cleaning). This combines novelty, methodological rigor, and broad applicability across multi-agent LLM systems. Paper 1 is valuable as an evaluation benchmark, but is more domain-specific and less mechanistically general.
Paper 1 challenges prevailing assumptions about multi-agent debate by rigorously demonstrating when it fails and why. By deriving a broadly applicable mathematical condition for debate benefit and validating it across numerous domains and published comparisons, it provides fundamental insights that will heavily influence future research in multi-agent systems and LLM reasoning, giving it a higher scientific impact than the engineering-focused framework in Paper 2.
Paper 2 provides a broadly applicable theoretical framework (debate benefit condition) with rigorous empirical validation across multiple benchmarks, models, and domains. Its findings about when multi-agent debate helps vs. hurts are fundamental to the rapidly growing field of multi-agent LLM systems, with immediate practical implications for data cleaning and broader generalizability (validated across 19 published comparisons in 7 domains). Paper 1, while innovative in modeling internal cognitive states for social simulation, addresses a narrower application domain (opinion dynamics) with less generalizable contributions. Paper 2's methodological rigor (factorial experiments, predictive conditions) and cross-domain applicability give it higher impact potential.
Paper 1 has higher impact potential due to a more novel, generalizable contribution: it identifies and explains a failure mode (critique-induced confusion), proposes a predictive condition for when debate helps vs hurts, and validates it via large factorial experiments plus external generalization across 19 prior studies and multiple domains. This yields actionable design principles for multi-agent systems broadly (beyond data cleaning). Paper 2 is timely and application-relevant, but primarily contributes a benchmark/evaluation framework in one domain; its broader methodological novelty and cross-field generalization appear more limited.
Paper 2 introduces a foundational analytical framework for AI-Driven Research Systems (ADRS), a rapidly expanding frontier in AI for scientific discovery and optimization. By formalizing component interactions and challenging existing structural assumptions, it offers broader theoretical and practical applicability across multiple domains compared to Paper 1's narrower, albeit rigorous, focus on multi-agent debate mechanics.
Paper 2 has higher likely impact due to a broadly applicable, novel characterization of when multi-agent debate helps vs. harms (including a predictive benefit condition) and a rigorously validated fix (adversarial separation + execution-grounded, evidence-gated generation) across large factorial experiments and external generalization to 19 prior studies. Its applications span data cleaning, LLM reliability, and multi-agent systems across many domains, making it timely and cross-field. Paper 1 is solid but narrower in scope (hydrologic streamflow inference) and mainly reports that LSTMs outperform an encoder-only Transformer in this setting.
Paper 2 offers broader, more generalizable scientific impact: it rigorously characterizes when multi-agent debate helps vs. harms, proposes a predictive benefit condition validated across many tasks/models, and demonstrates a causally grounded fix (adversarial separation with execution/evidence gating) with statistical significance. This yields transferable principles for LLM system design, evaluation, and reliability across domains. Paper 1 is highly useful and timely for biomedical VLM training with an open-source pipeline, but its contributions are more domain- and pipeline-specific, and headline comparisons (e.g., vs GPT-5.2) may be less methodologically stable.
Paper 1 offers a broader scientific impact by addressing a fundamental question in LLM multi-agent systems (when debate helps vs. hurts). It provides a rigorous theoretical framework, extensive empirical validation across thousands of conditions, and generalizes to multiple domains. Paper 2, while valuable for formal mathematics, targets a much narrower niche (refactoring Lean proofs) and relies on subjective rubric-based evaluations, limiting its broader applicability compared to Paper 1's generalizable findings.
Paper 2 addresses a highly timely and critical issue in modern AI (LLM multi-agent systems) with broad real-world applications in data cleaning. Its extensive empirical evaluation (over 6,000 pairs, 4 model families) and actionable insights ('debate benefit condition') offer significant, immediate utility across multiple AI domains. In contrast, Paper 1 focuses on a highly specialized, theoretical area of formal logic, which, while methodologically rigorous, is likely to have a much narrower impact.
Paper 1 introduces a comprehensive, reusable benchmark framework (NovelAPIBench) addressing a fundamental challenge in LLM tool use—novel API acquisition—with systematic diagnostic analysis across multiple dimensions. Its findings about complementary roles of retrieval and fine-tuning, and the decomposition of API knowledge into actionable components, have broad implications for code generation, tool-augmented LLMs, and continual learning. Paper 2 provides valuable insights into multi-agent debate dynamics with a useful theoretical condition, but addresses a narrower problem (data cleaning) with more incremental contributions. Paper 1's methodological infrastructure and generalizable insights likely yield broader and longer-lasting impact.
Paper 1 provides a rigorous, generalizable theoretical framework (debate benefit condition) explaining when multi-agent debate helps vs. hurts, validated across 6,000+ task-condition pairs and 19 published comparisons with zero false positives. Its identification of critique-induced confusion and the adversarial separation principle offers broad methodological insights applicable across many multi-agent LLM systems beyond data cleaning. Paper 2, while technically solid, addresses a more niche problem (skill selection in LLM agents) with narrower generalizability and incremental advances over existing baselines.
Paper 1 provides a comprehensive theoretical framework explaining when multi-agent debate helps vs. hurts, validated across extensive experiments (6,000+ task-condition pairs, four model families, three benchmarks) and 19 published comparisons. It introduces a predictive debate benefit condition with zero false positives, offering broad applicability beyond data cleaning to any multi-agent system. Paper 2, while practically useful, addresses a narrower problem (safety alignment efficiency) with an incremental contribution (localized distillation). Paper 1's foundational insights into multi-agent dynamics have broader cross-field impact and stronger methodological rigor.
Paper 2 provides fundamental theoretical and empirical insights into multi-agent LLM systems, identifying mechanisms of failure (critique-induced confusion) and formulating generalizable conditions for success. This deep methodological analysis offers broader scientific impact compared to Paper 1, which primarily contributes a practical, resource-efficient evaluation tool.
Paper 1 is likely to have higher scientific impact due to a stronger mix of novelty, rigor, and broad applicability: it identifies and explains a failure mode (critique-induced confusion), derives a predictive benefit condition, validates it via a large factorial experiment, and demonstrates an improved debate configuration with statistical significance plus external generalization across 19 published comparisons. These contributions can directly influence multi-agent LLM design across many domains beyond data cleaning. Paper 2 is timely and useful as a benchmark for research-agent evaluation, but its impact may be narrower and more dependent on benchmark adoption.
Paper 1 provides rigorous theoretical guarantees for safety in Reinforcement Learning under unknown environment dynamics. By bridging the gap between theoretical shielding and practical Robust MDPs, it addresses a critical bottleneck in deploying autonomous agents in safety-critical, real-world scenarios. While Paper 2 offers a timely and empirically thorough analysis of LLM multi-agent debate, Paper 1's mathematical proofs of soundness, optimality, and PAC guarantees offer foundational advancements in AI safety that are likely to have a deeper, longer-lasting scientific impact across fields like robotics and autonomous control.
Paper 1 provides a broadly generalizable theoretical framework (debate benefit condition) validated across multiple benchmarks, model families, and 19 published comparisons in seven domains. Its findings about when multi-agent debate helps vs. hurts have implications across all LLM-based multi-agent systems, not just data cleaning. Paper 2, while rigorous and practically useful, addresses a narrower problem (chest X-ray report generation) with a domain-specific solution. Paper 1's breadth of impact, theoretical contribution, and cross-domain generalizability give it higher potential scientific impact.
Paper 1 (Code-on-Graph) addresses fundamental bottlenecks in LLM-KG integration with a novel programmatic reasoning framework that demonstrates substantial improvements (up to 10.5%) on established benchmarks. It introduces a generalizable paradigm shift from predefined operators to code-based reasoning with schema abstraction, which has broad applicability across KG-dependent tasks. Paper 2 provides valuable empirical analysis of multi-agent debate for data cleaning with a useful theoretical condition, but its scope is narrower (data cleaning) and its contributions are more diagnostic than architectural. CoG's framework has wider potential adoption across the growing LLM-KG integration field.
Paper 1 addresses a fundamental mechanism in LLM reasoning (multi-agent debate) with exceptional methodological rigor, evaluating over 6,000 pairs and deriving a broadly applicable mathematical condition. Its findings generalize across multiple domains, offering foundational insights for the rapidly growing field of AI agents. While Paper 2 presents a valuable application in drug discovery, Paper 1's theoretical contributions and cross-domain generalizability promise a wider breadth of impact across the entire AI community.