When Agents Disagree: Bayesian Backward Reasoning as a Label-Free Anchor for Multi-Agent Collective Decision-Making

Scientific Impact Assessment

Core Contribution. The paper addresses a real problem in multi-agent LLM systems: when heterogeneous agents disagree, existing aggregation methods (voting, electoral rules, LLM-as-judge) all operate within "forward reasoning" — mapping evidence to labels along the same conditioning path as the agents they aggregate — and therefore inherit correlated errors. The central novelty is constructing a reverse posterior via Bayesian backward reasoning (an explicit likelihood P(e|d) times a prior P(d|a)) that offers a differently-factorized approximation of the same class posterior. Cross-path consistency, measured by Jensen-Shannon divergence between forward and reverse posteriors, then drives three progressively richer aggregation strategies (MinJS hard selection, FwdJS soft reweighting, LogLin log-linear fusion). The most conceptually interesting claim is the decoupling of *anchor utility* from *standalone accuracy*: the reverse posterior is often the weakest standalone predictor yet the most useful reference, because its errors are less collinear with the forward pool.

Methodological Rigor. The experimental design is reasonably careful for its scope. Baselines are appropriate and comprehensive (random selection, seven electoral rules from GEDI, two LLM judges). The ablations are the paper's strongest element: replacing R with the pool-mean forward posterior (MeanF) or a pool-external forward agent (GenF) consistently degrades performance, isolating the value of cross-path diversity rather than the log-linear operator alone. The label-collision statistic π and Matthews-φ analyses provide mechanistic support for the "errors collide less often" hypothesis. However, rigor is capped by a single dataset (DDXPlus, a synthetic closed-set diagnostic benchmark), no statistical significance testing or error bars (except pooled random seeds), and modest effect sizes (1.2–4.7pp on the Disagree subset). The reverse construction relies on hand-designed ordinal likelihood and context-activation maps whose specification is only partially exposed.

Potential Impact. Multi-agent LLM aggregation is an active area, and the framing of "forward vs. reverse factorization" is a genuinely fresh angle that could seed follow-up work on structurally-diverse references. The label-free, training-free nature lowers adoption barriers. However, the method is constrained to closed-set, evidence-to-label tasks with a discrete candidate space; it does not extend to open-ended generation. The single-round, same-backbone evaluation and reliance on one synthetic medical benchmark limit demonstrated breadth. The reverse construction's dependence on domain-specific ordinal maps (contextual vs. remaining evidence decomposition) may not transfer cleanly to other domains without re-engineering.

Timeliness & Relevance. Highly timely. Collective decision-making among disagreeing LLM agents is a recognized bottleneck, and the paper directly engages the correlated-error problem that voting and judge-based methods suffer. The citations (many dated 2025–2026) situate it at the current frontier.

Strengths. (1) A conceptually novel and well-motivated idea — using Bayes' theorem's dual factorization to inject structural diversity into aggregation. (2) The anchor-utility-vs-accuracy insight is counterintuitive and well-supported by ablations. (3) The three-tier method progression (selection → reweighting → fusion) is pedagogically clean and shows consistent gains. (4) The optional two-stage calibration extension adds practical value when labels exist.

Limitations. (1) Single dataset severely limits generalizability claims; the authors acknowledge this. (2) Gains are modest and diluted on the full "All" pool. (3) The reverse posterior requires domain-specific evidence decomposition (a→d→e) and ordinal maps, raising portability concerns. (4) Additional inference cost of one reverse inversion per instance. (5) No code release mentioned, and the reverse construction ("ordinal likelihood and context-activation maps," "exact replay on the Bayesian network") is under-specified for independent reproduction. (6) The theoretical justification is explicitly heuristic — the authors concede no exact Bayes identity links F_i and R, so the whole edifice rests on the empirical claim that errors are "less collinear."

Other Observations. The honesty about the method being a heuristic consistency signal rather than a correctness certificate is commendable and scientifically appropriate. The φ and π analyses reflect careful thinking about *why* the method works, not just *that* it works. However, the leap from a synthetic diagnostic benchmark to broader "collective decision-making" claims is not yet earned. The class-specific bias upper-bound experiment (Appendix B) usefully diagnoses that much calibration gain comes from class-prior adjustment rather than routing quality — a candid piece of self-critique.

Overall, this is a solid, thoughtfully-executed paper with a genuinely original conceptual angle, but its impact is bounded by narrow empirical validation (one dataset, closed-set only), modest effect sizes, and portability questions around the domain-specific reverse construction. It is likely to be cited and built upon within the multi-agent LLM aggregation subfield as an interesting alternative to voting/judging, but is unlikely to change practices broadly in its current form.