CommCP: Efficient Multi-Agent Coordination via LLM-Based Communication with Conformal Prediction

Scientific Impact Assessment: CommCP

Core Contribution

CommCP addresses the problem of multi-agent coordination in embodied question answering (EQA) by introducing a decentralized LLM-based communication framework that uses conformal prediction (CP) to calibrate the confidence of inter-agent messages. The paper formalizes a new problem setting—Multi-Agent Multi-Task Embodied Question Answering (MM-EQA)—where heterogeneous robots must cooperatively explore a 3D environment and answer questions related to their individual assignments. The key insight is that agents should only communicate information they are confident is relevant to partner agents' tasks, and CP provides a principled mechanism to filter out overconfident or irrelevant LLM outputs before transmission.

The novelty lies primarily in the intersection of three components: (1) extending single-agent EQA to a cooperative multi-agent setting, (2) using natural language as an inter-agent communication protocol powered by LLMs, and (3) applying conformal prediction to calibrate communication content rather than just planning or answer confidence. While each individual component draws from existing work (EUC [2] for single-agent EQA with CP, prior LLM multi-agent frameworks), their combination into a coherent communication framework for embodied tasks is genuinely novel.

Methodological Rigor

The methodology is generally well-structured, with clear module decomposition (perception, communication, planning, confidence check). The use of split conformal prediction to calibrate LLM outputs for message generation is technically sound in principle. However, several aspects warrant scrutiny:

CP Application: The i.i.d. assumption required for CP is argued through seven conditions, but some are debatable. Claim 3 ("LLM's probability outputs for different object pairs are not causally related") is questionable given that LLMs maintain contextual dependencies, and the argument that "each robot generates semantic values based solely on its current view, without considering past frames" (condition 7) seems like a design constraint rather than a natural property. The calibration set is drawn from only 20 HM3D scenarios, which may be thin for robust CP guarantees, though the authors acknowledge that ~1/δ samples suffice.

Experimental Design: The benchmark of 420 questions across 70 scenarios is reasonable but modest. The use of Prismatic-VLM-13B and LLaMA3-8B-instruct (rather than stronger models) is justified by the need for probability outputs, but this limits the absolute performance ceiling. The baselines (MMFBE, MMEuC) are appropriate but relatively simple—there are no comparisons against other LLM-based multi-agent communication methods adapted to this setting.

Evaluation Metrics: Success Rate and Normalized Time Cost are appropriate. The analysis across scene sizes, communication latency, and team sizes adds depth. However, the paper lacks statistical significance tests, and standard deviations appear large in several conditions.

Potential Impact

The paper addresses a practical need in multi-robot systems: how to make communication between heterogeneous agents effective rather than distracting. The idea of using CP to filter communication content has broader applicability beyond EQA—any multi-agent LLM system where agents exchange information could benefit from calibrated messaging.

Real-world applications include household service robots, warehouse logistics, and search-and-rescue scenarios where multiple robots must coordinate information gathering. The framework's decentralized nature is important for scalability and robustness.

Broader field influence: The paper contributes to the growing intersection of LLMs and multi-agent robotics. The specific insight that uncalibrated LLM communication can be worse than no communication (as shown by Ours-No-CP performing similarly to MMEuC) is valuable and likely generalizable. The MM-EQA benchmark, while modest in size, fills a gap in evaluation infrastructure for this problem setting.

However, the impact may be constrained by: (1) the relatively narrow evaluation setting (household environments, 2-3 robots), (2) the reliance on specific VLM/LLM architectures, and (3) the gap between simulated EQA and real-world multi-robot deployment where manipulation, localization errors, and hardware heterogeneity add complexity.

Timeliness & Relevance

The paper is highly timely. LLM-based multi-agent systems are experiencing rapid growth, and the question of how to make inter-agent communication effective (rather than just possible) is becoming increasingly important. The recognition that LLM hallucinations and miscalibration can actively harm cooperative performance addresses a real bottleneck. Conformal prediction is also gaining traction in robotics for uncertainty quantification, making this a natural and timely application.

The MM-EQA formulation captures an emerging need as service robot deployments scale from single to multiple agents. The paper correctly identifies that naive adaptation of single-agent methods to multi-agent settings is insufficient.

Strengths

1. Clear problem formulation: The MM-EQA problem is well-defined and fills a genuine gap between single-agent EQA and multi-agent cooperation.

2. Principled communication filtering: Using CP to calibrate messages is more principled than heuristic thresholding, providing statistical coverage guarantees.

3. Compelling ablation studies: The comparison between Ours, Ours-No-CP, and Ours-Com-Control convincingly demonstrates that calibrated quality matters more than quantity of communication—a key insight.

4. Scene size analysis: The finding that communication advantages grow with scene size (Fig. 5) is intuitive and well-demonstrated.

5. Scalability analysis: Extension to 3-robot teams shows the framework generalizes, and that uncalibrated communication becomes even more harmful with more agents.

Limitations

1. Limited scale: Only 2-3 robots are tested; the claim of scalability to "significantly larger robot teams" remains unvalidated.

2. Simplified communication model: The message template is rigid ("I see {relevant object} that may be relevant to your target..."), limiting the expressiveness of natural language communication.

3. No real-world experiments: All evaluation is in simulation (Habitat), with no transfer to physical robots.

4. Weak baselines: No comparison against other LLM-based multi-agent communication methods (e.g., adapted versions of [7] or [6]).

5. Limited question complexity: Multiple-choice format with four options is constrained; open-ended questions would better test the system's capabilities.

6. Calibration set concerns: 20 scenarios for calibration is minimal, and the paper doesn't analyze sensitivity to calibration set size or composition.

7. CP threshold sensitivity: The specific quantile choices (0.6 for Option A, 0.82 for Option B) appear somewhat arbitrary, and sensitivity analysis around these thresholds is absent.

Overall Assessment

CommCP makes a solid contribution at the intersection of multi-agent robotics, LLM-based communication, and uncertainty quantification. The core idea—that calibrated communication outperforms both no communication and uncalibrated communication—is well-supported and practically important. The MM-EQA benchmark, while modest, provides useful evaluation infrastructure. The paper's main limitation is its relatively constrained experimental scope and the absence of stronger baselines. Nevertheless, the principled approach to communication quality in multi-agent LLM systems addresses a genuine need and is likely to influence future work in this rapidly growing area.