Knowing When to Ask: Self-Gated Clarification for Hierarchical Language Agents

Paper 2 introduces a novel and broadly applicable concept—integrating clarification as a first-class action within hierarchical reasoning agents—that addresses a fundamental limitation in LLM-based agents across many domains. The self-gated clarification mechanism with mandatory/opportunistic modes offers a new framework for agentic AI. Paper 1, while technically rigorous with formal guarantees for adversarial attacks on data summarization, addresses a narrower problem. Paper 2's relevance to the rapidly growing LLM agent ecosystem, its evaluation across 9 LLMs and 4 families, and its potential to influence how autonomous agents handle uncertainty give it broader impact potential.

Paper 2 is more likely to have higher impact: it introduces a broadly applicable decision-theoretic framing (clarification as an explicit competing action via ACTION-RATING) plus interpretable emergent modes and a diagnostic (ISE) that can generalize across hierarchical agents and domains. It is timely for agent reliability and aligns with real deployments where “when to ask” matters. Paper 1 is a valuable benchmark, but its impact may be narrower (evaluation-focused, specific synthetic artifact design) and more dependent on subsequent adoption, whereas Paper 2 provides a transferable mechanism and evaluation signals with clearer immediate utility.

Paper 1 introduces a novel conceptual framework (ACTION-RATING) that integrates clarification directly into an agent's action space, addressing a fundamental challenge in hierarchical reasoning agents. Its contributions—emergent information-seeking modes, the ISE diagnostic, and the empirical separation between help localization and help quality—are more theoretically novel and broadly applicable beyond the specific domain. Paper 2 makes a solid but more incremental contribution by applying contrastive preference optimization with synthetic traces to multi-table QA, a useful but narrower methodological combination of existing techniques.

Paper 2 addresses a fundamental bottleneck in LLM agents—knowing when to ask for clarification during complex hierarchical reasoning. By integrating clarification directly into the action space, it significantly improves agent reliability and decision-making. This methodological advancement has profound, cross-disciplinary implications for deploying autonomous agents in any domain. While Paper 1 provides a valuable tool for spatial data mining, Paper 2's fundamental contribution to AI agent architecture offers a broader and more timely scientific impact.

Paper 2 likely has higher impact due to a more broadly applicable and practically consequential framework: claim-level market aggregation plus program synthesis/verification targets a common failure mode (silent numerical errors) in high-stakes domains. It demonstrates strong, multi-benchmark results with a fixed backbone and includes code/data, supporting rigor and reproducibility. The approach can generalize beyond finance to any grounded numerical/tabular/multimodal reasoning task. Paper 1 is novel in modeling clarification as an action and provides good diagnostics, but is more specialized (hierarchical taxonomy navigation) with narrower immediate application.

Paper 2 addresses a well-defined industrial optimization problem (open-pit mine scheduling) with clear practical impact, demonstrating that LLM agents guided by simulators can recover 94-99% of MILP-optimal NPV while scaling linearly. This has broad implications for applying LLMs to combinatorial optimization across industries. Paper 1, while methodologically interesting in its treatment of clarification as an action within hierarchical agents, addresses a narrower domain (tariff classification) with diagnostic metrics rather than deployment-ready results. Paper 2's combination of practical applicability, scalability demonstration, and cross-domain transferability of the framework gives it higher potential impact.

Paper 1 is more methodologically innovative and broadly relevant: it reframes clarification as an explicit competing action via a shared ordinal action-rating scale, yields emergent interpretable modes (mandatory vs opportunistic), introduces a local diagnostic (ISE), and tests separability between help-seeking localization and answer quality—supporting a more general theory of agent uncertainty/interaction. Its evaluation on a large 30k-node taxonomy with multiple LLM families strengthens rigor and generalizability. Paper 2 is application-driven and timely for industry, but its DMAIC-inspired orchestration and judge model are more incremental and narrower in cross-field impact.

Paper 1 addresses a fundamental bottleneck in AI agents—recognizing uncertainty and seeking clarification—by integrating help-seeking directly into the action space. This approach has broad, cross-disciplinary implications for improving the reliability and safety of autonomous LLM agents. In contrast, Paper 2 presents a valuable but highly domain-specific dataset and pipeline for architectural floor plans, which has practical industry applications but narrower foundational scientific impact.

ComBench addresses a critical gap in evaluating frontier LLMs on Olympiad-level combinatorics, a domain central to mathematical AI research. Its benchmark design separating analysis and construction reasoning provides novel diagnostic insights about distinct model capabilities. The benchmark serves the rapidly growing field of mathematical reasoning in LLMs and will likely be widely adopted. Paper 2, while methodologically interesting in its self-gated clarification approach for hierarchical agents, targets a narrower application domain (tariff classification) and has more limited generalizability despite its sound experimental design.

Role-Agent presents a more broadly applicable framework for improving LLM agents across diverse tasks through a novel dual-role co-evolution mechanism (agent as both actor and environment simulator). Its contributions—process reward via state prediction alignment and failure-mode-driven curriculum reshaping—are generalizable ideas applicable across many agent domains. Paper 1, while methodologically rigorous, addresses a narrower problem (clarification in hierarchical classification, specifically tariff codes) with domain-specific evaluation. Paper 2's broader applicability, novel training paradigm, and potential to influence the wider LLM agent research community give it higher estimated impact.