ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics

ComBench addresses a critical gap in evaluating frontier LLMs on Olympiad-level combinatorics, a domain central to mathematical AI research. Its benchmark design separating analysis and construction reasoning provides novel diagnostic insights about distinct model capabilities. The benchmark serves the rapidly growing field of mathematical reasoning in LLMs and will likely be widely adopted. Paper 2, while methodologically interesting in its self-gated clarification approach for hierarchical agents, targets a narrower application domain (tariff classification) and has more limited generalizability despite its sound experimental design.

Paper 2 offers broader real-world applications and interdisciplinary impact by applying structured LLMs to conflict resolution. Its rigorous human-subject experiments comparing AI against professional human mediators demonstrate immediate practical utility in economics, law, and psychology. In contrast, while Paper 1 provides a valuable benchmark for mathematical reasoning, it contributes primarily to a crowded subfield of AI evaluation and may have a shorter lifespan of relevance as models rapidly evolve.

While Paper 1 presents a valuable human-in-the-loop framework for engineering simulations, Paper 2 addresses a fundamental and highly active area in artificial intelligence: advanced mathematical reasoning in LLMs. By introducing a rigorous benchmark for Olympiad-level combinatorics, Paper 2 has a significantly broader potential impact across the vast AI research community, as evaluating and improving frontier models' reasoning capabilities is currently a central bottleneck in global AI development.

Paper 1 addresses the practically important and timely problem of LLM-based office automation with a large-scale, well-structured benchmark (200 tasks, 7,118 criteria) that directly evaluates real-world productivity software use. This has broader impact across industry and research, as office automation affects millions of users. Paper 2, while rigorous, targets a narrower niche (Olympiad combinatorics reasoning) with only 100 problems. Paper 1's findings on the significant gap between LLM capabilities and reliable document automation have more immediate implications for the rapidly growing LLM agent deployment ecosystem.

Paper 2 likely has higher scientific impact: it introduces a new, broadly useful benchmark (ComBench) targeting a timely, unsolved weakness in LLMs—rigorous proof reasoning and constructive combinatorics—with a careful evaluation protocol (rubric-guided grading + deterministic verification). Benchmarks often catalyze progress across many labs and subfields (LLM evaluation, reasoning, formal methods, education), and the results indicate it is not saturated. Paper 1 is a solid domain adaptation study but is more incremental (standard LoRA/NEFTune) with limited dataset size and narrower applicability (financial NER).

ActiveMem proposes a novel architectural framework addressing a fundamental limitation in LLM agents (context overload) by decoupling reasoning and memory. This innovation has broad applicability across numerous long-horizon tasks and agentic workflows, offering a scalable solution to improve efficiency and capability. While ComBench provides a rigorous evaluation for a specific niche (combinatorics), ActiveMem's contribution directly enhances model capabilities, promising wider real-world applications and higher overall scientific impact across AI research.

ComBench offers a concrete, reproducible benchmark with empirical results on frontier LLMs, addressing a well-defined gap in evaluating combinatorial reasoning. It provides actionable diagnostics and a clear evaluation protocol. Paper 2 proposes a speculative theoretical framework ('Soul Computing') with vague claims about AI consciousness, lacking empirical validation, rigorous methodology, or falsifiable hypotheses. Paper 1's methodological rigor, practical utility for the active LLM evaluation community, and timeliness give it substantially higher scientific impact potential.

ComBench addresses a timely and high-impact topic—benchmarking LLM reasoning capabilities in mathematical problem-solving. It introduces a novel benchmark with careful evaluation methodology distinguishing proof reasoning from constructive realization, relevant to the rapidly growing AI/LLM research community. Its breadth of impact across AI, mathematics education, and reasoning research is substantial. Paper 1, while competent, addresses a more incremental improvement in fault diagnosis using belief rule bases, a narrower domain with less transformative potential and smaller research community engagement.

Paper 1 addresses a fundamental and practical challenge in cross-modal knowledge distillation without paired data, providing both theoretical foundations and a principled algorithmic framework with guarantees. It has broader real-world applicability across multimodal AI systems where paired data is scarce. Paper 2, while valuable as a benchmark for evaluating LLM combinatorial reasoning, has narrower impact primarily within the LLM evaluation community and will likely become outdated as models improve. Paper 1's theoretical contributions on distributional alignment are more enduring and broadly applicable across machine learning.

Paper 1 proposes a novel, domain-agnostic framework for self-bootstrapping LLM agents, offering broad applicability across various autonomous tasks. Its algorithmic innovations (WIA and AIW) directly address core challenges in agent generalization and learning efficiency. In contrast, Paper 2 introduces a highly rigorous but domain-specific benchmark for combinatorics. While valuable for evaluation, Paper 1's general methodological advancements in agent architecture give it a significantly higher potential for widespread adoption and real-world impact across diverse fields.