Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
Fangzhou Wu, Sandeep Silwal, Qiuyi Zhang
Abstract
Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation. Existing clustering methods, which primarily rely on semantic taxonomies or embeddings, often fail to capture such latent capability requirements due to a misalignment between surface-level semantics and actual model performance. We propose ECC, an algorithm that calibrates prior semantic embeddings using limited posterior model comparisons to bridge the gap between surface-level semantics and latent capability requirements. ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands, jointly learning a flexible, capability-aware clustering structure that supports query-specific inference of LLM capabilities. Extensive quantitative and qualitative evaluations demonstrate that ECC significantly improves LLM capability ranking quality, outperforming human-labeled and embedding-based baselines by an average of 17.64 and 18.02 percentage points, respectively, and proves effective in downstream tasks such as query routing.
AI Impact Assessments
(1 model)Scientific Impact Assessment: "Capturing LLM Capabilities via Evidence-Calibrated Query Clustering"
1. Core Contribution
The paper introduces ECC (Evidence-Calibrated Clustering), an algorithm that bridges the gap between surface-level semantic similarity of queries and the latent capability demands those queries place on LLMs. The core insight is that semantically similar queries (e.g., both labeled "Mathematics") may require vastly different capabilities, while semantically dissimilar queries may demand the same underlying skills. ECC addresses this by jointly optimizing semantic embeddings and pairwise model comparison evidence through a soft-assignment clustering framework. Each cluster is characterized by both a semantic centroid and a Bradley-Terry capability profile, and queries are assigned soft responsibilities across clusters to accommodate mixed capability demands.
The key algorithmic innovation is the alternating optimization between system parameters (centroids + BT profiles) and soft responsibilities, with an entropy-regularized objective that admits closed-form updates. The "probe-informed mixture inference" mechanism—using a single pairwise comparison at test time to calibrate cluster assignments—is a particularly elegant and practical contribution.
2. Methodological Rigor
The methodology is well-grounded. The alternating optimization scheme has a formal convergence guarantee (Theorem 2.1, with complete proof in the appendix showing monotone decrease of a bounded objective). The convexity of the BT sub-problem (Lemma C.1) ensures global optimality of each cluster's capability profile update.
The experimental evaluation is thorough: three benchmarks (SPROUT, RouterBench, Open LLM Leaderboard v2), multiple inference signal configurations, extensive ablation studies (comparisons per query, λ, temperature T, number of clusters, embedding models, tied comparisons), and structural analyses (within-cluster consistency, inter-cluster separation, cluster overlap matrices). The 17.64 and 18.02 percentage point improvements over human-labeled and embedding-only baselines are substantial and consistent.
However, some concerns exist. The evaluation metric (per-query BT loss reduction) measures how well the inferred ranking explains held-out comparisons—this is a self-consistent metric within the BT framework and may not fully capture real-world utility. The comparison with P2L uses a relatively small model (Qwen2.5-1.5B), and P2L's poor performance may partly reflect insufficient model capacity rather than a fundamental limitation. The qualitative cluster analysis, while insightful, relies on GPT-5.2 for post-hoc summarization, which introduces potential interpretation bias.
3. Potential Impact
Immediate applications: Query routing (demonstrated with 16.6% improvement over embedding-only clustering) and sample-efficient model ranking (16.3pp gain) are directly applicable to multi-LLM serving platforms. The ability to infer query-specific capability rankings from a single probe comparison has clear practical value for systems like Chatbot Arena or production routing services.
Broader methodological impact: The principle of calibrating prior semantic representations with posterior performance evidence could generalize beyond LLM evaluation. The framework could influence how benchmarks are designed—moving from fixed taxonomies to capability-aware dynamic groupings. It could also impact model selection in AutoML, ensemble methods, and personalized recommendation systems where item similarity ≠ performance similarity.
Evaluation paradigm shift: The paper challenges the prevalent assumption that human-curated benchmark categories meaningfully capture capability structure. The finding that human labels yield marginal (or even negative) gains over a global BT model is provocative and could influence how the community designs and interprets benchmarks.
4. Timeliness & Relevance
This work addresses a current bottleneck: as the number of available LLMs grows rapidly, efficient capability-aware evaluation and routing become critical. The disconnect between semantic benchmarks and actual model performance is a widely recognized but under-addressed problem. The sample efficiency aspect—requiring only ~7 comparisons per query for clustering and 1 probe at inference—is particularly timely given the computational cost of LLM evaluation.
The work sits at the intersection of several active research threads: LLM routing, adaptive testing, and capability discovery. Its timing relative to the proliferation of specialized models and routing frameworks (RouteLLM, CARROT, GraphRouter) positions it well.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Scalability considerations: The method scales linearly with |Q| and K, and the BT fitting is lightweight. However, the pairwise comparison generation cost (O(queries × comparisons)) could be significant at scale.
Additional Observations
The paper's framing around "evidence calibration" is well-motivated and distinguishes it from pure semantic or pure comparison-based approaches. The demonstration that embeddings and comparisons capture fundamentally different structures (ARI=0.021 between embedding-only and comparison-only) is a valuable empirical insight. The out-of-distribution generalization results (cross-dataset transfer) suggest the learned capability structure has genuine transferability.
Generated May 19, 2026
Comparison History (23)
Paper 1 addresses a critical bottleneck in scaling renewable energy infrastructure, offering the first benchmark of its kind for floating offshore wind turbines. Its interdisciplinary approach bridges AI surrogate modeling with mechanical engineering and climate tech, promising immense real-world impact on cost reduction and design optimization for deep-water wind energy. While Paper 2 presents a solid methodological improvement for LLM evaluation, Paper 1 stands out for its profound potential impact on global climate change mitigation and establishing a foundational dataset in an underserved domain.
Paper 2 addresses a broadly relevant problem in LLM evaluation—capturing latent capability demands through calibrated query clustering—with strong quantitative results (17-18 percentage point improvements) and clear downstream applications like query routing. Its methodological contribution (bridging semantic embeddings with performance-based calibration via Bradley-Terry models) is broadly applicable across the growing LLM ecosystem. Paper 1, while innovative in formalizing VA workflows, targets a narrower community (visual analytics), relies on a small evaluation (17 workflows), and its utility demonstration via LLM-based comparison is less rigorous. Paper 2's timeliness and breadth of impact across the rapidly expanding LLM field give it higher potential impact.
Paper 2 offers broader scientific impact by addressing the critical challenge of AI safety-by-design while simultaneously reducing inference costs by 30% via the Safe-MoMA framework. The public release of a 35B parameter safety-enhanced model checkpoint provides a highly valuable resource that will directly catalyze follow-up research across the AI community. While Paper 1 presents a strong methodological improvement for LLM evaluation and query routing, Paper 2's contributions to trustworthy foundation models and scalable, cost-efficient agentic frameworks align more closely with urgent, widespread industry and academic priorities.
Paper 1 addresses a critical bottleneck in the rapidly growing field of LLM agents: optimizing multi-field skills under real-world constraints. Its introduction of Chebyshev scalarization to navigate non-convex multi-objective trade-offs is methodologically rigorous and solves a distinct failure mode (stagnation) in existing optimizers. While Paper 2 offers strong improvements for evaluation and routing, Paper 1's potential to directly enhance autonomous agent performance gives it a broader and more timely scientific impact.
Paper 2 likely has higher impact due to direct relevance to real-world LLM deployment: multi-model scheduling, offloading, and preemption under GPU memory constraints are ubiquitous operational problems. Its empirical characterization across models/hardware can generalize into system design principles and influence production schedulers, affecting performance/cost broadly across industry and systems research. Paper 1 is novel for capability-aware evaluation and could aid benchmarking and routing, but its impact is narrower and depends on adoption of a specific evaluation framework, whereas Paper 2 addresses an immediate, cross-cutting infrastructure bottleneck.
PEEK introduces a novel and practical architectural concept—context maps as reusable orientation caches for LLM agents—addressing a fundamental gap in how agents interact with recurring contexts. It demonstrates substantial improvements across multiple tasks, architectures, and real production systems (OpenAI Codex), with strong cost efficiency gains. The breadth of applications (document reasoning, code repositories, context learning) and the generalizable framework design give it wider impact potential. Paper 2 contributes a useful evaluation methodology but addresses a narrower problem (LLM capability clustering/ranking) with more limited downstream applications.
Paper 2 addresses a critical bottleneck in the field—safe deployment of LLM agents—by proposing a foundational, mathematically grounded architectural paradigm. While Paper 1 offers a strong, empirically validated method for evaluation and routing, Paper 2's focus on system-level safety guarantees has broader implications across all domains of AI agent deployment, potentially shaping future standards and research trajectories in AI safety.
Paper 1 introduces a highly novel, cognitively-inspired framework for multi-agent LLMs, addressing a critical bottleneck in agentic AI: overconfidence and rigid role assignment. Its self-assessment and dynamic delegation mechanisms have broad real-world applications across autonomous systems, showing strong empirical gains in both accuracy and efficiency. While Paper 2 offers a valuable methodological improvement for LLM evaluation and query routing, Paper 1 represents a more fundamental architectural paradigm shift for multi-agent systems, making it highly timely and likely to spur diverse follow-up research across AI and cognitive science.
Paper 2 likely has higher scientific impact due to its large-scale, real-world dataset (130,882 images) and broad relevance across cultural evolution, computational social science, HCI, network science, and human–AI co-creation. It studies an emergent, timely phenomenon (AI-mediated cultural production) in a naturalistic setting, yielding generalizable insights (attractors, novelty/complexity trade-offs, preference paradox) that can inform theory and platform design. Paper 1 is methodologically innovative and highly applicable to LLM evaluation/routing, but its impact is more domain-specific to NLP/LLM assessment practices.
Paper 2 addresses a broadly relevant problem in LLM evaluation—capturing latent capability demands through calibrated query clustering—with clear methodological contributions (Bradley-Terry parameterization, mixture weights), strong empirical improvements (17-18 percentage point gains), and practical downstream applications like query routing. It has broader applicability across the LLM research community. Paper 1 introduces an interesting commitment-validation framework for personalization, but its narrow scope (personalized memory systems), low availability (0.49-0.60), and highly specialized formalism limit its broader impact potential.