HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs
Yansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu
Abstract
Hybrid-reasoning large language models (LLMs) expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality against inference cost. However, existing methods for adaptive thinking-mode selection are typically evaluated under different models, datasets, and implementation assumptions, making it difficult to compare their practical behavior. We introduce HRBench, a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs. HRBench organizes the design space along two axes: three switching strategy families, prompt-based selection, external routing, and speculative execution, and four training regimes, training-free, SFT, offline and online RL, yielding 12 controlled evaluation settings. We evaluate these settings across 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, while reimplementing 12+ representative prior methods within the same pipeline. Our analysis characterizes how different switching strategies occupy distinct effectiveness-efficiency trade-off regions: prompt-based methods often provide favorable token-accuracy trade-offs, routing methods offer more stable cost reduction, and speculative methods tend to improve accuracy at higher token cost. We further find that training affects strategies differently, and that the preferred strategy varies with model scale and task domain. HRBench provides reference implementations and a unified evaluation platform to support more controlled research on efficient reasoning in hybrid-reasoning LLMs. Our data, code and repository are available at https://github.com/usail-hkust/HRBench.
AI Impact Assessments
(1 models)Scientific Impact Assessment: HRBench
1. Core Contribution
HRBench addresses a genuine and timely gap: the lack of a unified evaluation framework for comparing adaptive thinking-mode switching strategies in hybrid-reasoning LLMs. The paper organizes the design space along two orthogonal axes—three strategy families (Prompt-Tuning, Routing, Speculative) and four training regimes (training-free, SFT, DPO, GRPO)—yielding 12 controlled evaluation settings. It reimplements 12+ prior methods within a single pipeline and evaluates across 6 LLMs (2B–1.1T) and 5 benchmarks spanning math, science, and code.
The core problem solved is comparability: prior methods were evaluated under incompatible conditions (different models, datasets, decoding configs), making it impossible to draw reliable conclusions about relative merit. HRBench standardizes these comparisons and provides a taxonomy that organizes a rapidly growing but fragmented literature.
2. Methodological Rigor
Strengths in design: The 3×4 taxonomy is clean and principled. The controlled evaluation pipeline—same decoding parameters, same metrics, same data splits—is essential for fair comparison and is well-executed. The use of verl for training and vLLM for inference reflects current best practices.
Limitations in execution: Training-based evaluations are limited to Qwen3.5-9B only, which significantly weakens claims about training-scale interactions. The paper acknowledges this but it remains a substantial gap—the most interesting findings about scale-dependent strategy ranking (Findings 4-5) come from training-free evaluations only, leaving open whether training changes the scale-dependent picture.
The reimplementation of external methods introduces potential confounds. While the authors document deviations, some methods (e.g., AdaptThink, ADR) required retraining on different data than originally used, which may not perfectly capture the original method's behavior. The paper is transparent about this, which is commendable.
Statistical rigor is limited: results are reported without confidence intervals or significance tests, despite the stochastic nature of generation. For some benchmarks (AIME with only 30 problems), individual accuracy differences could easily be within noise. The claim that SPEC "surpasses PT at 20B" rests on a 36.8% vs 32.9% difference averaged over diverse benchmarks with small sample sizes.
3. Potential Impact
Direct practical value: For practitioners deploying hybrid-reasoning LLMs, HRBench's findings provide actionable guidance—e.g., PT for math tasks, SPEC for code, RT for stable cost reduction. The released code and unified platform lower the barrier for future research.
Research infrastructure: The benchmark fills an infrastructure gap similar to what GLUE/SuperGLUE did for NLU or HELM for holistic LLM evaluation. As hybrid-reasoning LLMs proliferate, a standardized evaluation framework becomes increasingly valuable.
Breadth of influence: The work is somewhat narrowly focused on the specific problem of thinking-mode switching. It doesn't contribute new methods per se—its value is organizational and empirical. The findings, while useful, are largely descriptive rather than explanatory (e.g., *why* does SPEC surpass PT at 671B?).
4. Timeliness & Relevance
This is highly timely. Hybrid-reasoning LLMs (Qwen3.5, gpt-oss, DeepSeek-V3.1) are very recent, and the question of efficient inference-time compute allocation is a current bottleneck for deployment. The paper addresses a real need in a rapidly evolving space. However, this timeliness is double-edged: the landscape may shift quickly, potentially dating the specific empirical findings even as the framework remains useful.
5. Strengths & Limitations
Key strengths:
Notable weaknesses:
6. Additional Observations
The paper's findings, while intuitive in retrospect, provide empirical grounding for what was previously anecdotal. Finding 1 (PT achieves Pareto-optimal trade-offs) and Finding 8 (RT benefits most from training) are particularly useful for guiding future method development. However, the "no single strategy dominates" conclusion, while accurate, is somewhat expected and doesn't push the field toward novel solutions.
The failure case analysis in Appendix C is valuable but underdeveloped—expanding this into a systematic error taxonomy could increase impact.
Generated May 28, 2026
Comparison History (20)
Paper 1 addresses the highly critical and timely challenge of balancing reasoning effort and inference cost in hybrid-reasoning LLMs. By providing a comprehensive, unified benchmark (HRBench) for adaptive thinking-mode selection, it establishes foundational infrastructure for a rapidly growing field. This standardizes evaluation across a fundamental paradigm shift in modern AI, making it likely to attract widespread adoption and citations, whereas Paper 2 focuses on a more specialized security threat within multi-agent systems.
HRBench addresses a broadly relevant and timely problem—efficient reasoning in LLMs—which impacts the entire NLP/AI community. It provides a comprehensive benchmark with open-source code, systematic evaluation across multiple models and methods, and actionable insights about strategy selection. This type of benchmarking work tends to have high citation impact by enabling future research. Paper 1, while technically solid, addresses a narrower domain (auto-bidding in advertising) with more limited cross-field applicability. The breadth of impact and timeliness of hybrid-reasoning LLM evaluation gives Paper 2 the edge.
Paper 1 introduces a comprehensive benchmark and unified evaluation framework for hybrid-reasoning LLMs, a rapidly expanding and highly relevant area of AI research (test-time compute scaling). Benchmarks that standardize evaluation across models, datasets, and methods tend to have broad, foundational impact by shaping future research directions and providing essential baselines. While Paper 2 offers an innovative multi-agent optimization method, Paper 1's structural contribution to a critical new paradigm gives it higher potential for widespread adoption and scientific impact.
HRBench addresses a broader and more fundamental problem—benchmarking and understanding reasoning strategies across hybrid-reasoning LLMs—providing a unified evaluation framework spanning multiple strategy families, training regimes, models, and tasks. Its systematic organization of a rapidly growing design space, with 12+ reimplemented methods and reproducible infrastructure, is likely to serve as a widely-adopted community resource. SkillGrad proposes a clever optimization analogy for agent skills but addresses a narrower problem with evaluation on only two benchmarks. HRBench's breadth, timeliness given the explosion of reasoning LLMs, and infrastructure contribution give it higher potential impact.
Paper 1 offers a timely, comprehensive benchmark for a highly critical issue in modern AI: balancing reasoning quality and inference cost in LLMs. Its rigorous empirical evaluation across multiple models, datasets, and strategies ensures broad applicability and foundational value for AI researchers. In contrast, Paper 2 presents an applied architecture tailored specifically to financial investment research, which, while valuable, limits its breadth of impact and methodological generalizability compared to the broader, foundational contributions of Paper 1.
Paper 2 (HRBench) likely has higher scientific impact due to broader applicability and timeliness: it delivers a unified benchmark and controlled evaluation framework for adaptive reasoning-effort switching across multiple strategy families, training regimes, model scales, and domains (math/science/code), with open-source implementations that can standardize future comparisons. This kind of infrastructure often becomes a community reference point and enables downstream methodological advances. Paper 1 is innovative and impactful within clinical RAG-RL, but its domain specificity and reliance on medical benchmarks likely limit breadth compared to a general-purpose evaluation platform.
OpenURMA presents a novel clean-room open implementation of Huawei's Unified Bus protocol, addressing fundamental datacenter RDMA bottlenecks with concrete hardware results (4.37x latency reduction, 2.80x throughput improvement). It opens a previously closed architecture for community research, enabling reproducible exploration of a potentially transformative networking paradigm. HRBench, while a solid benchmarking contribution for hybrid-reasoning LLMs, is primarily an evaluation framework that organizes existing methods rather than introducing fundamentally new capabilities. OpenURMA's hardware-level innovation with real synthesis results has broader and deeper impact across systems, architecture, and networking fields.
Paper 1 (HRBench) has higher likely impact because it introduces a unified, controlled benchmark and reimplementation suite for a timely, widely relevant problem: adaptive reasoning-cost control in hybrid-reasoning LLMs. Its methodology spans multiple strategy families, training regimes, model scales (2B–1.1T), and diverse reasoning tasks, enabling reproducible apples-to-apples comparisons that can standardize future research and system design. Paper 2 is useful for scientific figure generation, but its scope is narrower and more application-specific, with less cross-field influence than a foundational evaluation framework for efficient reasoning.
Paper 2 addresses a highly timely and critical challenge in the rapidly expanding field of Large Language Models: optimizing inference cost versus reasoning quality. By providing a comprehensive benchmark and evaluation framework (HRBench) across multiple models and domains, it offers broad applicability and will likely be widely adopted by researchers working on LLM efficiency. Paper 1 is a solid contribution to heuristic search and classical planning, but its impact is confined to a narrower, more mature subfield compared to the explosive growth and broad relevance of LLM reasoning strategies.
Paper 2 has higher potential scientific impact due to greater novelty and cross-disciplinary breadth: it links emergent LLM representation geometry to human perceptual organization across multiple modalities, offering mechanistic insight relevant to cognitive science, neuroscience, interpretability, and representation learning. Its findings (transient, layer-wise emergence/attenuation profiles) can generalize across models and inspire new analysis tools. Paper 1 is timely and useful for practitioners (efficiency benchmarking), but is primarily an engineering/evaluation framework with more incremental conceptual contribution and narrower scientific reach.
HRBench addresses a timely and broadly relevant problem—adaptive reasoning strategies for hybrid-reasoning LLMs—affecting the rapidly growing LLM community. It provides a comprehensive benchmark with 12+ methods, 6 models, and 5 benchmarks, offering a unified evaluation framework that can guide future research on efficient inference. Paper 2 tackles an important but more niche problem (complex query answering over KGs with multiple free variables). While methodologically sound, its impact is limited to the KG reasoning community. Paper 1's broader applicability to the mainstream LLM efficiency research and its timeliness give it higher potential impact.
Paper 2 (DenoiseRL) likely has higher scientific impact due to a more novel, broadly applicable training paradigm: using RL to learn from weak-model failures/noisy prefixes without relying on stronger teachers or curated datasets. This addresses a major scalability bottleneck in reasoning-model training, with clear real-world applicability and timeliness for post-training LLMs. If results are strong, the approach could generalize across tasks/models and influence RL-based alignment and reasoning research. Paper 1 is valuable infrastructure/benchmarking, but primarily consolidates and compares existing switching strategies, typically yielding narrower conceptual novelty.
Paper 1 likely has higher scientific impact due to its methodological rigor and broad, reusable contribution: a unified benchmark/framework (HRBench) that standardizes evaluation across models, tasks, and switching/training regimes, plus reimplementations of many prior methods. This enables controlled comparisons and can shape future research on efficient hybrid-reasoning and adaptive compute, affecting multiple subareas (reasoning, efficiency, training, systems). Paper 2 is timely and application-relevant for safety, but its innovations are less clearly specified and may be harder to generalize scientifically beyond the released model/framework.
Paper 1 addresses a highly critical and broad issue in modern AI: adaptive compute and reasoning efficiency in LLMs. By providing a comprehensive, unified benchmark (HRBench) for thinking-mode switching, it will likely serve as a foundational resource for the wider NLP and ML communities. In contrast, while Paper 2 presents an innovative use of Diffusion LLMs and achieves state-of-the-art results, its impact is largely restricted to the narrower subfield of Visual Speech Recognition.
Paper 2 is more novel and potentially broader-impact: it proposes an evidence-gated control plane and protocol for AI-assisted research, addressing a timely, cross-disciplinary problem (auditability and claim verification) with clear real-world applicability to computational science workflows and tooling. Its emphasis on durable state, formal transition rules, and claim-admission mechanisms could influence research practice beyond LLM evaluation. Paper 1 is rigorous and useful, but primarily advances benchmarking/analysis within hybrid-reasoning LLM efficiency, a narrower domain with more incremental novelty compared to a new research-governance/control framework.
Paper 1 introduces a comprehensive benchmark for hybrid-reasoning LLMs, addressing a critical and rapidly growing area of AI research: inference-time compute scaling and reasoning efficiency. Its systematic evaluation across diverse models and tasks provides foundational insights applicable to the broader AI community. In contrast, Paper 2 presents a valuable but more niche domain-specific dataset (medical speech), which, while practically useful, has a narrower scope of impact compared to the fundamental architectural and strategic evaluations in Paper 1.
Paper 2 likely has higher scientific impact due to stronger novelty and broader real-world relevance: it introduces a dynamic, contamination-resistant benchmark sourced from continually updated real exams and a new “Mock Exam” end-to-end evaluation that jointly scores correctness, rigor, and efficiency under realistic constraints. Its multimodal, multi-discipline scope and direct linkage to education/tutoring applications broaden cross-field impact (ML, multimodal reasoning, edtech, evaluation methodology). Paper 1 is rigorous and useful but primarily consolidates and systematizes an existing efficiency/eval design space for hybrid-reasoning mode switching, with narrower application breadth.
Paper 1 addresses a highly critical and timely challenge in LLM development: balancing inference cost and reasoning quality through hybrid-reasoning strategies. By providing a comprehensive benchmark (HRBench) across multiple models, regimes, and tasks, it establishes a foundational evaluation standard for a rapidly growing area (inference scaling). Paper 2 offers a valuable diagnostic tool for agent prompt policies, but Paper 1's focus on fundamental reasoning architectures, efficiency trade-offs, and open-source benchmarking will likely have a broader, more immediate impact on foundation model research and real-world deployment.
Paper 1 addresses a fundamental and pervasive flaw in current LLM agent research: the confounding of intrinsic model capabilities with scaffolding and environmental artifacts. By providing a unified framework with massive empirical validation (400K rollouts, 15 models) across diverse domains, it offers a crucial standardized foundation for the rapidly growing field of autonomous agents. While Paper 2 tackles an important and trendy topic (adaptive compute/hybrid reasoning), Paper 1's scope and potential to correct widespread methodological errors give it a broader and more foundational scientific impact.
Paper 2 introduces a highly novel conceptual shift by repurposing reasoning/thinking traces as context compressors. This innovative paradigm connects two critical areas of LLM research (reasoning and long-context efficiency) without requiring dedicated compression modules. While Paper 1 provides a valuable and rigorous benchmarking framework, Paper 2 offers a more foundational algorithmic insight that could broadly influence future architectures and optimization strategies for LLM inference.