Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
Tianyang Han, Hengyu Shi, Junjie Hu, Xu Yang, Zhiling Wang, Junhao Su
Abstract
Reinforcement learning with verifiable rewards has become a common way to improve explicit reasoning in large language models, but final-answer correctness alone does not reveal whether the reasoning trace is faithful, reliable, or useful to the model that consumes it. This outcome-only signal can reinforce traces that are right for the wrong reasons, overstate reasoning gains by rewarding shortcuts, and propagate flawed intermediate states in multi-step systems. To this end, we propose TraceLift, a planner-executor training framework that treats reasoning as a consumable intermediate artifact. During planner training, the planner emits tagged reasoning. A frozen executor turns this reasoning into the final artifact for verifier feedback, while an executor-grounded reward shapes the intermediate trace. This reward multiplies a rubric-based Reasoning Reward Model (RM) score by measured uplift on the same frozen executor, crediting traces that are both high-quality and useful. To make reasoning quality directly learnable, we introduce TRACELIFT-GROUPS, a rubric-annotated reason-only dataset built from math and code seed problems. Each example is a same-problem group containing a high-quality reference trace and multiple plausible flawed traces with localized perturbations that reduce reasoning quality or solution support while preserving task relevance. Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only whether a trace looks good, but also whether it helps the model that consumes it.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards"
1. Core Contribution
TraceLift introduces a planner-executor framework that separates reasoning trace generation (planner) from final artifact production (executor), training the planner to produce traces that are both intrinsically high-quality and demonstrably useful to a frozen downstream executor. The key innovation is an executor-grounded reward that multiplies a rubric-based Reasoning Reward Model (RM) score by measured executor uplift—the improvement in executor success rate when provided the reasoning trace versus operating without one. This addresses the credit-assignment problem in verifiable RL: final-answer correctness can reinforce traces that are "right for the wrong reasons."
The paper also introduces TraceLift-Groups, a 6,000-group rubric-annotated dataset where each group contains a reference reasoning trace and multiple targeted perturbations (7 error types per domain for both code and math), enabling training of a reasoning-quality judge that scores traces independently of final-answer correctness.
2. Methodological Rigor
Strengths in design: The experimental protocol is well-controlled. The executor is frozen during both training and evaluation, meaning improvements can only come from better reasoning traces. The Reasoning RM and uplift estimator are removed at test time, preventing information leakage. The paper tests across three model families (Qwen2.5-7B, Llama3.1-8B, Qwen3-4B) and two domains (code and math), with results averaged over three seeds.
Ablation quality: The ablation studies are thorough and well-structured. Removing uplift weighting (No-uplift), removing verifier anchoring (RM-uplift only), and replacing the trained RM with an LLM-as-judge all degrade performance, with clear explanations of each failure mode. The LLM-as-judge analysis revealing 95.5% score saturation is particularly informative. The K-sweep for executor comparisons (K=1,3,5) reveals a meaningful bias-variance tradeoff.
Concerns: The improvements, while consistent, are moderate in absolute terms (roughly 1-4.5 percentage points on micro-averages). The paper uses relatively standard benchmarks (GSM8K, HumanEval, MBPP) where the problems are well-studied. The theoretical analysis in Section 10, while extensive (~10 pages), primarily formalizes intuitive properties of the reward design rather than providing deep theoretical insights. The unbiasedness of the uplift estimator (Proposition 1) is straightforward, and the quality-weighted uplift interpretation (Proposition 2) follows directly from the reward definition.
The TraceLift-Groups dataset construction relies heavily on LLM-generated perturbations and LLM-based rubric judges, inheriting potential biases from these models. The RM validation (Section 7) is performed on held-out groups from the same distribution, so generalization to truly novel reasoning patterns remains uncertain.
3. Potential Impact
The conceptual contribution—that reasoning should be evaluated by its downstream utility to a consumer model, not just its surface quality—is compelling and broadly applicable. This framing could influence:
The TraceLift-Groups dataset and perturbation methodology could be reused for training reasoning evaluators in other contexts.
4. Timeliness & Relevance
This paper addresses a timely gap. The field has rapidly adopted RL with verifiable rewards (following DeepSeek-R1 and similar work), but the limitations of outcome-only supervision for reasoning quality are increasingly recognized. The paper's central insight—that correct outcomes can mask flawed reasoning—is practically important as reasoning-augmented systems become production tools. The planner-executor decomposition aligns with emerging trends toward modular, multi-stage LLM systems.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Missing comparisons: The paper does not compare against process reward models (PRMs) like Math-Shepherd or other intermediate-step supervision methods, which would strengthen the positioning.
Overall, TraceLift presents a well-motivated and cleanly executed framework that addresses a genuine gap in reasoning supervision. The conceptual contribution is stronger than the empirical gains, but the consistent improvements across models, domains, and benchmarks support the central thesis. The work opens a productive research direction in grounding reasoning quality in downstream utility rather than surface plausibility.
Generated May 6, 2026
Comparison History (30)
Paper 1 (TraceLift) addresses a fundamental limitation in RLVR-based reasoning training for LLMs—that outcome-only rewards can reinforce unfaithful reasoning. Its executor-grounded reward framework and rubric-annotated dataset introduce a principled methodology for ensuring reasoning traces are both high-quality and functionally useful. This has broad implications for multi-step reasoning systems, code generation, and math problem-solving. Paper 2 (BeliefMem) proposes a valuable probabilistic memory paradigm for LLM agents, but addresses a narrower problem. Paper 1's contribution is more timely given the explosive growth in reasoning LLMs and has broader potential to reshape training methodologies across the field.
Paper 1 is more scientifically impactful: it introduces a novel executor-grounded reward that directly addresses a central, timely problem in RL-for-reasoning—faithfulness and usefulness of intermediate traces—plus contributes a structured dataset (TRACELIFT-GROUPS) enabling learnable reasoning quality. The methodology is more rigorous (explicit reward decomposition, frozen executor uplift measurement, controlled trace perturbations) and the idea generalizes across planner-executor architectures in math/code and broader multi-step systems. Paper 2 is useful engineering for agent orchestration/IR, but is more incremental and narrower in scientific novelty.
Paper 2 introduces a foundational optimization algorithm capable of bridging the gap between SGD and Adam. Because optimizers are the bedrock of modern deep learning, a superior, theoretically-grounded alternative has universal applicability across all neural network architectures (CNNs, LLMs, diffusion models). While Paper 1 is highly timely for the specific subfield of LLM reasoning, Paper 2's potential breadth of impact is vastly larger, as it fundamentally improves the underlying training process for virtually every machine learning domain.
Paper 2 addresses a fundamental and highly timely challenge in AI: improving the faithfulness and utility of LLM reasoning traces during reinforcement learning. By introducing executor-grounded rewards, it provides a methodological advancement applicable across various domains (math, code, etc.). While Paper 1 presents an impressive and highly practical browser agent, Paper 2's focus on core reasoning and training methodologies gives it a broader potential scientific impact across the foundation model research community.
While Paper 1 introduces a highly practical and rigorous benchmark for AI agents, Paper 2 addresses a fundamental flaw in how Large Language Models are currently trained to reason via reinforcement learning. By proposing a method to reward faithful and useful reasoning traces rather than just correct final answers, Paper 2 offers a methodological innovation that could broadly influence foundation model training, alignment, and AI safety, giving it a higher potential for widespread scientific impact.
Paper 1 has higher likely impact: it introduces a novel, timely training signal for LLM reasoning (executor-grounded rewards) plus a purpose-built dataset (TRACELIFT-GROUPS), and demonstrates improvements on widely used math/code benchmarks. The approach is methodologically richer (planner–executor setup, reward decomposition, controlled trace perturbations) and broadly applicable across LLM alignment, reasoning, tool use, and multi-step agent systems, with clear real-world relevance to reliable AI assistants. Paper 2 is valuable for HCI/CSCW and team cognition but appears more domain-specific and baseline-level in modeling.
Paper 1 addresses a critical bottleneck in modern AI: improving the fidelity of LLM reasoning traces beyond simple outcome-based rewards. Its proposed planner-executor framework and executor-grounded rewards have immense, immediate applicability in training advanced reasoning models for math, coding, and multi-agent systems. Given the explosive current interest in LLM reasoning and reinforcement learning, Paper 1 demonstrates significantly higher timeliness, broader technological applicability, and greater potential for widespread scientific impact compared to Paper 2's narrower, albeit valuable, focus on human team dialogue and shared mental models.
Paper 2 addresses a fundamental limitation in reasoning-focused RL training for LLMs—that outcome-only rewards can reinforce unfaithful reasoning. TraceLift's executor-grounded reward framework and rubric-annotated dataset introduce a principled, generalizable methodology applicable across domains (math, code, and potentially beyond). This has broader impact on the rapidly growing field of LLM reasoning training. Paper 1, while addressing an interesting application of LLMs to UAV swarm control, is more domain-specific and primarily demonstrates current LLM limitations rather than providing a broadly transformative solution.
Paper 1 has higher potential impact due to a more novel training signal (executor-grounded rewards that tie reasoning trace quality to downstream utility), a new rubric-annotated dataset (TRACELIFT-GROUPS), and broad applicability to reasoning, RLHF/RLAIF, tool use, and multi-step agent systems. The methodological contribution is substantive and likely to influence how the field evaluates and optimizes reasoning traces beyond outcome correctness. Paper 2 targets an important systems niche (on-device multi-agent KV handoff) but presents narrower evidence (limited task/scale) and explicitly notes missing ablations and runtime comparability, reducing near-term scientific impact.
Paper 2 addresses a critical, highly timely issue (AI regulation and compliance, e.g., the EU AI Act) by providing a quantitative certification framework for black-box models. Its interdisciplinary approach bridges machine learning, statistics, policy, and law, offering a practical solution with massive real-world implications for AI deployment. While Paper 1 offers a strong methodological improvement for LLM reasoning, Paper 2's potential to shape global AI safety standards gives it a significantly broader and higher scientific and societal impact.
Paper 2 likely has higher impact: it proposes a novel, generalizable training framework (TraceLift) and a new grouped, perturbation-based dataset to address a central, timely problem in LLM reasoning—faithful and useful intermediate traces beyond outcome correctness. The executor-grounded reward is methodologically concrete and broadly applicable to multi-step agentic systems in code/math and beyond. Paper 1 is important and timely as a cautionary evaluation/position on AI peer review, but its impact is more policy/process-focused and narrower in technical reuse compared to a training method that can propagate across many LLM applications.
While Paper 1 presents a strong technical advancement in LLM reasoning and reinforcement learning, Paper 2 addresses an urgent, cross-disciplinary challenge at the intersection of AI safety, statistics, and global policy. By providing a mathematically rigorous framework to satisfy imminent regulatory requirements like the EU AI Act, Paper 2 has a significantly broader potential impact, extending beyond the machine learning community into law, public policy, and enterprise compliance.
Paper 2 addresses a fundamental challenge in LLM reasoning—ensuring intermediate reasoning steps are faithful and useful, rather than just optimizing for final-answer correctness. Its proposed planner-executor framework and executor-grounded rewards have broad implications for training advanced reasoning models across diverse domains like math and code. Paper 1 offers a valuable methodological contribution for evaluating forecasting, but its scope is narrower compared to Paper 2's potential to influence core LLM training and reinforcement learning paradigms.
Paper 1 tackles a fundamental issue in LLM reasoning—rewarding correct final answers derived from flawed logic—by introducing a novel planner-executor training framework with executor-grounded rewards. Improving core reasoning capabilities has profound, broad implications across numerous domains like mathematics, coding, and general problem-solving. While Paper 2 offers a valuable and rigorous benchmarking framework for forecasting evaluation, Paper 1's contribution to fundamentally enhancing how models learn to reason provides a broader and more highly sought-after methodological advancement in AI.
Paper 1 tackles a fundamental challenge in LLM reasoning—reward hacking where models arrive at correct answers via flawed logic—by introducing a novel planner-executor RL framework and specialized dataset. This advances core AI research in reinforcement learning and intrinsic model reasoning. Paper 2, while highly practical, presents an architectural engineering solution (log-based recovery) for agent reliability, which offers narrower theoretical contributions compared to fundamentally improving how language models learn to reason.
Paper 1 addresses a fundamental limitation in RL-based reasoning training for LLMs—that outcome-only rewards can reinforce flawed reasoning traces. The TraceLift framework introduces a novel executor-grounded reward that evaluates reasoning utility rather than just correctness, which is broadly applicable to multi-step reasoning systems. This tackles a timely, high-impact problem as reasoning LLMs proliferate. Paper 2 proposes a useful but more incremental contribution combining existing techniques (GRPO, mixture-of-rewards, federated learning) for VLM alignment. While practically relevant, its scope is narrower and the novelty is more in the combination than in fundamental insights.
Paper 2 addresses a critical flaw in current LLM reasoning paradigms: the reliance on final-answer correctness for reinforcement learning. By introducing executor-grounded rewards to validate intermediate reasoning steps, it offers a fundamental methodological advancement for training reasoning planners. While Paper 1 provides valuable tools for agent safety benchmarking, Paper 2's focus on the core training mechanisms of reasoning models gives it broader potential impact and applicability across foundational AI research.
ResearchEVO presents a fundamentally novel end-to-end framework for automated scientific discovery and documentation, combining LLM-guided algorithm evolution with autonomous paper writing. Its breadth of impact spans multiple fields (quantum computing, PINNs, AI for science), and it addresses the grand challenge of automating the scientific process itself. While TraceLift makes a solid contribution to reasoning quality in LLMs through executor-grounded rewards, it represents an incremental improvement within an existing paradigm. ResearchEVO's novelty as a first-of-its-kind system and its potential to transform how scientific research is conducted gives it higher long-term impact potential.
Paper 2 has higher potential impact due to a broadly applicable, timely contribution to LLM reasoning training: executor-grounded rewards that align intermediate traces with downstream utility, addressing a known failure mode of outcome-only RL. The TraceLift framework and TRACELIFT-GROUPS dataset generalize across math, code, and multi-step planner–executor systems, likely influencing evaluation and training paradigms beyond a single domain. Paper 1 is strong and useful for financial TSRMs, but its benchmark/task design and CoT strategies are more domain-specific, limiting cross-field breadth.
Paper 2 likely has higher impact due to broader applicability and timeliness: executor-grounded rewards for reasoning traces address a general failure mode in RL-for-reasoning (correct answers with unfaithful/unused reasoning) across math, code, and agentic planner-executor systems. The TraceLift framework and TRACELIFT-GROUPS dataset introduce a methodologically rigorous, widely reusable training signal that can influence evaluation and training paradigms beyond a single domain. Paper 1 is strong and useful for financial TS reasoning, but its benchmark/task design is more domain-specific and thus narrower in cross-field impact.