What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma, Peiji Li, Linyang Li, Qipeng Guo, Dahua Lin
Abstract
Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.
AI Impact Assessments
(1 models)Scientific Impact Assessment: "What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents"
1. Core Contribution
The paper tackles credit assignment in long-horizon LLM agent RL — a genuine bottleneck where a single sparse reward (success/failure) must be distributed across many actions, most of which are causally irrelevant. The core contribution is twofold: (1) a systematic study of five feedback sources × two placement granularities for environment feedback in multi-turn agent RL, and (2) SERL, a framework where task reward determines the update *direction* while environment-conditioned teacher signals adjust only the *magnitude and placement* of that update.
The key architectural principle — that hindsight feedback should modulate rather than replace the RL objective — is well-motivated and clearly articulated. The teacher–student log-probability gap (Eq. 9) is converted into a bounded, sign-aware reweight (Eq. 10) of the GRPO advantage, restricted to action tokens only, with decay over training. This is a clean separation of concerns that avoids the instability of unconstrained distillation while still providing dense credit signals.
2. Methodological Rigor
Strengths in experimental design:
Concerns:
3. Potential Impact
Direct impact: The framework addresses a real pain point in training LLM agents with RL. As LLM agents are deployed in increasingly complex multi-turn environments (web navigation, software engineering, embodied tasks), credit assignment will remain critical. SERL's principle of using environment feedback as a reweighting signal rather than a direct supervision target is transferable.
Practical considerations: The computational overhead is modest (Table 5), making SERL practical for adoption. The code is released, aiding reproducibility.
Broader influence: The systematic taxonomy of feedback sources and placement granularities provides a useful design vocabulary for the community. The finding that "grounded, action-relevant signals outperform richer but weakly causal context" is a meaningful empirical insight with implications for how researchers design reward shaping and auxiliary supervision in agentic settings.
Limitations on impact: The restriction to two benchmarks and a single base model (7B scale) limits confidence in broader applicability. The method is specifically designed for GRPO-compatible training, though the principles could generalize.
4. Timeliness & Relevance
This paper is highly timely. The convergence of (1) LLM agents deployed in interactive environments, (2) RL as a post-training paradigm for LLMs (post-DeepSeek-R1), and (3) the recognized limitations of trajectory-level credit assignment creates a clear need for this type of work. The paper positions itself well against concurrent work (GIGPO, HGPO, RLSD, SDPO), which have primarily been studied in single-turn reasoning or with coarser credit assignment.
The multi-turn agent setting is genuinely underexplored relative to single-turn reasoning RL, making the systematic study valuable even beyond the specific framework proposed.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
6. Additional Observations
The LLM-judged feedback ablation (Table 4) is an interesting forward-looking experiment showing that compressed trajectory summaries can substitute for raw feedback, but the dependence on judge quality and context length introduces another variable. The finding that Kimi-K2.6 (256K context) dramatically outperforms Qwen-7B (32K context) as a judge underscores practical deployment considerations.
The paper is well-written with clear figures and a logical flow from the systematic study to the framework design. The related work coverage is thorough.
Generated May 20, 2026
Comparison History (17)
Paper 1 likely has higher scientific impact: it introduces a novel, general training framework (SERL) that directly improves long-horizon credit assignment for multi-turn LLM agents by leveraging per-step environment feedback, and demonstrates strong, quantitative gains on widely used benchmarks (ALFWorld, WebShop). This contributes a broadly applicable learning method that can influence RL, agent training, and distillation research. Paper 2 is valuable and timely for tooling/diagnostics and shows practical gains, but its core contribution is more systems-oriented and may be less foundational than a new learning objective/framework.
Paper 2 likely has higher impact due to timeliness and broad applicability to LLM agent training, a rapidly growing area with immediate real-world relevance (web/task automation). It proposes a concrete learning framework (SERL) leveraging environment feedback for long-horizon credit assignment, and demonstrates strong empirical gains on established benchmarks, suggesting methodological rigor and adoptability. Paper 1 offers a principled metric family for uncertainty-augmented systems with solid theoretical framing, but evaluation metrics tend to diffuse more slowly and may have narrower near-term uptake than agent-training methods.
Paper 1 addresses a foundational, field-wide challenge by proposing a new paradigm and taxonomy for evaluating interactive AI systems. While Paper 2 offers a strong methodological improvement for training agents, Paper 1 has broader implications for how the entire community benchmarks, designs, and assesses future interactive LLMs, giving it higher potential for widespread scientific impact.
Paper 1 presents a novel, concrete technical contribution (SERL framework) with rigorous experimental validation showing clear improvements on established benchmarks. It addresses a specific, important problem in RL for LLM agents with a well-defined methodology. Paper 2 is a comprehensive survey/roadmap of AI for research automation—valuable for orientation but primarily synthesizes existing knowledge rather than introducing new methods. Surveys typically have high citation counts but Paper 1's methodological innovation in selective hindsight distillation is more likely to spawn follow-up research and advance the field technically.
WorldString proposes a novel unified framework for actionable object representation in physical world models, addressing a fundamental gap in how objects and their states are modeled. Its differentiable architecture enabling integration with policy learning and neural dynamics positions it as a foundational building block with broad impact across robotics, simulation, and embodied AI. Paper 2, while solid, offers incremental improvements to RL-based LLM agent training in a narrower domain. Paper 1's ambition to unify object state modeling as a primitive for world models has greater potential for cross-field impact.
Paper 2 likely has higher impact because it introduces broadly useful infrastructure: verifiable, auditable software worlds with structured state verifiers, task generation, and partial-credit rewards across 33 real applications and 1,000 tasks. This enables reproducible evaluation and training for computer-use agents, addressing a timely bottleneck (reliable benchmarks and reward signals) with clear real-world relevance. Its methodological contribution (verifier-grounded evaluation vs LLM judges, self-evolving verification) can influence multiple subfields (agent RL, benchmarking, HCI, software engineering). Paper 1 is strong but more incremental within RL/distillation.
Paper 1 addresses the fundamental challenge of long-horizon credit assignment in multi-turn LLM agents, a critical bottleneck for autonomous AI development. Its systematic approach to environment-reweighted learning (SERL) bridges RL and LLM distillation, offering deep methodological insights that are likely to broadly impact agentic AI research. While Paper 2 offers a clever and practical Bayesian optimization approach for prompt tuning, Paper 1's focus on autonomous agent capability advances a more complex and transformative area of AI.
Paper 1 likely has higher impact due to a concrete algorithmic contribution (SERL) that improves long-horizon credit assignment for multi-turn LLM agents using environment feedback, with strong empirical gains on widely used agent benchmarks (ALFWorld, WebShop). This is timely for tool-using agents and has clear real-world applicability and potential to generalize across interactive settings. Paper 2 offers valuable interpretability insights into SFT dynamics and practical guidance (early stopping), but is more explanatory/diagnostic and may translate less directly into broadly adopted training methods.
Paper 1 addresses the critical and timely challenge of credit assignment in multi-turn LLM agents using RL, a topic at the intersection of two rapidly growing fields (LLM agents and RLHF). Its systematic study of feedback sources and the SERL framework offer practical, broadly applicable insights for training LLM-based agents. Paper 2 tackles a more niche problem (multi-task unlearning) with solid contributions, but the problem scope and community interest are narrower. Paper 1's relevance to the booming LLM agent ecosystem gives it higher potential for citations and real-world adoption.
Paper 2 addresses a fundamental bottleneck in LLM agent training (long-horizon credit assignment) and proposes a novel framework (SERL) that achieves state-of-the-art results on widely used benchmarks like ALFWorld and WebShop. While Paper 1 provides a valuable negative result and theoretical hypothesis regarding environment feedback, Paper 2's broad applicability, proactive methodology, and strong empirical performance across general agent tasks give it higher potential for widespread adoption and citation impact.
Paper 1 introduces a novel cross-modal framework (GVG) that bridges EEG signals with visual representations through generative models, opening a new paradigm for brain-computer interfaces and neural signal understanding. Its approach of 'visualizing the invisible' by generating proxy images from non-visual EEG is highly innovative, combining neuroscience with multimodal LLMs. It demonstrates strong parameter efficiency and broad applicability to clinical settings. Paper 2, while solid, addresses a more incremental improvement in RL-based agent training through selective distillation. Paper 1's interdisciplinary nature and novel conceptual contribution give it higher potential impact.
Paper 1 addresses a fundamental challenge in AI—long-horizon credit assignment in reinforcement learning for LLM agents—which has broad applicability across autonomous agent research. Paper 2 focuses on a highly specific, albeit practically useful, enterprise application (NL2SQL). The methodological innovations in Paper 1 offer greater novelty and potential for cross-disciplinary impact in general agent training compared to the domain-specific multi-agent pipeline proposed in Paper 2.
Paper 1 addresses a critical bottleneck in the highly active field of LLM agents: long-horizon credit assignment and effective use of sparse rewards. By systematically leveraging per-step environmental feedback for selective distillation, it offers a highly practical and timely solution for web and software navigation agents. While Paper 2 presents a strong hierarchical world model for RL, the immediate real-world applicability, booming interest in multi-turn LLM agents, and strong empirical results on popular benchmarks (ALFWorld, WebShop) give Paper 1 a broader and more immediate potential impact across AI communities.
While Paper 1 presents a highly practical reinforcement learning framework for multi-turn agents, Paper 2 offers a foundational theoretical contribution. By clarifying the widespread misconception regarding the Turing-completeness of Transformers and formalizing the critical role of context management, Paper 2 reshapes our theoretical understanding of LLMs. Foundational theoretical corrections generally have a broader and longer-lasting scientific impact across the entire field of AI than specific algorithmic improvements on empirical benchmarks.
Paper 2 likely has higher impact: it offers a mechanistic, cross-model causal explanation for a timely and broadly relevant failure mode (multimodal hallucination under modality conflict), plus an inference-time mitigation (MACI) that transfers zero-shot. The head-level causal analysis across five open-source MLLMs strengthens rigor and generality, and the intervention is practically deployable without retraining, increasing real-world applicability. Paper 1 is solid and useful for RL agents, but is more domain-specific (multi-turn agent credit assignment) and may have narrower cross-field reach than mechanistic interpretability + safety in MLLMs.
Paper 2 addresses a highly timely and critical bottleneck in current AI research—long-horizon credit assignment for LLM agents. Given the rapid growth and investment in autonomous agents, its selective distillation framework offers immediate, broad applicability. While Paper 1 provides a novel theoretical bridge between ML and behavioral economics, Paper 2's practical improvements in training language agents are likely to yield a higher and more immediate scientific impact across the machine learning community.
Paper 2 has higher likely scientific impact due to broader applicability and timeliness: improving credit assignment for multi-turn LLM agents is central to current agentic AI, with direct relevance to web/navigation, tool use, and interactive environments. SERL’s framework of selectively leveraging diverse environment feedback sources is a generally useful training paradigm that can transfer across tasks and agent settings. While Paper 1 shows striking benchmark gains and efficiency, it is more specialized to TRM-style recursive solvers and test-time stochastic search, with narrower cross-field impact despite high novelty.