What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma, Peiji Li, Linyang Li, Qipeng Guo, Dahua Lin
Abstract
Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.
AI Impact Assessments
(1 model)Scientific Impact Assessment: "What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents"
1. Core Contribution
The paper tackles credit assignment in long-horizon LLM agent RL — a genuine bottleneck where a single sparse reward (success/failure) must be distributed across many actions, most of which are causally irrelevant. The core contribution is twofold: (1) a systematic study of five feedback sources × two placement granularities for environment feedback in multi-turn agent RL, and (2) SERL, a framework where task reward determines the update *direction* while environment-conditioned teacher signals adjust only the *magnitude and placement* of that update.
The key architectural principle — that hindsight feedback should modulate rather than replace the RL objective — is well-motivated and clearly articulated. The teacher–student log-probability gap (Eq. 9) is converted into a bounded, sign-aware reweight (Eq. 10) of the GRPO advantage, restricted to action tokens only, with decay over training. This is a clean separation of concerns that avoids the instability of unconstrained distillation while still providing dense credit signals.
2. Methodological Rigor
Strengths in experimental design:
Concerns:
3. Potential Impact
Direct impact: The framework addresses a real pain point in training LLM agents with RL. As LLM agents are deployed in increasingly complex multi-turn environments (web navigation, software engineering, embodied tasks), credit assignment will remain critical. SERL's principle of using environment feedback as a reweighting signal rather than a direct supervision target is transferable.
Practical considerations: The computational overhead is modest (Table 5), making SERL practical for adoption. The code is released, aiding reproducibility.
Broader influence: The systematic taxonomy of feedback sources and placement granularities provides a useful design vocabulary for the community. The finding that "grounded, action-relevant signals outperform richer but weakly causal context" is a meaningful empirical insight with implications for how researchers design reward shaping and auxiliary supervision in agentic settings.
Limitations on impact: The restriction to two benchmarks and a single base model (7B scale) limits confidence in broader applicability. The method is specifically designed for GRPO-compatible training, though the principles could generalize.
4. Timeliness & Relevance
This paper is highly timely. The convergence of (1) LLM agents deployed in interactive environments, (2) RL as a post-training paradigm for LLMs (post-DeepSeek-R1), and (3) the recognized limitations of trajectory-level credit assignment creates a clear need for this type of work. The paper positions itself well against concurrent work (GIGPO, HGPO, RLSD, SDPO), which have primarily been studied in single-turn reasoning or with coarser credit assignment.
The multi-turn agent setting is genuinely underexplored relative to single-turn reasoning RL, making the systematic study valuable even beyond the specific framework proposed.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
6. Additional Observations
The LLM-judged feedback ablation (Table 4) is an interesting forward-looking experiment showing that compressed trajectory summaries can substitute for raw feedback, but the dependence on judge quality and context length introduces another variable. The finding that Kimi-K2.6 (256K context) dramatically outperforms Qwen-7B (32K context) as a judge underscores practical deployment considerations.
The paper is well-written with clear figures and a logical flow from the systematic study to the framework design. The related work coverage is thorough.
Generated May 20, 2026
Comparison History (20)
Paper 2 introduces RosettaSim, a novel framework bridging LLMs and traffic simulation with unique architectural insights about transferability of attention mechanisms. It addresses a critical challenge in autonomous driving (long-horizon multi-agent simulation), proposes a new evaluation metric (RTE) that better correlates with simulation fidelity, and achieves SOTA on a major benchmark (WOSAC). The cross-pollination between LLMs and traffic modeling, plus the new evaluation paradigm, gives it broader methodological impact. Paper 1, while solid, offers more incremental improvements to RL-based agent training with selective distillation.
Paper 2 introduces a broadly applicable, actionable framework (SERL) for long-horizon, multi-turn agent training that leverages diverse per-step environment feedback—an important and timely bottleneck for real-world agents. It reports strong gains on established interactive benchmarks (ALFWorld, WebShop) and provides a systematic study of feedback sources and insertion granularities, suggesting methodological rigor and clearer deployment relevance. Paper 1 offers valuable diagnostics on multi-agent RL stability and policy-sharing tradeoffs, but is more analysis/conditioning-focused with impact concentrated on multi-agent workflow design rather than a general training method.
CausaLab introduces a novel scalable benchmark that addresses a fundamental gap in evaluating LLM agents' causal reasoning—separating predictive accuracy from genuine causal understanding. This tackles a core AI challenge (causal discovery) with broad implications for AI safety, interpretability, and scientific automation. The finding that strong prediction doesn't imply correct causal mechanisms is a significant insight. Paper 2, while solid, is more incremental—improving RL training for LLM agents via selective feedback. CausaLab's contribution as an evaluation framework with a DSL for inspectable causal hypotheses has greater potential to shape future research directions.
Paper 1 likely has higher scientific impact: it introduces a novel, general training framework (SERL) that directly improves long-horizon credit assignment for multi-turn LLM agents by leveraging per-step environment feedback, and demonstrates strong, quantitative gains on widely used benchmarks (ALFWorld, WebShop). This contributes a broadly applicable learning method that can influence RL, agent training, and distillation research. Paper 2 is valuable and timely for tooling/diagnostics and shows practical gains, but its core contribution is more systems-oriented and may be less foundational than a new learning objective/framework.
Paper 2 likely has higher impact due to timeliness and broad applicability to LLM agent training, a rapidly growing area with immediate real-world relevance (web/task automation). It proposes a concrete learning framework (SERL) leveraging environment feedback for long-horizon credit assignment, and demonstrates strong empirical gains on established benchmarks, suggesting methodological rigor and adoptability. Paper 1 offers a principled metric family for uncertainty-augmented systems with solid theoretical framing, but evaluation metrics tend to diffuse more slowly and may have narrower near-term uptake than agent-training methods.
Paper 1 addresses a foundational, field-wide challenge by proposing a new paradigm and taxonomy for evaluating interactive AI systems. While Paper 2 offers a strong methodological improvement for training agents, Paper 1 has broader implications for how the entire community benchmarks, designs, and assesses future interactive LLMs, giving it higher potential for widespread scientific impact.
Paper 1 presents a novel, concrete technical contribution (SERL framework) with rigorous experimental validation showing clear improvements on established benchmarks. It addresses a specific, important problem in RL for LLM agents with a well-defined methodology. Paper 2 is a comprehensive survey/roadmap of AI for research automation—valuable for orientation but primarily synthesizes existing knowledge rather than introducing new methods. Surveys typically have high citation counts but Paper 1's methodological innovation in selective hindsight distillation is more likely to spawn follow-up research and advance the field technically.
WorldString proposes a novel unified framework for actionable object representation in physical world models, addressing a fundamental gap in how objects and their states are modeled. Its differentiable architecture enabling integration with policy learning and neural dynamics positions it as a foundational building block with broad impact across robotics, simulation, and embodied AI. Paper 2, while solid, offers incremental improvements to RL-based LLM agent training in a narrower domain. Paper 1's ambition to unify object state modeling as a primitive for world models has greater potential for cross-field impact.
Paper 2 likely has higher impact because it introduces broadly useful infrastructure: verifiable, auditable software worlds with structured state verifiers, task generation, and partial-credit rewards across 33 real applications and 1,000 tasks. This enables reproducible evaluation and training for computer-use agents, addressing a timely bottleneck (reliable benchmarks and reward signals) with clear real-world relevance. Its methodological contribution (verifier-grounded evaluation vs LLM judges, self-evolving verification) can influence multiple subfields (agent RL, benchmarking, HCI, software engineering). Paper 1 is strong but more incremental within RL/distillation.
Paper 1 addresses the fundamental challenge of long-horizon credit assignment in multi-turn LLM agents, a critical bottleneck for autonomous AI development. Its systematic approach to environment-reweighted learning (SERL) bridges RL and LLM distillation, offering deep methodological insights that are likely to broadly impact agentic AI research. While Paper 2 offers a clever and practical Bayesian optimization approach for prompt tuning, Paper 1's focus on autonomous agent capability advances a more complex and transformative area of AI.