Environment Evolution for Terminal Agents
Zhiyuan Fan, Tinghao Yu, Yuanjun Cai, Jiang Zhou, Jiangtao Guan, Jincheng Liu, Yun Yang, Dingxin Hu
Fresh off-policy framing for a central agentic-RL bottleneck with strong controlled gains, tempered by single-benchmark evaluation and weak reproducibility.
Abstract
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.
AI Impact Assessments
(1 models)Scientific Impact Assessment
Core Contribution. This paper (Tencent Hunyuan Team) introduces *environment evolution*, an off-policy paradigm for generating increasingly difficult RL training environments for terminal (command-line) agents. The central insight is that existing agent–environment *co-evolution* methods depend on on-policy rollouts to expose model weaknesses, which (a) ties the generated environments to a specific policy and initial distribution and (b) saturates as models improve and failures become sparse. The authors instead derive a *model-agnostic* difficulty measure from the multi-turn learning objective, decomposing trajectory difficulty into three levers — scenario novelty, skill rarity, and execution length — by replacing policy-dependent probabilities with a reference "world-knowledge" distribution. They implement evolution via a two-loop multi-agent harness (plan refinement + environment refinement with oracle/invalid-test/rubric verifiers) and an Evolution-Lineage scheduler that admits harder generations only as the policy masters easier ones. Empirically, they report +14.4 and +18.0 pp gains on Terminal-Bench 2.1 for Qwen3.6-27B and Qwen3.6-35B-A3B, outperforming co-evolution and ensemble baselines.
Methodological Rigor. The experimental design is reasonably strong for a systems/empirical paper. Key strengths: matched-budget comparison against two competing paradigms (co-evolution, ensemble) with a fixed synthesis model (Claude Opus 5); ablations isolating both evolution effort (low/high/max) and evolution direction (scenario/skill/length), each cross-validated over 1-step vs 15-step lineages; and difficulty verification across three independent frontier models. The scheduler ablation (Fig. 5) convincingly shows more partially-solved rollout groups (informative GRPO advantage). Weaknesses: evaluation rests on a *single* held-out benchmark (Terminal-Bench 2.1); results are averaged over five runs but no variance/significance analysis is reported; the theoretical derivation, while elegant, is heuristic — the "reference distribution grounded in broad world knowledge" estimated by "a deep-research agent through web search" is under-specified and hard to validate. The pipeline also relies on human-in-the-loop review during development, which partly undercuts the "automatic/off-policy" framing.
Potential Impact. Environment scaling for agentic RL is one of the most actively pursued bottlenecks of 2025–2026, and this work targets it directly. The off-policy formulation is practically attractive: it decouples environment construction from the (expensive, ever-shifting) training policy, enabling reusable difficulty lineages. If the gains hold, industrial teams building terminal/coding/computer-use agents would have strong reason to adopt lineage-based curricula. The three-factor difficulty decomposition also provides a conceptual vocabulary others could reuse. The authors flag SWE and Computer-Use agents as natural extensions, suggesting cross-domain applicability within agentic RL.
Timeliness & Relevance. Highly timely. It sits squarely in the current shift from algorithm/infrastructure scaling toward *environment* scaling, and it engages directly with the very recent co-evolution literature (POET/UED lineage adapted to LLM agents). The problem it addresses — training-signal saturation as frontier models get stronger — is an emerging and real concern.
Strengths.
Limitations & Gaps.
Other observations. The dataset engineering (47,678 environments filtered to 127, then supplemented via SkillSynth to 500 seeds) is a modest but useful contribution to the terminal-agent training-data discussion, and the observation that co-evolution "primarily targets skill-selection errors under seed scenarios" while missing the L and scenario-novelty axes is an insightful critique of prior art that qualifies (rather than refutes) that line of work.
Overall, this is a solid, timely contribution to a hot subfield with a fresh conceptual framing and convincing head-to-head gains, tempered by single-benchmark evaluation, weak reproducibility, and a somewhat hand-wavy theoretical core.
Generated Sep 4, 2026
Comparison History (0)
Fresh off-policy framing for a central agentic-RL bottleneck with strong controlled gains, tempered by single-benchmark evaluation and weak reproducibility.