Not Every User Says It All: Training Tool-Using Agents Beyond Ideal Simulated Users
Han Bao, Zheyuan Zhang, Kaiwen Shi, Yanfang Ye
Well-motivated problem with a clean masking protocol and a plausible critic-free credit-assignment idea, but small-scale, noisy, partly single-run evidence and modest gains limit impact.
Abstract
Tool-using agents are trained and evaluated almost exclusively with strong simulated users who state every detail clearly, yet real users are often weaker. We construct weak users by masking sentences of a simulated user's replies, holding the task and the user's knowledge fixed, and cross-evaluate agents trained with strong and weak users on tau2-bench and ToolWOZ. Weak users lower the success of every policy, and training with them does not help: no policy trained with weak users, through fixed masks, mixtures or a curriculum, serves weak users better than the strong-trained policy beyond run-to-run noise, and prompting the agent to ask more does not help either. We propose StaGPO (State-Grouped Policy Optimization): the agent writes what the task needs, what is missing and how it is being filled, and GRPO uses this state to compare turns of different rollouts that reach the same situation, assigning step-level credit. Within the standard recipe of a small SFT cold start followed by RL, StaGPO makes weak-user training useful: on ToolWOZ, the Qwen3.5-2B policy trained with weak users reaches 76.8% with weak users, the highest of any Qwen3.5-2B policy we trained, 4.8 points above GRPO and 2.8 above the strongest baseline; it is also 5.4 points above GRPO with strong users, and generalises to an unseen mask rate; trained with strong users instead, it serves weak users at least as well as GRPO trained with weak users. On tau2-bench it scores highest with weak users for both models, by a small margin for Qwen3.5-4B.
AI Impact Assessments
(1 model)Core Contribution
The paper makes two contributions. (1) A controlled "weak user" protocol: sentences of a simulated user's reply are masked with `[inaudible]` at rate p, so the task, the user's knowledge and the reward stay fixed and only the channel is degraded. This cleanly separates information loss from changes in user behavior or simulator size. (2) StaGPO, a GRPO variant in which the agent writes a hidden need–gap–fill JSON state before each action. A fixed per-domain abstraction of that state (domain, action, known slots, last outcome) serves as a grouping key. Turns from different rollouts that share a key are compared by discounted return, which gives a critic-free step-level advantage on top of the episode-level GRPO advantage.
The empirical findings are partly negative. Weak users hurt every policy, and fixed-mask, mixed and curriculum weak-user training do not beat strong-user training beyond noise. Prompting the agent to ask more doesn't help, and repair or state-accuracy rewards fail. StaGPO then makes weak-user training useful: on ToolWOZ with Qwen3.5-2B it reaches 76.8% under weak users, 4.8 points above GRPO and 2.8 above InfoPO, and it transfers across mask rates.
Methodological Rigor
Strengths:
Weaknesses:
Potential Impact
The problem is genuine and under-studied: training agents against idealized simulated users. The negative result that "train under the mismatch" doesn't help is useful for practitioners. The grouping-by-agent-written-state idea is a plausible reusable primitive for critic-free step-level credit in free-text multi-turn settings, where identical environment states (as in GiGPO) rarely occur. The scale is small (2B/4B, LoRA, two benchmarks), so adoption depends on whether the idea holds at larger scales and with real users. The effect sizes on ToolWOZ (about 5–7 points) are moderate and the evidence is partly single-run.
Timeliness
Agentic RL with simulated users (UserRL, τ²-bench, SWEET-RL) is a hot area. User realism and credit assignment in multi-turn RL are well-recognized bottlenecks, so the paper is timely.
Strengths and Limitations
Strengths: a careful controlled protocol; honest, thorough diagnostics; a cheap SFT cold start (33 dialogues); the finding that STaGPO also helps with strong users; and the ablation that isolates grouping as the active ingredient.
Limitations: small scale; noisy evaluation; heterogeneous baselines; dependence on the cold start (without it STaGPO is below GRPO on ToolWOZ); a per-domain hand-designed key; synthetic weak users; and a mixed, partly inconsistent narrative on τ²-bench. The paper's framing, "weak-user training does not help unless you use StaGPO," is partly confounded by StaGPO also improving strong-user performance, so the benefit may be generic credit assignment rather than weak-user-specific.
Overall this is a solid, carefully reported, moderately impactful paper. The protocol and diagnostics are more likely to be reused than the method itself.
Generated Oct 20, 2026
Comparison History (0)
Well-motivated problem with a clean masking protocol and a plausible critic-free credit-assignment idea, but small-scale, noisy, partly single-run evidence and modest gains limit impact.