EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
Weiyuan Li, Aili Chen, Xintao Wang, Yikai Zhang, Qingqing Dong, Jinghan Xu, Hongru Hou, Wenxuan Zhao
Timely, well-engineered framework addressing a real bottleneck (train-time reward reliability), but marginal writing gains, narrow two-domain scope, and dependence on proprietary agents/judges limit its demonstrated impact.
Abstract
Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by and points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.
AI Impact Assessments
(1 model)Scientific Impact Assessment: EvoRS
Core Contribution
EvoRS reframes reward reliability in open-ended RL as a *train-time* property rather than a fixed design choice, and proposes evolving the entire reward system—criteria, scoring mechanisms, and signal composition—during training. Its central abstraction is an executable Reward-DAG whose nodes (continuous scorers, checklists, flaw-penalties, composition operators) can be revised through typed edits by an agentic "designer" that diagnoses on-policy rollouts and reward traces. Candidate DAG states are validated via matched replay (same rollout cases) before a transition is committed. The key conceptual advance over prior dynamic-rubric work (RLER, OpenRS) is the observation—supported by their motivating analysis—that adapting rubric *criteria* alone leaves scoring and composition failures unaddressed, so validity, coverage, and informativeness failures can persist.
Methodological Rigor
The experimental design is reasonably thorough for the subfield: a unified GRPO pipeline shared across baselines (RLAIF, RaR, RLER, OpenRS), three independent judges with method-blind evaluation, paired bootstrap uncertainty estimates, generality checks across policy scale (4B→8B) and GRM family (Qwen3.5-27B vs RewardAnything-8B), and audits (HR/CFR taxonomies frozen before comparison, plus a blind list-wise ranking audit with human validation, Fleiss' κ=0.705). The ablations are well-targeted: separating "criteria-only adaptation" and "fixed-final-DAG" from full evolution directly tests the paper's core claim, and the results support it (large CoSER drops for both restricted variants).
However, several gaps temper confidence. The WritingBench improvement is marginal (+0.454 over RaR; bootstrap interval [−0.022, +0.918] crosses zero), so the headline "2.107 points over policy" masks a near-tie with the strongest baseline on one of two domains. The substantive gains are concentrated in CoSER roleplay. Only two domains and one policy family (Qwen3) are tested, and the designer relies on a strong proprietary model (GPT-5.4), leaving open whether gains stem from reward-system evolution per se or from a capable agent's general competence. The self-validation thresholds are hand-set implementation defaults.
Potential Impact
Reward hacking and reward-signal degradation are acute, widely-recognized bottlenecks in RLHF/RLAIF for non-verifiable tasks. The framing "reward reliability must be measured on on-policy responses, not fixed distributions" is a clean, transferable insight. The Reward-DAG as a modular, editable reward representation is a plausible reusable building block, and the taxonomy of reward mechanisms (Appendix B) is a useful reference contribution in its own right. If the approach generalizes to agentic/tool-use settings (as the authors speculate but do not demonstrate), impact could broaden considerably.
Timeliness & Relevance
Highly timely. The paper sits directly on the current frontier of post-training LLMs for open-ended tasks, engaging with a dense cluster of 2025–2026 dynamic-rubric and reward-overoptimization work. The problem it targets is a live pain point for anyone doing RL on writing, roleplay, or workflow agents.
Strengths & Limitations
Strengths: (1) A genuinely novel angle—evolving scoring mechanisms and composition, not just criteria; (2) careful audit methodology with blinding and independent human/LLM validation that guards against judge gaming; (3) the behavioral analysis (Table 18) showing baselines inflate length/repetition while EvoRS stays near base length is convincing corroboration that reward hacking is real and mitigated; (4) exceptionally detailed appendices (schema, prompts, tool interfaces, legality checks).
Limitations: (1) marginal writing gains; (2) narrow domain/model scope; (3) heavy dependence on a proprietary agent and proprietary judges; (4) added compute and system complexity (designer adds ~243s per evolution step, ~50% step-time overhead); (5) reproducibility hampered by unreleased datasets and proprietary models with undisclosed parameters. Notably, the paper uses forward-dated citations and next-generation model names (GPT-5.6, DeepSeek-V4, Qwen3.5-27B), which cannot be independently verified and slightly undermine external credibility.
Other Observations
The contribution is more a *framework/system* than a fundamental theoretical result. Its value will depend on whether the community adopts the Reward-DAG abstraction or whether simpler evolving-rubric methods prove "good enough." The insight that fixed comprehensive reward systems underperform evolving ones is somewhat expected given the reward-overoptimization literature, so the surprise is in magnitude/demonstration rather than direction.
Generated Sep 14, 2026
Comparison History (0)
Timely, well-engineered framework addressing a real bottleneck (train-time reward reliability), but marginal writing gains, narrow two-domain scope, and dependence on proprietary agents/judges limit its demonstrated impact.