One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets
Jiaying Zhou, Jiachen Liu, Qingchao Chen
Neat signed-offset framing with a sensible anchor control, but gains over the anchor are small, unreplicated across seeds, and require a costly strategy-sampling construction that underperforms vanilla GRPO on pass@1.
Abstract
Group Relative Policy Optimization (GRPO) estimates advantages against the within-prompt mean reward, a baseline shared across the group. In practice, rollouts within a prompt are not always homogeneous, differing in their solution strategies and reasoning trajectories, along which they may group into subgroups. We further find that these heterogeneous subgroups may exhibit distinct reward distributions. The global baseline, however, collapses them onto a single credit axis, where the credit reflects neither a rollout's standing within its subgroup nor the subgroup's standing relative to the rest. We therefore study how to adjust the baseline when the rollouts of a prompt fall into heterogeneous subgroups. We define the \emph{subgroup offset} as the deviation of a subgroup mean from the global mean, and shift the baseline along it by a single signed coefficient . The sign of yields two opposing directions: \emph{Alignment} () adds the offset to the baseline, drawing credit toward a rollout's relative position within its subgroup; \emph{Contrast} () subtracts it, drawing a rollout's credit toward the relative quality of the subgroup it belongs to. The adjustment acts on the baseline alone, leaving the reward signal and the optimization objective unchanged. In this work, we induce subgroups from strategies, grouping rollouts that share a strategy into the same subgroup. We evaluate both directions on Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, against a anchor that shares the same strategy-conditioned sampling and differs only in the baseline. Against this anchor each direction yields modest but consistent gains in multi-sample coverage, with Contrast the stronger of the two.
AI Impact Assessments
(1 model)Core Contribution
The paper proposes a one-parameter family of baseline adjustments for GRPO when a prompt's rollouts split into heterogeneous subgroups. The baseline becomes b̃_i = μ_g + λ·δ_i, where δ_i is the subgroup mean minus the global mean. For λ>0 ("Alignment") the advantage is a convex combination of the global and within-subgroup advantages, so λ=1 recovers Stratified-GRPO-style within-subgroup centering. For λ<0 ("Contrast") the advantage adds a term proportional to the subgroup offset, which amplifies between-subgroup differences. Subgroups come from PTA-GRPO-style presampled "strategy previews". The algebra is a one-line rewrite (Eq. 7/9), and the family is simple. The conceptual contribution is mainly the framing: it is a signed continuum, and the negative half, which favors rollouts from stronger strategy subgroups, is largely unexplored. Beyond that, the paper is an incremental recombination of existing pieces (stratified baselines, plan-conditioned rollouts, tunable baselines).
Methodological Rigor
The design has real strengths. The anchor is the right control: Strategy-GRPO with λ=0 shares the same sampling and differs only in the baseline. The offset-shuffling ablation is a thoughtful causal check showing that gains depend on the correct subgroup-offset pairing. The paper is also honest about caveats: it admits shuffling doesn't separate "informative offset" from "any miscalibrated offset hurts", and it scopes its λ conclusions to Qwen.
The weaknesses are substantial:
Potential Impact
The idea is a clean, cheap plug-in that could be reused wherever rollouts have natural partitions (tools, branches, tree search). The paper notes this generality but verifies only strategy-induced subgroups on competition math. The Contrast result, that amplifying between-strategy differences helps pass@16, is interesting and could inform exploration/diversity work. However, the effect sizes are modest, the requirement for strategy presampling is a practical barrier, and the evidence that this beats simpler alternatives is absent. Adoption is likely limited to a niche of credit-assignment and plan-guided RLVR papers.
Timeliness
GRPO variants and advantage estimation are very active areas, so the topic is timely. The area is also crowded, with many incremental baseline tweaks. The unified-family framing is neat but offers limited new insight.
Strengths
Limitations
Overall, this is a modest, competent incremental contribution with a pleasing formulation but thin empirical support and limited likely influence.
Generated Nov 3, 2026
Comparison History (0)
Neat signed-offset framing with a sensible anchor control, but gains over the anchor are small, unreplicated across seeds, and require a costly strategy-sampling construction that underperforms vanilla GRPO on pass@1.