On the Primacy Bias in RLVR Training

Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Parshin Shojaee, Chinh Duc La, Duy Minh Ho Nguyen, Rui Zhang, Khoa D. Doan, Ting Hua

Nov 2, 2026arXiv:2611.04061v1

cs.AI

Scorecard· 16/16

5.5/10 impact

Timely, coherent framing of the RLVR sharpening-vs-expansion debate with useful diagnostics, but causal claims rest on a loose toy analogy and small, single-seed effects.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a promising approach for enhancing the reasoning capabilities of large language models. However, it still remains unclear whether RLVR can extend reasoning capabilities beyond what is already encoded in the base model. In this paper, we study this question through the lens of primacy bias: the tendency of models to over-rely on already-learned representations, disproportionately improving performance on problems they already handle well while underexploring harder ones that require departing from those representations. Through our controlled experiments and large-scale analysis, we show that RLVR training on mixed data tends to further sharpen performance on easier problems while yielding only marginal gains on harder ones. We find that although hard problems lead to modest gains overall, they are actually the ones that help most in expanding the model's reasoning capabilities and pushing the representations. However, prolonged training on these hard problems may also hurt and lead to forgetting on easy problems. These findings help to better understand some of the prior conflicting observations about RLVR, specifically, why on (OOD) domains where the base model performs poorly, improvements on hard problems can expand reasoning; while on well-trained domains, forgetting can reduce performance and mask genuine gains obtained from hard problems.

AI Impact Assessments

(1 model)

Impact Assessment: "On the Primacy Bias in RLVR Training"

Core Contribution

The paper tries to reconcile the conflicting literature on whether RLVR expands reasoning capability or merely sharpens the base model's existing distribution. Its explanation combines two ideas from the plasticity and grokking literature: primacy bias and the lazy (NTK-like) training regime. The argument has four steps:

1. In an MNIST toy setting with Zipf-skewed pre-training, RL fine-tuning from a pre-trained model shrinks Pass@k coverage, while random initialization does not. Larger skew gives sparser updates and delayed learning on rare classes, and it lowers CKA alignment between the initial NTK and the hard-class labels.

2. Public RLVR checkpoints (DeepScaleR, DeepCoder, Polaris, and others) are well approximated by first-order linearization around the base model. KL to the linearized policy is about 0.04–0.12 for RL, versus 12–18 for SFT. These checkpoints also show small principal-angle and spectrum drift and small eNTK relative change.

3. On MATH with Qwen2.5-3B-Instruct, difficulty-stratified GRPO yields gains mainly on easy problems. Training on hard problems alone gives slightly better Pass@128 than training on the full or easy sets.

4. Prolonged hard-only training causes forgetting on easy problems, and the linearization begins to break down. The paper argues this forgetting masks gains on base-unsolvable problems, and it uses this to explain why OOD or synthetic domains show "expansion" while math shows none.

The most interesting empirical contribution is the RL-versus-SFT linearization comparison. It gives a quantitative signature for the "RL changes little" intuition, consistent with the "RL finetunes small subnetworks" finding. The framing of Pass@k dilution (computing Pass@k only on base-unsolvable problems) is also practically useful.

Methodological Rigor

The paper has several weaknesses.

Causal chain. The causal claim runs from primacy bias to lazy regime to unlearnability, then to forgetting, and finally to reconciling the literature. It is supported mostly by correlational evidence and analogy. The MNIST toy is a long way from LLM RLVR. Pre-training on a Zipf-skewed class distribution is not obviously analogous to LLM pre-training, and "difficulty = class rarity" is assumed rather than shown.

Linearization test. Small KL between the fine-tuned and linearized policies may reflect the small parameter change typical of low-learning-rate RL (small KL penalties, few steps, on-policy data) rather than a distinctive "lazy regime." The comparison with SFT confounds algorithm with the size of the update and the data distribution. The SFT checkpoints are distillation models from a different base, which is not a clean comparison. No control varies RL learning rate or steps to show that laziness, rather than small movement, drives the effect.

Difficulty stratification. Difficulty is defined via the RL-trained policy's pass@1, after filtering zero-accuracy problems. This induces selection effects, and "hard" subsets suffer from regression-to-the-mean and noise. The claim that hard-only training "yields little improvement" is partly built into the measurement: the headroom and reward variance differ across subsets, and the paper's own gradient-norm appendix hints at this.

Pass@128 results. The gaps are tiny (+1.1 on OlympiadBench, +0.2 on MATH-500), from single seeds with no error bars. The Llama experiment shows hard-only training still below base on two of the three benchmarks. The claim that hard-only training "expands coverage" rests largely on Pass@k restricted to base-unsolvable problems. These problems have few members and a small absolute number of solves (single-digit percentages), so the effect is fragile, and part of that gain could reflect sampling noise or answer-format artifacts.

Scale and scope. Only 3B Instruct models are tested, only math, and only GRPO. Coding was not run. The paper does not test its explanation against the cited conflicting studies (ProRL, grokking, composition) with matched setups, so "reconciles conflicting findings" is a hypothesis rather than a demonstration.

Prior work. The remark that prolonged training lowers Pass@k "in contrast to [28]" cites an Adam-vs-SGD optimization workshop paper, which looks like a citation error and may mean the intended ProRL comparison was never made.

Internal inconsistency. The abstract says hard problems "help most in expanding reasoning," while the paper also finds hard-only training gives little learning. The explanation of why gains arise at all, if the model is lazy, is thin.

Potential Impact

This is mostly an analysis paper with no new algorithm. Its value lies in offering a conceptual vocabulary (primacy bias, lazy regime, forgetting masking hard-problem gains) and some useful diagnostics: KL-to-linearization, principal-angle drift, and Pass@k on base-unsolvable subsets. If it holds up, it could shape curriculum design, data-mixing choices, and how "reasoning boundary" claims are evaluated. The practical lessons are modest: easy data gives sharpening, hard data gives forgetting risk, and evaluation should disaggregate by base solvability. The link to the NTK/lazy-regime theory could inspire more theory-driven work on RLVR dynamics, but the paper stays empirical and does not derive predictions that would let others test it.

Timeliness

The "does RLVR expand the base model?" debate is among the most active in LLM post-training, so the paper is very timely. A unifying story is appealing, which should draw citations. The risk is that the explanation is not decisive, and later work with larger models and controlled pre-training may revise it.

Strengths

It addresses a hot, contested question with a coherent narrative.

The linearization and feature-drift diagnostics on public checkpoints are cheap and reproducible.

The RL-versus-SFT contrast is a clear and memorable result.

The point that Pass@k aggregation dilutes gains is useful.

Code is released, and the compute (4 H100s) is accessible.

Limitations

The paper has no error bars or multiple seeds.

Models are small and only a single domain is used.

The toy-to-LLM analogy is loose.

The causal evidence is thin: no interventions that escape the lazy regime and measure the effect, such as higher learning rate or LoRA versus full fine-tuning.

The reconciliation of prior work is asserted rather than tested.

Several conclusions rest on small differences.

Some text is repetitive, and there are citation oddities (the Adam-vs-SGD paper cited for prolonged RL, and a duplicated reference).

Overall, this is a moderately novel, timely, but evidentially soft analysis paper. It is likely to be cited in the RLVR-debate literature without becoming foundational.

Rating:5.5/ 10

Significance 5.5Rigor 4Novelty 6Clarity 5.5

Generated Nov 4, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

5.5/10 impact

Timely, coherent framing of the RLVR sharpening-vs-expansion debate with useful diagnostics, but causal claims rest on a loose toy analogy and small, single-seed effects.