Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Aishik Nagar, Cheston Tan

Oct 29, 2026arXiv:2610.39182v1

cs.AIcs.CL

Scorecard· 16/16

5.5/10 impact

Timely, well-controlled toy-domain study with a clear thesis and many appendix controls, but the evidence leans on fragile t-SNE statistics, a single seed, tiny models and a single solved game.

Abstract

Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.

AI Impact Assessments

(1 model)

Core Contribution

The paper asks whether RL with verifiable rewards (GRPO) produces "reasoning" or merely better exploitation of prompt-level cues. It trains small instruction-tuned LLMs (Qwen2.5-0.5B/1.5B, Llama3.2-1B) on Tic-Tac-Toe, a fully enumerable domain (5,478 reachable states) with minimax ground truth. It then traces internal representations across GRPO checkpoints with SAEs trained on generic C4 text. Seven hypotheses are tested: turn, game progression, strategic situation, symmetry, legality, best-move correctness, and line purity.

The headline claim is that GRPO improves accuracy sharply (for example Qwen-1.5B legal-move accuracy rises from ~12% to 100%, and best-move accuracy from ~8% to ~75%). The mechanistic story is that this comes from reorganising the dominant variance axes toward prompt-visible properties (game progression F grows 8.5×, turn F falls 17×). Higher-order abstractions (forks, strategic situation, symmetry) stay weak and static. Raw-activation probes show prompt-level properties are already perfectly decodable in the base model. Activation patching localises the decision in late layers (66/100 flips at layer 25, against 12/100 for a random donor).

The framing, "RL neither adds nor removes concept information but reorganises which features dominate," is a clean, quotable thesis. Tracking representation across the RL trajectory in a solvable domain is a real, if modest, novelty. Tic-Tac-Toe LLM-RL studies and OthelloGPT-style probing both exist, but their combination does not.

Methodological Rigor

*Strengths:*

The exhaustive state space with an exact oracle is a good design, and the canonical-symmetry split is a thoughtful control.

The appendices add many checks: a structured-generation format control (0.4–8% vs 27–76%), chance-corrected skill by branching bin, a behavioural equivariance test (56.7% on trained groups vs 57.7% on held-out), a memorisation check, an arena evaluation, full-space probes with permutation nulls, and patching with random-donor and layer controls.

The authors openly acknowledge circularity in the SAE/t-SNE/ANOVA statistics and add SAE-free probes.

*Weaknesses:*

Much of the headline evidence rests on t-SNE ANOVA F-statistics along the "principal axis" of a 2D projection. This is a fragile, visualisation-dependent measure, and the p-values are trivially significant, as the authors concede. The 17× and 8.5× numbers come from one model, one layer and one seed. The authors admit the reorganisation is noisier in the smaller models, and they do not vary the training seed.

The SAEs are trained on generic C4, not on Tic-Tac-Toe activations. They are then applied to heavily out-of-distribution board prompts, so the dictionary may be poorly suited to the domain. Reconstruction fidelity and downstream loss are not reported.

The central negative claim, that strategic abstractions are not represented, is weakly supported. Probe AUC for H3 (strategic situation) is ~0.68, but the label is a 5-class categorical, class imbalance and baseline decodability are unclear, and an MLP probe reaches only 0.75. This shows weak decodability at the last prompt token, not absence of search or lookahead computed during chain-of-thought generation. The model generates a `` trace, and the computation may happen in the generated tokens. Reading only the final prompt position largely misses this, and the authors acknowledge it.

The title question ("reasoning or pattern exploitation?") is only partly answered. The 66% patching result indicates the late-layer state is causal. It does not distinguish memorised lookup over 5.5k states from computation, which is a fundamental ambiguity in a domain this small (80% of states are in training). The held-out-equals-train accuracy is evidence of some generalisation across states, and it sits uneasily with a pure "prompt cue" story. The model reaching ~75% on best-move may well reflect learned heuristics that the hypothesis set does not capture. The authors note H1–H7 are a closed set.

The inference "symmetry curves track random-split curves, so no symmetry abstraction" is plausible but indirect. The SFT baseline failed (models copied the prompt), so no supervised comparison supports the claim that RL is the cause.

The claim that "benchmarks overestimate reasoning" is a generalisation from a toy, solved domain with ≤1.5B models.

Potential Impact

The paper is a useful case study and methodological template for studying what RLVR changes inside models. It fits the active debate (does RL expand reasoning or sharpen existing capability?). The mechanistic finding that information is already present and RL reshuffles feature prominence is consistent with "RL sharpens rather than teaches." That makes the result plausible, but it is not strongly surprising. Practical adoption will be limited. The pipeline is bespoke, the models are tiny, and a Tic-Tac-Toe SAE study will not change how frontier RL is done. It may be cited in the interpretability and RL-reasoning critique literature. The main benefit is as a controlled testbed and a cautionary framing.

Timeliness

Very timely: GRPO and RLVR, the "reasoning vs. shortcuts" debate, and SAE-based analysis of training dynamics are all current. The gap in tracing representations across RL training is real.

Strengths & Limitations

*Strengths:*

Clear thesis, exhaustive ground truth, and extensive controls in the appendix, including probes that do not depend on SAEs.

A causal intervention (patching and steering) rather than purely correlational analysis.

Honest limitations sections.

*Limitations:*

Single domain, tiny models, GRPO only, and no seed variation.

Generic-corpus SAEs with a fragile t-SNE-based headline statistic.

Representation read at the last prompt token only.

A closed hypothesis set, so negative findings are not conclusive.

An overreaching generalisation to "reasoning benchmarks overestimate abilities."

Several claims ("for the first time") are strong.

The paper is dense, with key evidence deferred to appendices.

Overall: A solid, carefully hedged small-scale mechanistic study with a memorable thesis, but with limited scope and some methodological fragility. Likely moderate impact.

Rating:5.5/ 10

Significance 5Rigor 5.5Novelty 5.5Clarity 5.5

Generated Oct 30, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

5.5/10 impact

Timely, well-controlled toy-domain study with a clear thesis and many appendix controls, but the evidence leans on fragile t-SNE statistics, a single seed, tiny models and a single solved game.