Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues
Zain Naboulsi
Rigorous, transparent pre-registered study with a robust prompt-cue×post-training effect, but the central reward-hacking claim is explicitly unresolved and measurement relies on an imperfect single-annotator-validated judge.
Abstract
Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi:10.17605/OSF.IO/S53HY. Code and data: https://github.com/zainnab-sparq/citation-reward-hacking
AI Impact Assessments
(1 model)Core Contribution
The paper asks whether citation fabrication by post-trained LLMs is reward hacking, as opposed to a capability limit. It uses four pre-registered experiments on a frozen 900-question set (3 domains × known-real / obscure-but-real / unanswerable) and 12 open-weights checkpoints (Tülu-3 8B/70B, OLMo-2-13B, each at base/SFT/DPO/RLVR stages), later refit on 15.
The conclusion is "possibly, but not proven." Plain instruction-following remains a live alternative explanation.
Methodological Rigor
- Pre-registration with hashed question sets, pinned checkpoint SHAs, a Holm-corrected hypothesis family, and an exhaustive deviations table.
- Both a "corrected" and a "registered" pipeline are reported throughout.
- Instrument validity gets unusual attention: judge validation against a trivial baseline, a drift analysis, differentiality tests, and a paired re-judge sub-study.
- Negative and awkward findings are reported honestly. The T1 null is reported as a null and the null title is not swapped in. The rejected unfalsifiable draft of amendment #10 is disclosed.
- The outcome is produced by an LLM judge with about 80% precision and 0.78 recall on FABRICATED. The judge's labels were validated by a single annotator (the author), on small samples (n = 50 and n = 150).
- The verifier-REAL trust step failed on this corpus, requiring a post-hoc corrective re-judge.
- The judge drifted mid-collection. The drift is non-differential by most tests, but the condition×grounding test (D3) rejects.
- Only three base-model families (the fourth adds a base model that behaves like a post-trained one), so the family-level random effects are singular and weakly identified.
- The roster was cut from 23 registered checkpoints to 12 without recording the cut, leaving power at 73%. The 15-checkpoint repair was done after seeing the null.
- The judge has no CourtListener access, so legal fabrication is likely under-counted.
- Base models mostly don't cite, so the E2 "base fall" partly reflects non-compliance. The Qwen base contradicts the sign reversal.
- The central causal question (does reward cause it?) is untestable with this design. The authors say so clearly.
Potential Impact
- The within-policy prompt-lever design with a length-matched control is a reusable template.
- The instrument-drift and differentiality methodology for LLM-judge pipelines is a useful contribution to eval practice.
- The frozen question set, with its capability-held-constant obscure-but-real cell, is a reusable benchmark idea, though only 50-question subsets are public.
- It does not move the reward-hacking debate causally.
- The E1 null and the "possibly, not proven" conclusion weaken the headline narrative.
- The study uses 8B–72B open models, not frontier or closed models, and covers no retrieval-augmented setting.
- The single-author, self-funded setting limits visibility.
Timeliness & Relevance
Hallucinated citations in legal, medical, and scientific work are a salient and recurring problem. Linking them to training incentives (Kalai et al., Wen et al.) is topical. The pre-registered, honest-null style is rare in LLM research and is timely given reproducibility concerns.
Strengths & Limitations Summary
- Exceptional transparency.
- A clean E2 effect that survives numerous robustness checks.
- Careful treatment of measurement validity.
- Honest calibration of claims.
- Public code and data (partly masked).
- The reward-hacking thesis is unresolved.
- The measurement foundation is a single annotator plus an imperfect, drifting judge.
- The design is underpowered for E1 and has few model families.
- The paper is very long and dense with amendment bookkeeping, which hurts readability.
- E2's interpretation is confounded with instruction-following.
- The mechanistic E3 result is limited to one quantized 8B model and was not run on the registered 70B target.
Overall: A methodologically conscientious, honest empirical paper. Its robust finding (a prompt-induced fabrication spike specific to post-trained models) is real but somewhat expected, and the paper's central causal question stays open. Its influence is likely to come from methodological practice and the dataset/design template more than from the headline result.
Generated Nov 3, 2026
Comparison History (0)
Rigorous, transparent pre-registered study with a robust prompt-cue×post-training effect, but the central reward-hacking claim is explicitly unresolved and measurement relies on an imperfect single-annotator-validated judge.