Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
Nick Merrill, Jaeho Lee, Ezra Karger
Abstract
We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The pattern appears on ForecastBench-Sim (FBSim), a contamination-free, simulated-world benchmark we release, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation. A per-quantile decomposition shows the failure concentrates at the upper tail, which more capable models shift upward to track aggressive extrapolations of growth, while the lower tail stays put. A within-family study of Llama-3.1 shows that both model scale and post-training independently contribute to this effect. Domain knowledge does not reliably rescue calibration. This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs. Single-threshold scoring at conventional cutoffs misses the upper-tail cost; tail-inclusive scoring reverses the sign of the capability--accuracy relationship on the same outputs. We recommend that LLM forecasting evaluations use continuous (and unbounded) measures of accuracy alongside bounded binary threshold metrics.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper identifies a structurally specific class of inverse scaling in LLMs: distributional forecasting on time series exhibiting superlinear growth followed by regime change. The key finding is that more capable models produce *worse* distributional forecasts on these tasks, and critically, this degradation is invisible under single-threshold binary metrics (like Brier score) commonly used in LLM forecasting benchmarks. The paper introduces ForecastBench-Sim (FBSim), a contamination-free benchmark built on procedurally generated strategy-game rollouts, and demonstrates the phenomenon across synthetic and real-world domains.
The contribution is dual: (1) an empirical phenomenon — "competence-driven overcommitment" where capable models aggressively extrapolate growth trajectories in their upper tails — and (2) a methodological critique showing that scoring rule choice can *reverse the sign* of capability-accuracy relationships on identical outputs.
Methodological Rigor
The experimental design is notably thorough and multi-layered:
Causal identification strategy. The authors address the key challenge that cross-family ECI correlations conflate many variables by deploying three complementary approaches: provider fixed effects, within-lineage replication (OpenAI N=10, p=.008), and a controlled 2×2 within-family experiment on Llama-3.1 ({70B, 405B} × {base, instruct}). The 2×2 design cleanly isolates scale and post-training contributions and shows they compound — a particularly convincing result.
Mechanism isolation. The synthetic SIR vs. linear control experiment is well-designed: both share crash structure, but only the superlinear-growth series produces inverse scaling (non-overlapping CIs). This rules out crash-alone as the trigger and pinpoints the combination of superlinear growth + regime change.
Per-quantile decomposition. The pinball-loss analysis localizes the failure to the upper tail (p90), showing it's not generic miscalibration but asymmetric overcommitment. The calibration and sharpness analysis in Appendix E further confirms distributions widen asymmetrically rather than simply shifting.
Statistical rigor. Bootstrap CIs, exact permutation tests for small N, Wilcoxon signed-rank tests for paired comparisons, and multiple robustness checks (leave-one-provider-out, one-per-lineage collapse, reasoning-mode partition) are appropriate. The authors are transparent about limitations at small N (hyperinflation N=12, housing N=19).
Potential weaknesses in rigor: The cross-family analysis relies on ECI as a capability measure, which aggregates many axes. While the within-family ablation addresses this partially, alternative capability orderings could yield different patterns. The reasoning-effort clamping to "minimal" for time-series experiments is conservative but means the full reasoning-on results are only partially characterized. Domain selection for three of four real-world datasets is post-hoc (selected on known regime change), though the 35-season measles cohort is genuinely ex-ante.
Potential Impact
Immediate practical implications. The paper directly challenges the adequacy of current LLM forecasting evaluation practices. Major benchmarks (ForecastBench, KalshiBench) use exclusively binary/threshold metrics. The finding that metric choice reverses the sign of capability scaling — not merely attenuates it — is a strong call to action for the evaluation community. The recommendation to use CRPS or log scores alongside Brier is concrete and actionable.
Domain-specific consequences. The identified failure mode affects exactly the domains where tail accuracy matters most: epidemic forecasting (resource allocation, intervention timing), financial risk management (VaR, expected shortfall), and monetary policy. The measles finding is particularly timely given resurgent outbreaks. The demonstration that LLMs systematically miscalibrate the upper tail during regime change in epidemic forecasting should concern the growing community applying LLMs to disease surveillance (e.g., Du et al., 2025 in Nature Computational Science).
Broader ML implications. The finding extends the inverse scaling literature in an important direction: unlike prior adversarial or prompt-dependent examples, this inverse scaling is structurally identifiable, persists across families and prompting interventions, and hasn't resolved at frontier scale. The knowledge-calibration gap (models correctly identify hyperinflation crises in 46/48 probes yet fail to translate this into calibrated tails) raises fundamental questions about how LLMs translate retrieved knowledge into probabilistic outputs.
Training implications. The within-family result showing post-training (RLHF/instruction tuning) independently amplifies the failure suggests current training objectives may actively worsen distributional calibration on tail-sensitive tasks.
Timeliness & Relevance
The paper is exceptionally timely on multiple fronts: (1) LLMs are being actively deployed for forecasting in epidemiology and finance; (2) the AI safety community is increasingly concerned about inverse scaling and capability-risk relationships; (3) evaluation methodology for LLM forecasting is actively being developed; (4) measles resurgence makes epidemic forecasting a live public health concern. The gap between what benchmarks certify and what actually matters in deployment is a current bottleneck this paper directly addresses.
Strengths & Limitations
Key strengths:
Notable limitations:
Overall Assessment
This is a well-executed empirical paper with a clear, important finding that has immediate practical implications for LLM forecasting evaluation and deployment. The combination of a new benchmark, synthetic mechanism isolation, real-world replication, controlled within-family ablation, and the metric-reversal demonstration constitutes a thorough and convincing contribution. The main finding — that capability is liability for distributional forecasting under regime change, and that standard metrics hide this — deserves serious attention from both the LLM evaluation community and domain practitioners deploying LLMs for consequential forecasting.
Generated May 22, 2026
Comparison History (27)
Paper 2 has higher potential impact because it addresses a fundamental training bottleneck for the next generation of 'thinking' LLMs. While Paper 1 provides valuable insights into forecasting failures, its impact is constrained to applied domains. Paper 2 challenges the prevailing assumption that privileged-context self-distillation universally aids self-improvement, revealing that it actively harms learning self-correction and reasoning traces. This foundational insight into on-policy distillation will broadly influence how researchers design training pipelines for future reasoning models, giving it a higher citation ceiling and broader impact across foundational AI development.
Paper 2 likely has higher scientific impact because it identifies a counterintuitive, safety- and decision-critical inverse-scaling phenomenon (more capable LLMs making worse distributional forecasts) in domains with tail risk, validated across simulated benchmarks and multiple real-world datasets. It also surfaces an evaluation failure mode (threshold metrics masking upper-tail errors) with immediate implications for forecasting benchmarks and deployment in finance/epidemiology. While Paper 1 is methodologically careful and useful for LLM training practice, its impact is more specialized to post-training CoT compression choices, whereas Paper 2 affects evaluation, reliability, and high-stakes applications broadly.
Paper 1 identifies a fundamental and counterintuitive failure mode of LLMs—inverse scaling on forecasting tasks with superlinear growth and tail risk—with broad implications across AI safety, finance, epidemiology, and LLM evaluation methodology. It introduces a new benchmark, provides mechanistic explanations via per-quantile decomposition, and challenges prevailing assumptions that more capable models are uniformly better. This finding has wide-reaching consequences for how LLM forecasting is evaluated and deployed. Paper 2 presents a useful but more incremental neuro-symbolic framework for clinical diagnosis that achieves performance comparable to (not exceeding) existing methods, limiting its impact.
Paper 2 identifies a critical 'inverse scaling' failure mode in LLMs for high-stakes forecasting tasks, challenging the prevailing 'scale is all you need' paradigm. Its implications for finance, epidemiology, and AI safety are profound, highlighting how increased capabilities can paradoxically increase tail risk. While Paper 1 offers a valuable methodological improvement for self-evolving agents, Paper 2's discovery of a systematic flaw in advanced models across broad, real-world domains gives it a wider scientific and societal impact.
Paper 2 identifies a critical, counter-intuitive inverse scaling phenomenon in high-stakes forecasting domains like epidemiology and finance. Discovering that increased model capabilities degrade performance on tail-risk predictions addresses urgent AI safety and reliability concerns. Its profound implications for real-world decision-making and LLM evaluation methodology give it a broader, more significant potential scientific impact than Paper 1's narrower focus on personality perception.
PopuLoRA introduces a novel population-based co-evolutionary framework for LLM post-training that addresses a fundamental limitation of single-agent self-play (self-calibration to easy problems). It demonstrates consistent improvements across 10 benchmarks in both code and math reasoning, with practical LoRA weight-space evolution operators. Paper 2 makes an important observation about inverse scaling in forecasting with tail risk, but is more diagnostic/evaluative in nature. PopuLoRA's methodological contribution—combining population-based training, asymmetric self-play, and weight-space evolution—opens broader research directions for scalable LLM training and has more transformative potential.
Paper 1 identifies a fundamental and counterintuitive failure mode of LLMs—inverse scaling on forecasting tasks with superlinear growth and tail risk—with broad implications for high-stakes domains like finance and epidemiology. It challenges prevailing assumptions that more capable models are uniformly better, provides mechanistic analysis, and offers actionable evaluation recommendations. The finding is validated across synthetic and real-world datasets. Paper 2 introduces a useful but narrower benchmark for web retrieval agents. While well-executed, it addresses a more specialized problem with less paradigm-shifting potential.
Paper 1 identifies a counterintuitive inverse-scaling failure mode in LLM probabilistic forecasting under superlinear growth and regime-change tail risk, validates it across simulated and multiple real-world domains, and shows how common benchmark metrics can mask the issue—directly impacting evaluation practice and deployment safety in high-stakes settings (finance, epidemiology). Its methodological contributions (tail-focused decomposition, within-family scale vs post-training analysis, metric critique) are broadly applicable beyond forecasting. Paper 2 is valuable as a standardized drug-design agent benchmark, but its impact is more field-specific and primarily infrastructural.
Paper 2 likely has higher scientific impact: it identifies a counterintuitive inverse-scaling failure mode in LLM forecasting under superlinear growth and regime-change tail risk, validates it across simulated and multiple real-world domains, and pinpoints evaluation-metric artifacts that can flip conclusions. This is novel, timely, and broadly relevant to ML evaluation, calibration, and decision-making in high-stakes settings (finance, epidemiology). Paper 1 is valuable but more domain- and benchmark-specific, primarily extending evaluation for spreadsheet-agent workflows rather than uncovering a generalizable modeling pathology.
Paper 1 identifies a critical and counterintuitive vulnerability in modern LLMs (inverse scaling on tail risks), challenging current assumptions about AI capabilities. Its findings have immediate, high-stakes implications for real-world applications in epidemiology and finance. Furthermore, introducing a new benchmark and highlighting flaws in standard evaluation metrics provides broad value to the AI community. While Paper 2 offers a solid methodological contribution to XAI, Paper 1 addresses a more urgent, timely issue with broader cross-disciplinary impact.