Kurate.org
Rankings
Back to Rankings

"Close enough?" An investigation into multiplication errors in LLMs

William Bolton, Bezawit Tilaye, Huw Day

Oct 28, 2026arXiv:2610.38634v1
cs.AI
Share
Scorecard· 16/16
4.0/10 impact

Interesting parity-preservation regularity in Qwen models and a negative result for noisy quantisation, but purely behavioural, narrow in models, and with mechanistic claims left untested.

Abstract

Large language models (LLMs) frequently fail at multi-digit arithmetic, but the underlying mechanisms governing these failures remain poorly understood. While recent work demonstrates that addition errors follow a "noisy quantisation" model localised around carry boundaries, whether this framework generalises to multiplication remains an open question. In this work, we evaluate multiplication capabilities across several small base LLMs on a dataset of 16,991 problems spanning varied operand lengths. We show that multiplication errors do not conform to the addition-style noisy quantisation model. Instead, we uncover a novel error regime in small Qwen models: errors in lower-order columns predominantly preserve parity. Conversely, higher-order digit errors shift toward off-by-one carry failures. Our findings suggest that arithmetic computation in LLMs relies on distinct, multi-periodic internal representations where parity is tracked more robustly than exact digit identity, providing new mechanistic insights into numerical reasoning in transformers.

AI Impact Assessments

(1 model)

Core Contribution

This workshop paper (NeurIPS NewInML) tests whether the "noisy quantisation" model of addition errors (Wen et al.), in which errors cluster near carry-potential integer boundaries and are mostly ±1, carries over to multiplication in small LLMs. The authors evaluate five small base models (Qwen3.5-2B, Qwen3.5-0.8B, Qwen2.5-Math-1.5B, Llama-3.2-3B, Gemma-4-E2B) on 16,991 multiplication problems across seven operand-size tiers. They report two findings:

1. The noisy-quantisation "bathtub" shape does not appear for multiplication. The fit has R² = −1.74 for Qwen3.5-2B on 2×2 problems.

2. In Qwen3.5 models, wrong digits in lower-order columns (especially tens) preserve the parity of the correct digit far above chance (92.7% vs ~44% null for Qwen3.5-2B). In the highest-order columns, errors are instead mostly ±1.

The authors interpret the parity result as evidence for multi-periodic internal representations in which mod-2 information is tracked more robustly than mod-10 information.

The parity-preservation observation is the most interesting contribution. It is a concrete, easily stated empirical regularity that connects to the Fourier-feature and helix literature. The negative result on noisy quantisation is useful but modest. The authors themselves offer only a hand-wavy explanation (entangled partial products).

Methodological Rigor

Strengths:

  • The dataset is reasonably sized, exhaustive for small tiers, and randomly sampled for larger ones.
  • Greedy decoding and a minimal prompt keep the setup simple.
  • The authors construct a model-specific null baseline for parity matching, which accounts for digit-distribution bias. They also note the units-column confound: the even-digit skew of products inflates parity matching there. Reporting that the units effect is only modest (89.8% vs 79.1%) is honest.
  • They acknowledge that the independence assumption in the binomial test is violated.

    • Weaknesses:

  • The analysis is purely behavioural. The mechanistic claim ("distinct, multi-periodic internal representations") is not tested; probing and patching are deferred to future work. The abstract's claim of "new mechanistic insights" therefore overreaches.
  • There are alternative explanations for parity preservation that are not ruled out. Parity of a product's tens digit is influenced by structured arithmetic. For example, the tens digit of a product depends on carries from the units column plus partial products, so errors in carry (a ±1 error in the carry) would flip parity, while errors from other sources might not. A model that computes mod-2 correctly may reflect a shortcut such as operand-parity heuristics. The paper does not examine whether the effect is just a consequence of specific carry or partial-product structure, or whether an "even distance" error corresponds to systematic ±2 or ±4 errors from, say, miscomputed partial products. The distribution of error magnitudes (±2, ±4, etc.) is not reported.
  • The conditioning on incorrect digits within a column creates selection effects. Errors in the tens column conditioned on prior digits being correct or incorrect are entangled through autoregressive generation.
  • The carry-potential analysis for multiplication adapts a definition built for addition, and the R² on a single model and tier is thin evidence for "does not transfer." Only the 2×2 tier and one model are shown. The fitted model may be misspecified for multiplication, because the proper carry potential differs, so the negative result may reflect a poor adaptation rather than a true failure of the framework.
  • No seeds, confidence intervals, prompt-format variation, or few-shot controls are reported. The accuracy of base models with a 10-token cap and a minimal prompt may be sensitive to formatting.
  • Only one Qwen family shows the effect strongly. Qwen3.5-0.8B shows it weakly (69.5%) and Qwen2.5-Math shows it mildly (67.9%). Llama and Gemma show none, possibly because their accuracy is low. The generality of the claim is therefore very limited, and the paper partly concedes this. The "larger model encodes parity more strongly but has lower accuracy" observation rests on two models and is speculative.
  • A possible confound is that Qwen's pretraining or synthetic data could produce the pattern; the paper does not test this.

    • Potential Impact

    The finding may spark follow-up mechanistic work, such as probing mod-2 versus mod-10 representations in Qwen models. It adds a small data point to the "numbers as Fourier or helical features" line of work and offers a cheap diagnostic (parity-preservation rate) that others can reuse. Practical impact is minimal: tool use already circumvents arithmetic failures, and small-model error structure is unlikely to matter in deployment. The paper might inform error-correction or steering ideas, but this is distant. As a workshop paper with a hypothesis-generating style, it is likely to be cited modestly, mainly by interpretability-of-arithmetic papers.

    Timeliness & Relevance

    Arithmetic interpretability is an active area (Fourier features, helix, digit-by-digit circuits, sinusoidal fidelity). Extending from addition to multiplication is a natural next step, and understanding error structure rather than accuracy alone is a reasonable angle. The model choices are current, but small, so the findings may not transfer to frontier models.

    Strengths & Limitations

    Strengths: a clear and checkable empirical observation with a large effect size (92.7% vs ~44%); careful null baseline construction; honest caveats about the units column and statistical independence; released code; and a low-cost design that others can replicate.

    Limitations: the work is correlational and behavioural, and the mechanistic interpretation is speculative. The effect is model-specific, so the headline claim about "LLMs" is narrow. The negative result on noisy quantisation is based on a single model and tier. Error-magnitude distributions and conditioning on correct upstream digits are not analysed. The tokenisation and accuracy confounds across models are acknowledged but not controlled. The abstract overstates the conclusions. The paper is short and reads as preliminary.

    Overall

    This is a modest but interesting empirical note. It is likely to be a minor citation within the arithmetic-interpretability niche, and its value depends on whether follow-up mechanistic work confirms the parity story. The surprising parity result has some intrinsic appeal, but the limited evidence and scope keep expected impact low to moderate.

    Rating:4/ 10
    Significance 3.5Rigor 4Novelty 5Clarity 6

    Generated Oct 30, 2026

    Comparison History (0)

    No comparisons yet.