Emergence of the transformer architecture as the score of entropic laws of random measures

Giacomo Greco

Oct 29, 2026arXiv:2610.39614v1

cs.AImath.PRstat.ML

Scorecard· 15/16

5.5/10 impact

Elegant, rigorous max-entropy derivation of attention heads, untied QK/V and causal masking, but partly circular, purely theoretical, single-layer, and without empirical consequences.

Abstract

We show that the transformer architecture emerges as the score of last-token laws in a natural statistical model of language, where sentences are empirical measures of tokens, a language is a law on $P (S^{d})$ , and our prior belief is that sentences carry no structure at all. By the maximum-entropy principle, we select the law closest to this prior in relative entropy among those reproducing an observed sentence-level statistic. We discuss the relevance of the statistic in the posterior interaction between tokens and we build a statistic, able to capture that interaction, with Luce's choice axiom truncated at the lowest relational order. This procedure gives the softmax choice rule, with the attention cross-moment as its mean parameter. The query--key matrix is then the Lagrange multiplier of a maximum-entropy problem inside each sentence, and the value matrix the multiplier of the constraint over the language, so that the two are untied. Under the optimal entropic law we then show that a transformer head appears in the score of the last-token law. For ordered sentences, causal masking follows from the masked reference choice rule, and with a next-token cross-moment the score equals one causally masked attention head. We also discuss multi-head attention, weight-tied unembedding, and a heuristic reading of residual blocks as projected Langevin steps.

AI Impact Assessments

(1 model)

Core Contribution

The paper offers a statistical "origin story" for softmax attention. Sentences are empirical measures on the sphere, and a language is a law on P(P(S^d)). The null model is structureless: i.i.d. uniform tokens with a random length. The author selects the KL-closest law that reproduces an observed sentence-level statistic. Pair-additive or moment statistics are shown to give no key competition. Relational structure is then built from Luce's choice axiom, and a harmonic truncation at lowest relational order gives the bilinear softmax rule with matrix Λ = QᵀK. The statistic is the attention cross-moment S_Λ, which is the mean parameter of the choice-rule exponential family. The Gibbs tilt has the value matrix V as its Lagrange multiplier, so Λ and V are untied. This contrasts with the free-energy statistic, which forces V = λΛ.

Unordered model: The score of the last-token law contains the transformer head, plus a key-side term and a correction term that do not vanish.

Causal model: Placing the mask in the reference choice rule and using a shifted next-token cross-moment makes the last-token law exactly von Mises–Fisher. Its score is one causally masked attention head, tangentially projected, with no extra terms.

Extensions: Multi-head attention (a sum over several statistics), weight-tied unembedding (projection onto the vocabulary, with an exact discrete version in Appendix B), and a heuristic reading of residual blocks as projected Langevin steps.

Methodological Rigor

The mathematics is careful and largely correct.

Theorem 2.1 is standard exponential-family and entropic-projection theory, rigorously adapted.

Non-degeneracy of S_Λ is proved nontrivially via exponential-polynomial independence on circles, which is a genuine technical contribution.

The first-variation computations and the Lemma 2.3 differentiability argument are detailed.

The proofs that the model is well-posed, with unique multipliers for any interior constraint, are sound.

The weaknesses are in the modeling logic rather than the proofs.

The "emergence" is conditional on a series of designed choices: the statistic is chosen to be the attention cross-moment, the shifted statistic is chosen precisely so that the extra terms vanish, and the harmonic truncation is a modeling choice that the author concedes is not necessarily dominant.

Because the statistic is itself defined through softmax attention, the result is partly circular. Attention appears twice, in the statistic and in the score. Luce's axiom justifies softmax over keys but not the specific use of this statistic.

The Langevin and Euler–Maruyama link to residual blocks is acknowledged as heuristic. The drift is fixed across steps, so depth is not explained.

The last-token law ignores the autoregressive structure of intermediate tokens (Remark 4.7), so the model is not a generative model of language in the usual sense.

There are no experiments, so there is no test of whether the model fits real language or yields new predictions.

The weight-tying claim depends on an arbitrary σ nonlinearity, and the identification with trained transformers is loose.

Potential Impact

This is a theory-flavored paper in the "why does the architecture look like this" genre. It connects to maximum-entropy language models (Berger, Rosenfeld), minimax entropy (Zhu–Wu–Mumford), discrete choice, and mean-field and measure-based views of transformers. Its main value is conceptual: untied QK/V as inner and outer multipliers, the explanation of why moment statistics produce no competition, and masking as a consequence of a masked reference measure. It does not change practice and offers no new architectures or training algorithms. Its suggestions, such as a fixed unembedding or higher-order harmonic attention, are speculative and untested. The audience is mathematicians working on transformer theory (probability, optimal transport, mean-field), with limited reach to mainstream ML practitioners. Its influence may be modest but real within the mathematical-theory niche, and it is an elegant derivation others could cite.

Timeliness & Relevance

Mathematical understanding of transformers is active (Geshkovski et al., Rigollet, Sinkformers, entropic OT readings of attention), so the paper is relevant. The shift from studying dynamics given attention to asking where attention comes from is a fresh framing. The paper positions itself well against Litman, Hopfield-network derivations, and Fraiman. However, the "derivations of attention" space is crowded, and derivations that do not yield predictions often have limited uptake.

Strengths

Coherent, mathematically rigorous framework with exact (not asymptotic) statements in the causal case.

Explains the untied query-key and value matrices, and the causal mask, in a principled way.

Honest about limitations: heuristic parts are flagged, and the depth and intermediate-token gaps are acknowledged.

A nontrivial non-degeneracy proof that works for non-symmetric Λ.

Good literature positioning.

Limitations

No empirical validation.

Circularity between the chosen statistic and the resulting architecture; the result is an existence-of-model statement rather than a uniqueness or necessity one, and other statistics could plausibly yield other architectures.

The shifted statistic is engineered to remove extra terms.

Only a single layer and a single token's law are explained; MLPs and depth are unexplained.

Dense presentation, with a long introduction that effectively duplicates the results.

The normalization 1/(n−1) makes the concentration vanish for long sentences, a sign that the model is a poor fit to real language.

Other Observations

Reproducibility is not applicable in the empirical sense. All proofs are included, and the AI-assisted-checking disclosure is notable. Clarity is decent but heavy: the notation is dense, with many nested definitions, though the Meta-Theorems help.

Rating:5.5/ 10

Significance 4.5Rigor 7Novelty 6.5Clarity 6

Generated Oct 30, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 15/16

5.5/10 impact

Elegant, rigorous max-entropy derivation of attention heads, untied QK/V and causal masking, but partly circular, purely theoretical, single-layer, and without empirical consequences.