Quantitative Gaussian-Process limits of Tensor Programs
Andrea Agazzi, Eloy Mosig García, Dario Trevisan
Abstract
We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance. Our main result gives explicit finite-width error bounds, of order inverse square-root of the widths between finite-network executions and their Gaussian-process limits. The framework is architecture-agnostic and covers feed-forward models together with weight-sharing schemes relevant for recurrent and transformer-type architectures.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper establishes quantitative finite-width error bounds for the convergence of neural network outputs to their infinite-width Gaussian process limits, formulated within the tensor program framework of Yang. The main result (Theorem 1.2) provides an explicit Wasserstein-p bound of order between the finite-width and infinite-width executions of a Netsor program. This is a meaningful advance because it lifts previously architecture-specific quantitative CLT results (which existed primarily for fully-connected networks) to an architecture-agnostic framework covering feed-forward networks, recurrent networks with weight sharing, and—via the NetsorK extension—attention-based architectures.
The companion result (Corollary 1.3) provides a quantitative law of large numbers for empirical kernel matrices, which is both a consequence and an ingredient of the main theorem's proof, used in a carefully structured inductive argument.
Methodological Rigor
The proof strategy is well-constructed: a line-by-line induction over the tensor program, where the kernel LLN for the truncated program feeds into the next induction step without circularity. The key technical ingredients—structure lemmas (3.5 and 3.6) characterizing conditional Gaussian laws via Gram matrices and generalized Schur complements, Greville's formula for pseudoinverse updates, and the non-degenerate representative reduction (Lemma 3.9)—are developed carefully.
The treatment of degeneracy is notable: the authors show that any Netsor program can be reduced to an equivalent non-degenerate program where all MatMul lines produce non-degenerate conditional Gaussians. This avoids technical issues with singular covariance matrices. The coupling construction at MatMul lines—using the same standard Gaussian noise for both executions—is a clean application of optimal transport ideas.
The extension to NetsorK (Theorem 2.4) handles the additional complexity of scalar kernel variables and locally Lipschitz nonlinearities needed for attention layers, though under an explicit non-degeneracy assumption. The authors are transparent about this limitation and explain precisely why the full Moment instruction of Yang's Netsor+ is harder to handle quantitatively.
One limitation is the Lipschitz continuity requirement on activation functions, which excludes ReLU in the strict mathematical sense (though ReLU is used in experiments). The rate, while matching classical Berry-Esseen scaling, is known to be suboptimal for fully-connected architectures where rates have been established by Trevisan (2023) and Favaro et al. (2025). The authors acknowledge this gap but do not discuss whether their framework could be sharpened.
Potential Impact
Theoretical impact: This work bridges the gap between Yang's qualitative master theorem for tensor programs and the quantitative CLT literature for specific architectures. It provides a unified quantitative framework that researchers can apply to new architectures without re-deriving convergence rates from scratch. The architecture-agnostic nature is the key selling point.
Practical relevance: Understanding finite-width corrections is important for calibrating neural network Gaussian process approximations used in Bayesian deep learning, uncertainty quantification, and neural architecture search. Explicit error bounds inform practitioners about when the infinite-width approximation is reliable.
Adjacent fields: The connection to quantum neural network GP limits mentioned in the related work, and the attention layer coverage, suggest broader applicability. The NetsorK formalism for attention is particularly timely given the dominance of transformer architectures.
Timeliness & Relevance
The paper addresses a genuine current need. While qualitative GP limits for broad architecture classes have been established via tensor programs, quantitative rates remained confined to specific architectures (MLPs, shallow networks). The concurrent works by Celli (2026) and Giovagnini-Kotitsas-Romito (2026) address related questions but through different approaches. This paper's contribution is distinguished by its use of the tensor program formalism itself as the vehicle for quantitative bounds.
The NetsorK extension covering attention layers is timely given the explosion of transformer research, though the coverage is still limited (single-head attention with LLN-scaling rather than the fluctuation scaling).
Strengths & Limitations
Strengths:
Limitations:
Generated Jul 8, 2026
Comparison History (25)
Paper 1 addresses the critical 'black box' problem of modern LLMs, offering high real-world applicability in AI safety, auditing, and control. While Paper 2 presents rigorous, foundational theoretical bounds for neural network scaling, Paper 1's focus on mechanistic interpretability addresses an urgent, widely applicable challenge. Its synthesis of cutting-edge techniques like Sparse Autoencoders with neurosymbolic AI makes it highly timely, guaranteeing broader interdisciplinary impact across AI alignment, practical deployment, and cognitive science compared to the narrower theoretical contributions of Paper 2.
Paper 1 provides a novel theoretical explanation for a widely observed empirical phenomenon (self-supervised learning's label efficiency), derives a concrete fast rate O(1/n_L) with explicit dependence on augmentation quality, and bridges theory and practice in a highly active area. Paper 2 provides quantitative finite-width bounds for GP limits of neural networks, which is technically solid but more incremental—qualitative GP convergence results already exist, and the practical relevance of these bounds is limited since infinite-width limits are increasingly seen as insufficient to explain real network behavior. Paper 1's broader relevance to SSL practice gives it higher impact potential.
Paper 1 addresses a practical and timely problem in reinforcement learning from reasoning (GRPO for LLM reasoning), offering a simple yet effective method (AdaPrefix-GRPO) with strong empirical results (2.1x accuracy improvement) and practical implementation ease. The approach is directly applicable to the rapidly growing field of LLM post-training for reasoning. Paper 2 provides rigorous theoretical contributions (quantitative GP convergence bounds for tensor programs), but its impact is more incremental within an established theoretical line of work, with less immediate practical relevance given the field's shift toward finite-width training dynamics and away from infinite-width GP limits.
Paper 2 likely has higher impact: it delivers quantitative finite-width convergence bounds to GP limits in Wasserstein distance via tensor programs, a broadly applicable theoretical result spanning many architectures (including transformers/weight sharing). This directly informs understanding, scaling laws, and approximation guarantees across modern deep learning, making it timely and widely reusable. Paper 1 is novel and rigorous, with clear scientific-ML applications, but its optimal-control/a posteriori depth adaptation is more specialized to architecture refinement and may have narrower uptake than a general theory of neural network limits.
Paper 1 provides quantitative, finite-width error bounds for Tensor Programs across modern architectures like transformers and RNNs, addressing a major foundational gap in deep learning theory and scaling laws. In contrast, while Paper 2 offers rigorous mathematical insights into safe training dynamics using Langevin Dynamics, its strict assumption of a strongly convex loss landscape severely limits its real-world applicability to modern, highly non-convex neural networks. Paper 1's architecture-agnostic framework ensures much broader relevance and immediate theoretical utility in the AI community.
Paper 1 introduces Agon, a novel competitive cross-model RL framework that addresses a fundamental limitation of current reasoning model training (grading only final answers). It demonstrates substantial empirical gains (doubling GRPO's pass@1 on hard benchmarks) across multiple model families and domains. The approach is timely given the explosion of interest in reasoning models, offers immediate practical applications, and opens new research directions (latent-space reasoning). Paper 2 provides rigorous theoretical contributions with finite-width convergence bounds for tensor programs, but its impact is more incremental within an established theoretical line of work, with narrower immediate practical implications.
Paper 2 likely has higher scientific impact due to its direct applicability to a major real-world need: privacy-preserving, communication-efficient federated learning under heterogeneity. It proposes a concrete, scalable system with formal (ε,δ)-DP guarantees, secure aggregation, and strong empirical performance across settings, making it timely and broadly relevant across ML, privacy, and distributed systems. Paper 1 is methodologically rigorous and novel in theory (quantitative Wasserstein bounds for GP limits via tensor programs), but its impact is more specialized and indirect for practitioners.
Paper 1 likely has higher scientific impact due to its foundational theoretical contribution: quantitative finite-width error bounds (Wasserstein) for GP limits via tensor programs, with architecture-agnostic coverage including weight sharing (recurrent/transformer-like). This advances methodological rigor and broad relevance across deep learning theory, statistics, and random matrix/probability, and can influence how practitioners reason about scaling laws and approximations. Paper 2 is timely and highly applicable clinically, but its technical novelty is more incremental (graph imputation + conformal/BH FDR control) and its impact is narrower to healthcare ML.
Paper 2 has higher impact potential due to timely relevance to widespread LLM training/evaluation pipelines (self-play, RLAIF, LLM judges), clear real-world implications (reward hacking and safety/reliability failures), and a concrete, generalizable mitigation (judge “commit-first” / blind solving) supported by strong empirical evidence and transfer across model families. It can influence multiple fields—alignment, evaluation methodology, RL, and deployment practices. Paper 1 is rigorous and broadly useful theoretically, but its incremental impact may be narrower and slower to translate into practice.
Paper 2 offers a fundamental theoretical contribution to deep learning by providing architecture-agnostic, finite-width error bounds for the Gaussian-process limits of Tensor Programs. While Paper 1 addresses an urgent societal problem with a novel applied approach, Paper 2's rigorous mathematical framework applies universally to modern neural architectures like Transformers. This foundational nature ensures Paper 2 will likely have a wider-reaching, longer-lasting scientific impact on understanding and scaling large AI models, influencing theoretical machine learning across multiple subfields.