How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
Leena Chennuru Vankadara, Moritz Haas, Luke Hayward, Sebastian Bordt, Alessandro Breccia
Abstract
Recent frontier large language models predominantly rely on Mixture-of-Experts (MoE) architectures. Despite empirical progress, there is still no principled understanding of how hyperparameters should scale with network width , expert width , number of experts , sparsity , and depth to ensure both stability and optimal performance at scale. We take a principled step toward resolving this gap by analyzing three different scaling regimes: (I) co-scaling , (II) co-scaling , and (III) full proportional scaling of , and . For each regime, we develop a novel Dynamical Mean Field Theory (DMFT) description of the limiting training dynamics of MoEs that provides a formal foundation for our analysis. Within this framework, we derive the unique parameterization for SGD and Adam satisfying all maximal-update () desiderata. We then show that the resulting P prescription does not reliably induce monotonic improvement with scale or robust learning-rate transfer. We trace these pathologies to scale-dependent observables in the aggregation dynamics, which motivates a refined set of desiderata that we term maximal scale stability. Guided by this principle, we derive a Maximally Scale-Stable Parameterization (MSSP) for both SGD and Adam in all three scaling regimes, and characterize the corresponding limiting dynamics - qualitatively distinct from the P limit - through a separate DMFT analysis. Experiments verify that MSSP robustly recovers learning rate transfer and monotonic improvement with scale across regimes. Combined with existing depth-scaling theory, these results provide a complete scaling prescription for MoE architectures as a function of width, depth, expert width, and number of experts.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper addresses a critical gap in the principled scaling of Mixture-of-Experts (MoE) architectures — now dominant in frontier LLMs (Mixtral, DeepSeek-R1). The authors identify that the standard Maximal Update Parameterization (µP), which works well for dense networks, fails to deliver its promised benefits (learning rate transfer, monotonic improvement with scale) when applied to MoEs in regimes where the number of experts M and/or expert width Ne scale with network width N.
The paper introduces the Maximally Scale-Stable Parameterization (MSSP), which strengthens the µP desiderata by requiring that *every* primitive interaction term (init, propagating, effective) in both forward and backward passes remains Θ(1). The key insight is that cross-expert aggregation in MoEs creates CLT-vs-LLN imbalances that µP's forward-only desiderata miss. The authors derive regime-specific fixes: amplified expert-output initialization variance (σ² = M/Ne instead of 1/Ne) in Regime II, and shared expert initialization in Regime III.
Methodological Rigor
The paper is exceptionally thorough in its theoretical treatment:
1. Three complementary levels of analysis: intuitive scaling arguments (Appendix E), detailed signal propagation analysis tracking all terms through 3 forward and backward passes (Appendix F), and rigorous Dynamical Mean Field Theory (DMFT) derivations (Appendices G-K).
2. The CLT/LLN taxonomy (Mechanisms A-D in §E.7) provides a clean classification of how cross-expert sums behave, identifying precisely which terms are suppressed by 1/√M and why. This is the analytical backbone that makes the imbalance diagnosis convincing.
3. DMFT contributions are novel: The Regime III MSSP limit exhibits a four-level conditional mean-field hierarchy (global → shared expert-hidden → expert/router → within-expert), qualitatively distinct from µP's three-level hierarchy. This is a genuine theoretical contribution beyond the scaling prescriptions.
4. Exhaustive experimental validation: The MLP experiments involve 5⁶ = 15,625 hyperparameter configurations per setting at base width, which is unusually thorough. The transformer experiments scale to 2.5B parameters on 2.5B tokens.
However, the DMFT derivations are at "physicist rigor" — no convergence proofs are provided. The signal propagation analysis tracks only the first few steps heuristically, though the empirical verification over 1000 steps suggests the predictions hold more broadly.
Potential Impact
Immediate practical value: Table 1/B.1 provides a complete, actionable recipe for scaling modern Transformer MoEs across width, depth, expert width, and number of experts. This is directly applicable to frontier model development.
Regime II (bottleneck experts) is particularly timely, as the trend toward fine-grained experts (8/256 in DeepSeek-R1) makes this regime practically relevant. The finding that µP fails here — and that a simple variance amplification fixes it — has immediate engineering implications.
Broader theoretical framework: The maximal scale stability principle generalizes µP's desiderata and could apply to other architectures with sparse routing, conditional computation, or aggregation operations where CLT/LLN imbalances arise.
The DMFT limits open routes to studying expert specialization dynamics, finite-size corrections, and the role of auxiliary losses — all currently open problems in MoE research.
Timeliness & Relevance
This paper arrives at a critical moment: MoE architectures dominate frontier LLMs, yet practitioners rely on ad hoc hyperparameter choices (e.g., the ubiquitous 0.006 initialization from Fedus et al., 2022). The paper directly addresses compute-optimal scaling — perhaps the most important practical question in current LLM development.
Strengths
1. Completeness: Three scaling regimes × two optimizers × two parameterizations, each with full signal propagation tables, DMFT derivations, and experiments. The 155-page appendix is a reference work.
2. The diagnosis is as valuable as the fix: The precise identification of *why* µP fails in MoEs (CLT/LLN imbalances in aggregation) provides generalizable insight. The mechanism taxonomy (§E.7) is pedagogically excellent.
3. Regime-specific fixes are structurally distinct: The fact that Regime II requires variance amplification while Regime III requires shared initialization (and these cannot be interchanged) reveals genuine architectural structure, not just parameter tuning.
4. Practical validation at scale: The transformer experiments (Figure 6) demonstrate clean LR transfer in MSSP where µP fails, at scales (2048 width, 2.5B tokens) relevant to practice.
Limitations
1. The Regime II divergence: The per-expert h₃,ᵢ scaling as Θ(√M) is argued to be benign but could cause numerical precision issues at very large M. The suggested fused kernel workaround is not validated.
2. Limited depth analysis: Depth scaling is deferred to existing theory (Dey et al., 2025), and the composability argument (Theorem E.3) is only sketched. Joint depth-width-expert scaling experiments would strengthen the claims.
3. Softmax routing: The theoretical analysis primarily uses sigmoid gating; softmax is handled by noting equivalence of scaling exponents. The interaction with auxiliary losses (load balancing, z-loss) is acknowledged but not analyzed theoretically.
4. Sparsity regime M/K → ∞: The practically important regime of increasing sparsity is explicitly noted as a limitation.
5. Compute-optimal comparison: The paper enables but does not perform the compute-optimal comparison across regimes, which would be the ultimate practical deliverable.
Overall Assessment
This is a landmark contribution to the scaling theory of MoE architectures. It combines deep theoretical analysis with practical prescriptions, addressing a problem of immediate relevance to frontier AI development. The quality and thoroughness of both theory and experiments substantially exceed the typical standard. The main limitations are in the gap between the analyzed settings and the full complexity of production systems.
Generated May 15, 2026
Comparison History (52)
Paper 2 addresses MoE scaling, foundational to frontier LLMs powering the current AI revolution. Its principled DMFT-based theory and MSSP prescription solve a critical, unsolved problem affecting virtually all large-scale model training, offering broad, immediate impact across the AI field. Paper 1 is solid and useful for materials design but targets a narrower engineering domain with incremental methodological novelty (combining VAE, diffusion, surrogates). Paper 2's timeliness, theoretical rigor, and enormous practical relevance to the dominant LLM paradigm give it substantially higher potential impact.
Paper 2 addresses a critical, timely problem in scaling MoE architectures that underlie frontier LLMs, providing a principled DMFT-based theory with verified experimental results (learning rate transfer, monotonic improvement). Its practical prescription directly impacts training efficiency at scale for a dominant architecture. Paper 1 offers an interesting copyright detection framework but relies on unavailable counterfactual assumptions and provides only theoretical instantiations without clear empirical validation. Paper 2's combination of rigorous theory and validated, broadly applicable results yields higher and more immediate scientific impact across the LLM community.
Paper 2 likely has higher scientific impact due to its fundamental, theory-driven contribution: a DMFT-based analysis of MoE training dynamics across multiple scaling regimes and a new Maximally Scale-Stable Parameterization with verified learning-rate transfer and monotonic scaling. This can influence how frontier MoE LLMs are designed and trained across academia and industry, with broad relevance to optimization, theory, and large-scale deep learning. Paper 1 is a strong, timely systems/quantization advance for diffusion transformers, but its impact is more specialized to efficient inference/PTQ on specific model classes and hardware kernels.
Paper 2 addresses a fundamental challenge in training frontier AI models by providing a principled theoretical framework and a novel parameterization (MSSP) for scaling Mixture-of-Experts architectures. While Paper 1 offers a highly practical systems-level optimization for long-context LLM inference, Paper 2's contribution to scaling laws and hyperparameter transfer has far-reaching implications. It eliminates costly hyperparameter tuning at scale, fundamentally impacting how future large-scale foundation models are trained. This provides broader methodological rigor, foundational scientific value, and wider industry impact across all MoE development.
While Paper 1 offers a rigorous framework for scaling MoE models in AI, Paper 2 demonstrates broader cross-disciplinary scientific impact. By framing X-ray crystallography refinement as Bayesian inference using diffusion priors, CrystalBoltz solves a major experimental bottleneck in structural biology. Achieving superior accuracy and a 33x speedup in protein structure determination directly accelerates drug discovery and molecular biology. Paper 2 bridges predictive AI and wet-lab experiments, offering profound real-world applications beyond the computational AI community itself.
Paper 1 provides a rigorous, principled theoretical foundation (via DMFT) for scaling MoE architectures, addressing a fundamental open problem in how hyperparameters should scale across width, depth, experts, and sparsity. MoE is central to frontier LLMs, so a complete, verified scaling prescription has broad, lasting impact and strong methodological rigor. Paper 2 offers a clever, timely RLVR optimization approach with promising empirical results, but its impact is narrower and more incremental, building on prior work with less foundational generality. Paper 1's combination of novel theory, breadth, and practical relevance gives it higher estimated impact.
Paper 1 addresses a fundamental, highly costly challenge in modern AI: scaling Mixture-of-Experts (MoE) architectures for frontier Large Language Models. By introducing a novel Dynamical Mean Field Theory and Maximally Scale-Stable Parameterization (MSSP), it solves hyperparameter transfer across scales, saving massive computational resources. While Paper 2 offers solid improvements in privacy-preserving distributed learning, Paper 1's theoretical rigor and immediate, critical application to the most dominant AI architectures today make its timeliness, industrial relevance, and potential for widespread paradigm-shifting impact significantly higher.
Paper 2 addresses a foundational open problem—principled hyperparameter scaling for MoE architectures that dominate frontier LLMs. Its DMFT-based theory and MSSP prescription offer broad, lasting impact on how large models are trained industry-wide, with strong methodological rigor and timeliness. Paper 1 is a solid, practical engineering contribution for CPU quantization, but its impact is narrower and more incremental relative to existing quantization work. Paper 2's theoretical generality and applicability to the dominant model paradigm gives it wider breadth and higher scientific significance.
Paper 2 likely has higher impact due to its broader, more foundational contribution: a principled scaling theory for MoE architectures (dominant in frontier LLMs), with DMFT-based derivations and a new parameterization (MSSP) intended to ensure stable training and learning-rate transfer across multiple scaling regimes and optimizers. This advances methodology for building larger models, affecting many labs and applications. Paper 1 is novel and practically useful for inference/RL efficiency, but is a more targeted improvement and may be easier to supersede by alternative decoding/training heuristics.
Paper 1 addresses a fundamental, timely problem: principled scaling of MoE architectures that underpin frontier LLMs. Its DMFT-based theoretical framework and derivation of MSSP fills a genuine gap with broad applicability to how large models are trained, potentially saving enormous compute. Paper 2 offers an elegant, practical control interface for flow matching, but its impact is narrower and more incremental. Paper 1's combination of rigorous theory, empirical validation, and direct relevance to the dominant paradigm gives it higher breadth and lasting impact.