Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Impact Assessment

Core Contribution. This paper addresses a specific, well-posed mechanistic question in sparse mixture-of-experts (MoE) models: *why does a token benefit from multiple experts rather than one?* The dominant intuition — "geometric complementarity," where co-selected experts contribute disjoint representation directions — is shown to conflate three distinct properties: route coherence, candidate quality, and candidate-by-context interaction. The paper's contribution is a diagnostic apparatus that cleanly separates these: (1) the Expert Subspace Separation Index (ESSI), which normalizes inter-expert subspace distance by within-expert local tangent dispersion; (2) a prefix-controlled 2×2 factorial whose difference-in-differences isolates the interaction term; and (3) frozen-route NLL interventions plus a matched-compute Top-1/Top-2 training study for functional value. The synthesized finding — "coherent overlap" (experts overlap geometrically, routes remain token-coherent, selected candidates are individually strong, yet the actual context *narrows* rather than amplifies their geometric advantage, while multi-expert computation still helps functionally) — is a genuine conceptual clarification. The practical upshot, that input-subspace overlap cannot be used as a proxy for redundancy or pruning value, is a useful cautionary result for the active MoE compression literature.

Methodological Rigor. This is the paper's strongest dimension. The factorial design is well-conceived and directly targets the confound (Figure 3's sign reversal when both candidate and context vary vs. when context is held fixed is a compelling demonstration of why naive comparisons mislead). The evidence is comprehensive: all 39 factorial cells reported with 95% bootstrap CIs, three sensitivity variants (raw gain, nearest-residual matching, strict caliper), an independent CPU tall-SVD numerical cross-check, and a three-seed matched-compute training study that controls parameters and FLOPs to <0.04%. Claims are appropriately hedged (15 of 39 frozen-route additions labeled "inconclusive" rather than overclaimed). The consistency of the negative interaction across architectures and variants is convincing.

Potential Impact. The work is diagnostic/interpretive rather than method-producing; it will not directly improve model performance. Its influence will most plausibly land on: (a) MoE interpretability researchers studying specialization, and (b) the pruning/merging/compression subfield, where the explicit warning against subspace-overlap-as-redundancy could redirect methodology. This is a meaningful but bounded slice of the field. The Top-2 > Top-1 matched-compute result is a small-scale confirmation of accepted practice rather than a driver.

Timeliness & Relevance. Highly timely. MoE architectures (Mixtral, DeepSeek, OLMoE, Qwen) are central to current LLM scaling, and the question of whether experts truly "specialize" is actively contested. The paper engages directly with a cluster of recent work challenging the specialization narrative (e.g., "myth of expert specialization," "standing committee" papers).

Strengths & Limitations. Strengths: rigorous factorial identification, extensive robustness checks, cross-architecture coverage (six models for geometry, three for full analysis), careful and honest scope statements. Limitations that cap impact: (1) the geometric metric is purely *linear, rank-128, on router inputs before the nonlinear expert transform* — the authors acknowledge this cannot capture nonlinear feature specialization, which arguably is where "specialization" actually lives; (2) the functional training study is a single small (six-layer) matched-compute configuration, so the causal claim rests on narrow evidence; (3) the headline conclusions, while cleanly demonstrated, are only mildly surprising given the field's growing skepticism of clean specialization. The contribution is more "conceptual hygiene / careful measurement" than a new capability.

Reproducibility & Reuse. The appendices are unusually thorough: seeds, split rules, support thresholds, anchor counts, corpus composition, and complete cell-level tables. No code release is mentioned, but public models and datasets plus the detailed protocol make independent replication plausible. ESSI and the prefix-controlled factorial are reusable diagnostic primitives, giving the paper moderate foundationality as an analysis toolkit.

Overall. A methodologically careful, well-executed analysis paper that sharpens a fuzzy but consequential question in MoE research and delivers a defensible, useful negative-ish result. It will be cited and its framework reused within MoE interpretability/compression, but it neither introduces a new capability nor overturns a load-bearing assumption dramatically enough to reshape the field.