MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference

Renjie Wei, Haochen Huang, Chuyu Qiu, Jinqi Wen, Dehao Xu, Meng Li

Nov 1, 2026arXiv:2611.01704v1

cs.LGcs.AI

Scorecard· 16/16

6.0/10 impact

Well-engineered algorithm-system co-design with solid ablations and real hardware speedups, but the novelty is largely a recombination and baselines/controls are limited.

Abstract

Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) since it reduces the computational cost through sparse expert activation. However, the massive parameter scale of MoE models hinders their deployment on resource-constrained edge devices. Although existing mixed-precision quantization reduces the model size, they are inherently static, fixing the bit-width of each expert across decoding steps. Such static schemes yield suboptimal performance as they overlook the dynamic variation in expert importance during decoding. Furthermore, they fail to adapt to the dynamic latency constraint driven by the fluctuating system utilization. To address these challenges, we propose MatMoE, a dynamic mixed-precision quantization framework for efficient MoE inference. MatMoE features: (1) a Matryoshka MoE PTQ algorithm that enables experts to operate flexibly at multiple precisions within a stringent memory budget; (2) a floor-and-compensation quantization method that eliminates redundant memory accesses in the Matryoshka representation; and (3) a dynamic bit-width allocation strategy that determines the precision for each expert based on their routing scores to satisfy the dynamic latency requirement. We implement our MatMoE system atop llama.cpp for both CPU and GPU platforms. Extensive evaluations across three modern MoE models and challenging benchmarks demonstrate that MatMoE achieves $4.0\% \sim 18.3\%$ higher accuracy than state-of-the-art static mixed-precision method. Compared to llama.cpp, MatMoE delivers $1.3\times \sim 3.7\times$ speedup on Raspberry Pi 5, and $1.2\times \sim 1.3\times$ speedup on NVIDIA RTX 4090 and Jetson AGX Orin.

AI Impact Assessments

(1 model)

Core Contribution

MatMoE combines three ideas for running MoE LLMs on edge devices: Matryoshka-style nested multi-precision quantization, per-token dynamic bit-width assignment at expert granularity, and a co-designed llama.cpp implementation for CPU and GPU.

Offline, an ILP picks each expert's maximum bit-width (2/3/4) under a memory budget. The metric is routing score × Fisher-approximated Hessian × quantization error.

An OmniQuant-based PTQ then trains shared quantization parameters so that the 2-bit prefix and the higher-precision representation both work.

A "floor-and-compensation" (F&C) trick replaces Matryoshka Quantization's extra rounding bit. It truncates the top bits and adds a constant 2^Δ−1)/2 to offset the expected bias of the dropped residual.

At runtime, experts whose routing score exceeds a threshold run at high precision and the rest at low precision. A binary search over a calibration score distribution sets the threshold to hit a target average bit-width.

Bitplane storage, a LUT-based CPU kernel (extending T-MAC to MoE) and a dp4a GPU kernel complete the system.

The conceptual novelty is mostly in the combination. Per-token, per-expert dynamic precision for MoE is a natural but previously unexplored fit, because routing scores give a free importance signal. The F&C trick is simple and practical, though its theory is thin: a constant bias correction for a uniform residual.

Methodological Rigor

The design is generally sensible. It uses three MoE models, three generative benchmarks (GSM8K-CoT, HumanEval, MBPP), a SOTA static mixed-precision baseline (PMQ), and an ablation that separates kernel, dynamic allocation and F&C contributions. The ablation is the most convincing part: PMQ with the new kernels gives the same accuracy as PMQ on llama.cpp but is 2.9× faster, and MatMoE at similar TPOT raises accuracy from 22.8% to 35.3%.

Weaknesses:

Confounded comparison. MatMoE uses OmniQuant-style learned training, while PMQ uses GPTQ. Part of the accuracy gain may come from the better PTQ and the better ILP metric rather than from dynamic allocation. The only partial control is "Matryoshka + Dynamic" versus MatMoE.

Unclear fairness at equal average bits. PMQ's bit-widths at "target precision" are static, whereas MatMoE's average bit-width is enforced only in expectation. Real per-token bit-widths vary, and the paper does not report the variance in latency or memory traffic.

Missing baselines. There is no comparison to uniform 2/3/4-bit GPTQ/OmniQuant, and no dynamic baseline such as DP-LLM or a layer-wise Matryoshka variant. The headline "4.0–18.3%" gain is relative to one baseline.

Evidence for F&C is thin. Table 2 shows F&C is similar to Matryoshka Quantization, even slightly worse on MBPP (15.8 vs 17.0). The 0% result for plain floor is striking and suggests either a collapse or a bug, and it is not analyzed further. The uniform-residual assumption is only illustrated with a histogram.

Weak motivational evidence. The claim that expert importance is dynamic rests on one layer of one sample. No statistics are reported.

No statistical reporting. There are no seeds, error bars or significance tests, and accuracy is noisy on small benchmarks such as HumanEval (164 problems).

Baseline strength unclear. Speedups over llama.cpp are measured against a baseline that does not use the same low-bit formats at the same precision. The 3.7× multi-thread speedup on Raspberry Pi is large and partly reflects the weak llama.cpp MoE CPU path.

Offloading comparison. The HOBBIT comparison uses the authors' own simulator, since no code is available.

GPU gains. They are modest (1.2–1.3×) and flat across precisions, because the kernel converts to INT8 before dp4a. This undermines the claimed latency adaptivity on GPU.

Potential Impact

The work is relevant to on-device MoE inference, where memory is the bottleneck. A single nested model that supports multiple operating points, with a cheap runtime knob, is practically attractive. Open-source integration into llama.cpp could allow adoption. The nested-bitplane design also benefits SSD-offloading regimes, as the Qwen3-30B-A3B experiment shows (1.1–2.0× over HOBBIT). The ideas of routing-score-driven precision and nested representations are reusable by other MoE inference systems. However, the gains are incremental over existing multi-scale and MoE quantization methods, and the venue (ICCAD) suggests a systems and co-design audience.

Timeliness

MoE models (Qwen3, DeepSeek, OLMoE) and edge LLM deployment are active areas. Dynamic precision under fluctuating load is an emerging need. The models evaluated are somewhat dated (Qwen1.5-MoE, DeepSeek-V2-Lite), and larger modern MoEs (Qwen3-30B, Mixtral-class) appear only in the offloading experiment.

Strengths

Algorithm–system co-design with a real implementation on three hardware platforms.

Clear component-level ablation that separates kernel, allocation and F&C effects.

A simple, deployable runtime control (a threshold via binary search).

Large accuracy gains at low average bit-widths, especially on OLMoE and Qwen.

Limitations

Mild novelty: it assembles existing pieces (Matryoshka Quantization, OmniQuant, T-MAC, Fisher-based ILP).

Confounded baselines and no variance reporting.

Evaluation limited to batch size 1, a few models and weight-only quantization.

GPU benefits are limited by the kernel design.

Dynamic precision may cause irregular latency and cache behavior, which is not studied.

Latency adaptivity is demonstrated only through static target precision sweeps, not under real fluctuating system load.

Code release is unclear.

Rating:6/ 10

Significance 5.5Rigor 5.5Novelty 5.5Clarity 7

Generated Nov 3, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

6.0/10 impact

Well-engineered algorithm-system co-design with solid ablations and real hardware speedups, but the novelty is largely a recombination and baselines/controls are limited.