Kurate.org
Rankings
Back to Rankings

Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs

Smit Kadvani, Kriti Faujdar, Spurthi Tallam

Nov 3, 2026arXiv:2611.04380v1
cs.AI
Share
Scorecard· 16/16
4.5/10 impact

Useful, honest evidence audit with a reusable noise yardstick and checklist, but no new measurements, a qualitative rule, and a short shelf life limit impact.

Abstract

A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We assemble every public measurement of these numbers that we could verify and find that the record supplies them only in pieces. Only two checkpoints, both INT4 builds of Gemma 3 12B, have both a vLLM timing and an accuracy score, and that score covers a single MMLU subject, one that the AWQ build was calibrated on. The batch-1 timings predate current engine releases, AMD and Intel Gaudi have no timing in this band at all, and FP4 has none against BF16. The timings that do exist follow one rule. Decoding a small batch is limited by memory bandwidth, so 4-bit weights roughly double speed (2.12x for Qwen2.5-14B on an A100), but the gain vanishes once a large batch makes the GPU compute-bound (1.02x for Gemma 3 12B at batch size 256), where FP8 is the recipe that can still help. Many published accuracy deltas, including every HumanEval delta, lie within benchmark sampling noise. We close with a seven-item reporting checklist for future measurements.

AI Impact Assessments

(1 model)

Scientific Impact Assessment: "Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs"

Core Contribution

This 7-page letter is an evidence audit, not an original benchmark. The authors ask whether the public record gives a practitioner serving a 9B–20B open model on vLLM or SGLang the two numbers needed to choose a quantization recipe: speedup over BF16 and accuracy lost. They gather verifiable timings and accuracy deltas from vendor docs, model cards (Red Hat AI, PyTorch, AngelSlim), and a few reports. They then make three contributions:

1. A roofline-style rule explains the timings. Small-batch decode is bandwidth-bound, so W4A16 gives about 1.35–2.12×. Large-batch or long-prompt work is compute-bound, so the gain collapses (1.02× at batch 256 for Gemma 3 12B) and FP8 is the recipe that could still help.

2. Many published accuracy deltas, including every HumanEval delta, lie within single-run binomial standard error. The authors compute these themselves because no source reports any.

3. A confidence-graded evidence table and a seven-item reporting checklist.

The most useful findings are about gaps. Only two checkpoints have both a vLLM timing and an accuracy score, and that score is a single MMLU subject on which the AWQ build was calibrated. AMD and Gaudi have no in-band timings, and FP4 has no BF16 comparison.

Methodological Rigor

The search protocol is described, with named sources, a date cutoff, an inclusion criterion (the source must name engine, GPU, checkpoint, and workload), and numbers verified against sources. The standard-error yardstick is simple and correct as a lower bound, and the authors state its limits. They note that paired flips between runs are unreported and that questions are assumed independent. The observation that Qwen3-14B BF16 baselines differ by 20.77 GSM8K points across harnesses is a sharp demonstration of harness sensitivity. The authors also disclose that an AI assistant helped extract numbers.

Weaknesses:

  • The "one rule" is a qualitative roofline argument, not a quantitative model. It is never fitted or tested against the data, and the data are too sparse to test it. The claim that it "explains all" timings rests on about 10 numbers from heterogeneous setups.
  • The Table 1 groups use different metrics (tokens/s, latency, prompt-inclusive throughput), so cross-group inference is limited.
  • The batch-256 claim rests on one model, one GPU, and one source, and the FP8 prediction at large batch is untested.
  • Calling a delta "within noise" does not show it is zero. Absence of significance is not equivalence, and the paper offers no power analysis.
  • No original measurements were made, so the paper cannot close any gap it identifies.

    • Potential Impact

    Practitioners get a clear picture of what is and isn't known. The takeaways are that INT4 is a memory choice rather than a throughput choice at high batch, that FP8 is the likely winner for throughput, and that GSM8K should be checked for INT4. The checklist is sensible and cheap to adopt, and it could shape how vendors and model-card authors report results. It may also help reviewers and benchmark builders. The paper does not offer a new method and is unlikely to be heavily cited in the algorithmic quantization literature, though it could be cited as a reporting-standards argument. Its direct value will fade as engines and hardware change; the batch-1 timings it relies on already predate current releases.

    Timeliness & Relevance

    Self-hosted mid-size models and quantized serving are highly relevant. Hardware is shifting toward FP8, FP4 (NVFP4/MXFP4), and AMD alternatives, and the paper points to real gaps there. The reproducibility and "benchmark noise" theme matches broader community concerns about evaluation rigor. However, the memory-versus-compute explanation is well known to systems practitioners, as is the point that small differences on HumanEval are noise. The paper packages them for this specific band.

    Strengths

  • It is honest about what is not known, with an explicit confidence-graded table.
  • Statistical framing of the deltas is a concrete, easily reusable contribution.
  • It draws a practical decision-relevant distinction (latency-bound vs throughput-bound serving).
  • It flags contamination: AWQ calibration overlaps the evaluation subject.
  • The checklist is actionable, and the paper is concise and readable.

    • Limitations

  • It has no new measurements, so its conclusions are only as strong as scattered vendor model cards, which are themselves uncontrolled.
  • The sample is small (a handful of models and recipes), and gpt-oss-20b has only an engine comparison.
  • Because the evidence base is thin, the generality of the "twice as fast until it isn't" headline is limited. It is essentially one A100 data point and one H100 data point.
  • The paper is an independent-author letter with a time-sensitive web-source basis and a future-dated cutoff. Link rot and version drift will hurt longevity.
  • It does not run a quick experiment that would validate its own central rule, such as a batch sweep on an open GPU. That would have raised the paper's value considerably.

    • Overall

    A useful, honest, modest contribution. It will be valued as a reference for practitioners and a reporting-practice nudge, but it has limited novelty, no new data, and a short shelf life. Expected impact is moderate-low.

    Rating:4.5/ 10
    Significance 4.5Rigor 5Novelty 3.5Clarity 8

    Generated Nov 4, 2026

    Comparison History (0)

    No comparisons yet.