ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration

Anshu Raina, Yao Fu, Zhang Jiejing, Peyman Razaghi, Shiksha Patel, Ravi Gupta, Deval Shah, Adeem Jassani

Nov 2, 2026arXiv:2611.04032v1

cs.AIcs.DC

Scorecard· 16/16

6.5/10 impact

Practically valuable, thoroughly validated serving-performance predictor with a clear ~20x measurement saving, but single-platform, incremental over prior hybrid-projection work, and with a thinly evaluated agent component.

Abstract

Serving an LLM efficiently depends on the engine, how the model is split across GPUs, precision, scheduler settings, speculative decoding, KV-cache management, and prefill-decode disaggregation. The best configuration depends on the model, hardware, workload, and objective. Operators usually find it by trial and error. Exhaustively measuring the candidates in our study would take about 25,000 GPU-hours. ROCm InferaSim predicts serving performance from a few short measurements. Choices such as batch size, prompt length, and GPU count change the amount of work per GPU without changing the kernels. ROCm InferaSim benchmarks each distinct set of kernels once, on at most one node. We call this measurement an anchor. Analytical laws carry that anchor to other deployments that run the same kernels. For multi-node deployments, ROCm InferaSim reduces the configuration to a single-node equivalent, measures it, and scales the result analytically. A simulator replays multi-turn traffic, and a tuning agent searches configurations using the predictions. On AMD Instinct MI355X GPUs with vLLM, SGLang, and ATOM, 11 anchors took 9.1 GPU-hours, against 204 GPU-hours to measure the 226 fixed-length configurations they cover. On 172 configurations, time per output token has 8.1% median error. On a multi-turn coding workload, throughput is within 8%. For 159 disaggregated deployments, none used as an anchor, throughput is within 5.5%. A five-candidate short list contains a deployment within 12% of the best feasible pairing of model and latency target.

AI Impact Assessments

(1 model)

Core Contribution

ROCm InferaSim is a performance-prediction framework for LLM serving. Its central observation is that many serving choices (batch size, context length, TP/EP/PP width, replicas, host KV offload) change per-GPU work but not the kernels that run. The framework therefore measures one "anchor" per kernel set, on at most one node, and transfers it analytically to other deployments. The key technical insight is that transfer must preserve the fixed per-step cost (launch, sync, sampling, scheduling) and scale only the GPU-divisible part. This cuts error on a DeepSeek-V4-Pro TP4→TP8 transfer from 44% (naive halving) to 0.9%.

The paper extends a training-side "reduce–measure–restore" idea to inference. Its additions are:

disaggregated prefill/decode pools, priced from colocated anchors;

host-KV tiers;

pipeline hop and host-overlap models;

recorded-router expert imbalance;

a discrete-event simulator for multi-turn agentic traffic with a prefix cache;

a tuning agent (an LLM proposes configurations, the predictor scores them) and a mix planner.

Methodological Rigor

The paper is empirically thorough, and several design choices are careful:

Anchors are fixed before scoring.

Held-out evaluation is used throughout; for example, 159 disaggregated splits are predicted from colocated anchors only.

The paper reports ranking fidelity (pair accuracy, top-1, regret) rather than only absolute error, which is the right metric for search.

It compares against Vidur ported to MI355X (34–40% error against 7–14%).

It includes component ablations (PP model against the serial model, the EP busiest-rank factors, harvest-condition checks such as real weights and prefix caching off) and noise floors for run-to-run variation.

Weaknesses:

Everything runs on a single hardware platform (MI355X), and the generality claim is untested.

Anchor applicability is checked by metadata fields, not by kernel traces.

The agentic accuracy figures cover only runs where an anchor applies (95 of 184).

Error is moderate: 8% median TPOT, 13.6% TTFT, 23% TTFT factor on agentic traffic.

The tuning-agent evaluation is thin. The 192 searches used only the deterministic seed stage with no LLM, and the mix planner is validated against a finite pool of already-measured runs, so the "best" is the best among candidates rather than a global optimum.

Several terms are fitted (host time, Bnode, the gpt-oss coverage curve), and some validation sets are small (nine EP runs, three SGLang EP runs).

The 25,000 GPU-hour figure is extrapolated, not measured.

The comparison with AIConfigurator and others is by reported numbers, not head-to-head.

Potential Impact

This is a practical systems tool. Operators and inference-platform teams spend large amounts of GPU time on configuration sweeps, and the roughly 20× measurement saving (9.1 against 204 GPU-hours) is compelling. The findings on expert imbalance are reusable: random-token benchmarks overstate imbalance, and imbalance must be evaluated at both the anchor's and the target's EP. The finding that disaggregation pays mainly at long prompts and mid-tier interactivity is operationally useful. Support for AMD hardware, which most existing simulators lack, matters for the ROCm ecosystem. The planned release of anchors and scripts would raise this value, but it is promised only "with the camera-ready", so it is not yet available.

Timeliness

The paper is highly timely. The configuration space has exploded (MoE, EP, disaggregation, speculative decoding, agentic multi-turn workloads, KV offload), and trial-and-error is expensive. Agentic prefix-reuse workloads are an emerging need that many simulators do not handle.

Strengths

The simple, principled insight about the fixed per-step cost, with a large demonstrated gain.

Broad validation across 3 engines, many models, and fixed-length, agentic, disaggregated and multi-node settings.

Ranking-centric evaluation and honest reporting of failure modes (a 2.01 ITL factor for DeepSeek-V4-Pro on SGLang, the 33.8% throughput error at TP8×PP2).

Detailed appendices that make the modeling transparent.

Limitations

It is vendor-authored and single-platform, so the claim that it generalizes to NVIDIA is untested.

The method needs a physical harvest for each kernel set (precision, backend, graph mode, speculative method, engine build). The kernel-set space grows combinatorially, and any engine update requires re-harvesting.

The LLM agent is a secondary component with little evidence of its added value over grid or Pareto search.

The paper is dense and sprawling, which hurts readability.

It is incremental relative to the authors' own training-prediction work and to hybrid measured-analytical simulators.

Overall

This is a solid, engineering-heavy systems contribution with strong practical value, particularly for AMD deployments and agentic workloads. It does not offer a fundamental conceptual breakthrough, and its generality is not yet proven.

Rating:6.5/ 10

Significance 6.5Rigor 7Novelty 5.5Clarity 6

Generated Nov 4, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 16/16

6.5/10 impact