Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
Mert Yildiz, Pietro Spadaccino, Alexey Rolich, Francesca Cuomo, Andrea Baiocchi
Abstract
Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware. This setting introduces new challenges for resource allocation, dispatching, and scheduling, particularly under GPU memory constraints where partial CPU-GPU offloading and preemption become necessary. While existing systems primarily optimize throughput for a single model, comparatively little work addresses multi-model scheduling under these conditions. In this paper, we present an empirical study of how different LLMs behave across hardware platforms, focusing on the performance implications of layer offloading and preemption. We show that offloading leads to strongly non-linear and model-dependent degradation in decode throughput, with smaller models exhibiting sharper sensitivity to reduced GPU residency. We further demonstrate that preemption incurs substantial overhead, largely dominated by model state reload rather than key-value cache transfer, and that this cost varies significantly across models and hardware platforms. Additionally, we highlight the role of sequence length and interconnect bandwidth in amplifying data movement and execution inefficiencies. Based on these findings, we identify a set of key features that future schedulers must consider, including model-specific offloading sensitivity, workload characteristics, and the cost structure of preemption and data transfer. These insights provide guidance for the design of next-generation LLM serving systems capable of efficiently managing heterogeneous, multi-model workloads with hybrid CPU-GPU execution.
AI Impact Assessments
(1 model)Scientific Impact Assessment
Core Contribution
This paper presents an empirical study of two critical operations in multi-model LLM serving: (1) partial CPU-GPU layer offloading and (2) job-level preemption with KV cache migration. The main findings are:
The paper distills these into five key features for future multi-model schedulers: non-linear offloading sensitivity, model architecture dependence, sequence length interaction, preemption cost structure, and hardware awareness.
Methodological Rigor
The experimental methodology is reasonably systematic but has notable limitations:
Strengths:
Weaknesses:
Potential Impact
The paper's primary value lies in establishing empirical baselines that the systems community needs but has largely neglected. Specifically:
1. The "flat curve" finding for preemption is practically useful: schedulers can model preemption cost as a constant per (model, hardware) pair rather than tracking generation progress. This simplifies scheduler design significantly.
2. The non-linear offloading curves provide actionable data for placement algorithms. Knowing that the last 5-10% of GPU residency yields disproportionate throughput gains can inform resource allocation.
3. The dominance of model reload over KV cache transfer redirects optimization efforts: investing in faster model loading (e.g., memory-mapped weights, pre-staged CPU buffers, faster storage) would yield far greater returns than optimizing KV cache transfer for preemption scenarios.
However, the practical impact is limited because:
Timeliness & Relevance
The paper addresses a genuinely important and timely problem. Multi-model serving is increasingly common in production (as evidenced by systems like Aegaeon deployed at Alibaba), and the gap between single-model optimization and multi-model scheduling is well-recognized. The paper correctly identifies that most prior work evaluates at fixed configurations rather than sweeping the offloading ratio as a continuous variable.
However, the paper's contribution is primarily characterization rather than a solution, which limits its immediate impact compared to systems papers that also build and deploy solutions.
Strengths
1. Systematic parametric sweep: Varying offloading fraction as a continuous variable is genuinely underexplored.
2. Clean decomposition of preemption costs: The four-step breakdown provides clear, actionable insights.
3. Cross-platform consistency: Testing on two GPUs adds credibility to qualitative claims.
4. Clear presentation: Results are well-organized with effective figures and tables.
5. Practical takeaway: The "fixed penalty per model-hardware pair" finding is immediately useful for scheduler designers.
Limitations
1. No scheduler implementation: The paper identifies features but doesn't validate that using them improves scheduling decisions.
2. Limited model diversity: 3 models per experiment is insufficient for strong generalization claims.
3. Single-request execution only: Ignores batching effects, which fundamentally change memory pressure and throughput dynamics.
4. No MoE models: Given that MoE architectures have fundamentally different activation patterns, this is a significant gap.
5. Storage-bound reload: The finding that model reload dominates is partly an artifact of loading from SSD rather than pre-staged CPU DRAM, which would be the more realistic scenario in optimized multi-model serving systems.
6. No theoretical modeling: The paper is purely empirical with no predictive models fitted to the data.
7. Conference venue: Targeting MAIN 2026 (a regional conference), which suggests the contribution may be considered incremental by top-tier venues.
Overall Assessment
This is a competent empirical study that fills a genuine gap in understanding offloading and preemption costs for multi-model LLM serving. The findings are useful, particularly the constant-cost preemption result and the non-linear offloading curves. However, the contribution remains at the characterization level without delivering a system, algorithm, or predictive model. The experimental scope (3 models, single requests, 2 GPUs) limits generalizability. The paper provides a reasonable foundation for future work but represents an incremental contribution to the field.
Generated May 20, 2026
Comparison History (23)
Paper 1 likely has higher scientific impact: it targets a core bottleneck in real-world LLM infrastructure—multi-model serving under GPU memory constraints—where empirical characterizations of offloading/preemption costs can directly influence scheduler and systems design across industry and academia. Its findings are broadly applicable to many deployment settings and could inform future hardware/software co-design. Paper 2 is novel and useful for qualitative analysis workflows, but its impact is more domain-specific, and evaluation via similarity-to-human codes may limit perceived rigor and generalizability compared to systems-level performance insights.
Paper 1 presents a novel, validated pipeline applying multi-agent LLMs to a clinically important problem (detecting delusional content in naturalistic speech), combining methodological innovation with direct real-world mental health applications. It offers concrete findings about prompt design and agent adjudication strategies with strong performance metrics. Paper 2, while useful, is primarily an empirical benchmarking study of LLM scheduling/offloading behavior that provides incremental engineering insights without proposing a new system or method. Paper 1's cross-disciplinary impact (NLP + clinical psychiatry) and novelty of the application domain give it higher potential impact.
ScenePilot presents a novel framework with a clear methodological contribution (feasibility-guided boundary-driven scenario generation using constrained multi-objective RL), demonstrates quantitative improvements (+6.2pp collision rates while preserving physical validity), and addresses a critical practical need in autonomous driving safety testing. It introduces concrete innovations (RSS-derived feasibility scoring, step-level shielding, boundary-band concept) with reproducible code. Paper 1, while useful, is primarily an empirical measurement study identifying features for future schedulers rather than proposing new methods, limiting its immediate impact.
Paper 2 likely has higher impact due to timeliness and breadth: multi-model LLM serving on heterogeneous hardware is a central, fast-moving systems bottleneck with immediate industry relevance. Its empirical characterization of offloading/preemption costs across models/platforms can inform scheduler design broadly (systems, ML infra, datacenters) and generalize beyond a single benchmark. Paper 1 is novel and rigorous within VRP learning-to-search and shows strong benchmark gains, but its applicability is narrower to combinatorial optimization routing and may have more limited cross-field and near-term deployment impact.
Paper 2 addresses a critical bottleneck in the real-world deployment of LLMs: efficient multi-model serving on shared hardware. Its insights into offloading and preemption provide foundational guidance for next-generation LLM schedulers, offering broad and immediate practical impact across the AI infrastructure ecosystem. While Paper 1 offers valuable behavioral insights into sycophancy, Paper 2's potential to optimize computational resources and scalability gives it a higher overall scientific and practical impact.
Paper 2 addresses a critical and highly timely bottleneck in AI deployment: serving multiple LLMs on constrained hardware. Its empirical insights into offloading and preemption directly inform the design of next-generation serving systems, offering massive potential for real-world application and cost reduction in cloud infrastructure. While Paper 1 introduces a valuable niche reasoning benchmark, Paper 2's focus on systems optimization has a broader, more immediate impact across the machine learning and distributed systems communities.
SceneCode introduces a novel framework bridging indoor scene synthesis with executable, editable programmatic representations—addressing a concrete gap in embodied AI, robotics, and simulation. Its contributions span multiple fields (scene generation, articulated object synthesis, robot interaction) and offer immediately usable outputs (simulation-ready assets). Paper 1 provides useful empirical insights for multi-model LLM scheduling but is primarily a benchmarking/profiling study offering guidelines rather than a new method or system, limiting its transformative impact.
Paper 1 introduces a novel framework (POST) addressing a clearly defined problem (spatial over-generalization) in multivariate time series anomaly detection with a new adversarial learning paradigm, a new benchmark with channel-wise annotations, and demonstrates SOTA results. It offers methodological innovation, reproducible artifacts, and advances both detection and localization. Paper 2, while timely and practically useful, is primarily an empirical study of existing LLM serving behaviors without proposing a new system or method—it identifies design considerations rather than solutions, limiting its direct impact.
Paper 2 has higher likely impact: it introduces a new, scalable benchmark (POLAR-Bench) targeting an urgent, widely relevant problem—privacy-utility trade-offs for LLM agents under adversarial interaction—enabling standardized evaluation across many models and domains. Benchmarks often catalyze rapid follow-on work (training, alignment, red-teaming, regulation, and product safety). Its methodology (multi-domain dataset, controlled axes for policy/attack, deterministic scoring) supports broad adoption. Paper 1 offers valuable empirical systems insights but is less novel and narrower to LLM serving/scheduling research.
Paper 1 offers higher potential scientific impact by introducing a novel algorithmic solution to a fundamental challenge in LLM alignment: diversity collapse during reinforcement learning. By replacing computationally expensive scalar reward models with pairwise preference rewards and group-based diversity metrics, it advances open-ended text generation. While Paper 2 provides highly valuable empirical insights for LLM systems engineering, Paper 1 contributes a scalable, methodological innovation to the core AI training paradigm, likely generating broader downstream applications and citations across the rapidly evolving field of LLM alignment.