Get early access

Solving LLM Serving Latency Interference

Cutting LLM inference latency with prefill-decode disaggregation and hybrid scheduling

Summary

LLM inference latency spikes when the two phases of serving compete for the same GPU: prefill, which processes the whole prompt at once and is compute-bound, and decode, which generates the answer one token at a time and is memory-bandwidth-bound 1, 4, 7. A long prompt's prefill stalls every decode batched with it, so streamed tokens stutter and Time-Per-Output-Token (TPOT) spikes; favouring decode instead makes new requests wait and inflates Time-To-First-Token (TTFT) 1, 6. The two main fixes are chunked prefill, which splits a prompt into chunks scheduled alongside ongoing decodes 8, and prefill-decode disaggregation, which runs the phases on separate GPUs 6, 7, 9. As prompts grow to tens and hundreds of thousands of tokens, modern high-throughput inference frameworks are moving beyond monolithic batching toward Prefill-Decode (PD) disaggregation, chunked prefill, and hybrid latency-shifting architectures that optimize end-to-end "goodput" under stringent Service-Level Objectives (SLOs) 1, 3.

Key Takeaways

  • Prefill vs. Decode Heterogeneity: Prefill is compute-bound, saturating GPU tensor cores with parallel matrix multiplications over prompt tokens, while decode is memory-bandwidth-bound, repeatedly loading model weights and cached Key-Value states for single-token autoregressive generation 1, 4.
  • Latency Interference in Aggregated Serving: In conventional co-located architectures, bursty prefill phases preempt or delay ongoing decode iterations, causing severe TPOT violations and inter-token streaming jitter 1, 4.
  • Architectural Disaggregation: Physically separating GPU instances into dedicated Prefill clusters and Decode clusters eliminates cross-phase interference, but risks instance underutilization when workloads do not exhibit extreme single-metric skew 1.
  • Hybrid Latency-Shifting (TaiChi): Recent research demonstrates that unifying aggregation and disaggregation through configurable instance ratios and request-level latency shifting (flowing decode and length-aware prefill) boosts hardware goodput by up to 77% under balanced SLO regimes 1.
  • Production Cache & Memory Optimizations: Industry systems leverage complementary techniques—Anthropic's prompt caching (cutting input latency by up to 85%), DeepSeek's Multi-Head Latent Attention (compressing KV cache footprints), and vLLM's PagedAttention (eliminating fragmentation)—to mitigate memory bandwidth saturation 2, 3, 4.

Problem Background

Modern enterprise generative AI applications—ranging from multi-turn coding assistants to long-document analysis workflows—routinely process input prompts spanning 10,000 to over 100,000 tokens.

What are prefill and decode in LLM inference?

Prefill reads the whole prompt and produces the first token; decode then produces every later token, one per step. Autoregressive transformer serving divides execution into these two structurally mismatched phases:

  1. Prefill (Prompt Processing): The model processes all prompt tokens simultaneously in parallel matrix multiplications to generate the initial Key-Value (KV) cache and emit the first token. This phase is compute-bound and governed by the Time-To-First-Token (TTFT) Service-Level Objective (SLO) 1, 4.
  2. Decode (Token Generation): The model generates subsequent tokens sequentially, one by one. Each forward pass requires reading all model weights and the cumulative KV cache from High-Bandwidth Memory (HBM) to compute a single new token. This phase is memory-bandwidth-bound and governed by the Time-Per-Output-Token (TPOT) SLO 1, 4.

Because decode doesn't need the compute of the newest GPUs, the Splitwise authors argue it can run on lower-power, lower-cost machines than prefill 7.

What are TTFT and TPOT?

They are the two latency numbers users feel, one per phase 6, 10:

Metric What it measures Set by
TTFT, time to first token How long a user waits before the answer starts Prefill
TPOT, time per output token How fast the answer streams once it has started Decode
ITL, inter-token latency The gap between consecutive streamed tokens; vLLM's name for the decode-side metric Decode
Goodput The highest request rate a GPU can sustain while meeting both the TTFT and TPOT targets Both

Goodput is the number to optimize: a system with high raw throughput but frequent SLO misses has low goodput 6.

Why do LLM token streams stutter?

When prefill and decode are executed within the same GPU workers (Prefill-Decode Aggregation), an incoming long-prompt prefill burst consumes the streaming multiprocessors. Active decoding sequences batched on the same GPUs are forced to wait, causing severe TPOT latency spikes and noticeable streaming jitter for end users 1.

Conversely, statically dedicating separate GPU pools to prefill and decode (Prefill-Decode Disaggregation) isolates execution and stabilizes TPOT, but introduces load-balancing fragility. If the incoming traffic shifts, one cluster starves while the other forms request queues, degrading TTFT and hardware goodput 1.

Proposed Experiments / Examples

To evaluate interference patterns and quantify the trade-offs between serving configurations, engineers can benchmark continuous batching, chunked prefill, and disaggregated topologies.

1. Workload Profiling Setup (Simulation)

from dataclasses import dataclass

@dataclass
class ServingRequest:
    req_id: str
    prompt_tokens: int
    output_tokens: int
    ttft_slo_ms: float
    tpot_slo_ms: float

# Heterogeneous benchmark workload mixing long RAG contexts and conversational queries
benchmark_workload = [
    ServingRequest("rag_doc_1", prompt_tokens=32768, output_tokens=128, ttft_slo_ms=1500.0, tpot_slo_ms=40.0),
    ServingRequest("chat_turn_1", prompt_tokens=512, output_tokens=256, ttft_slo_ms=300.0, tpot_slo_ms=30.0),
    ServingRequest("code_gen_1", prompt_tokens=4096, output_tokens=512, ttft_slo_ms=600.0, tpot_slo_ms=35.0),
]

2. Deployment Configurations in vLLM

# Setup A: Aggregated continuous batching with chunked prefill
# Chunks long prompts into manageable token blocks (e.g., 2048) to avoid stalling decode steps
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --enable-chunked-prefill \
    --max-num-batched-tokens 2048 \
    --gpu-memory-utilization 0.90

# Setup B: Disaggregated serving topology
# Prefill-dedicated node
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --port 8100 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.92 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Decode-dedicated node (receives KV cache over high-speed interconnect)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --port 8200 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.92 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

3. Benchmarking Matrix: SLO Attainment & Latency

Illustrative figures, not measurements: the table shows the expected shape of the trade-offs, not results from a benchmark run.

Serving Architecture Mean TTFT (Prompt=32K) Mean TPOT (Decode) TPOT P99 Jitter Goodput (% Requests Meeting All SLOs)
Vanilla Continuous Batching 1,120 ms 48 ms ±34 ms (Severe preemption) 56% (Frequent TPOT violations)
Chunked Prefill (Chunk=2048) 1,320 ms 28 ms ±7 ms (Controlled) 82% (Balanced execution)
Static Disaggregation (Splitwise style) 1,650 ms 22 ms ±2 ms (Near-deterministic) 77% (Prefill queue bottleneck)
Unified Latency-Shifting (TaiChi) 1,220 ms 24 ms ±4 ms (Optimized) 94% (Optimal cross-request balance)

How Big Companies Solve This

Leading AI platforms and labs address KV cache scaling and latency contention across algorithmic, framework, and hardware layers.

Anthropic — prompt caching

Anthropic introduced prompt caching for Claude API workloads, allowing developers to cache frequently reused contexts (such as system instructions and few-shot examples) 2. By reusing precomputed KV states across API calls, prompt caching reduces prompt processing latency by up to 85% and cuts input token costs by up to 90%, drastically reducing the prefill load on serving infrastructure 2.

DeepSeek — Multi-Head Latent Attention

Rather than expanding physical GPU memory footprint to accommodate massive KV caches, DeepSeek (in DeepSeek-V2, V3, and R1) introduced Multi-Head Latent Attention 3, 5. MLA compresses keys and values into a low-rank latent representation during caching, which is projected back during attention computation. This reduces the per-token KV cache memory footprint by multiple folds compared to standard Multi-Head Attention, significantly alleviating memory bandwidth bottlenecks during autoregressive decoding 3, 5.

vLLM — PagedAttention and disaggregated prefill

vLLM introduced PagedAttention, an algorithm that partitions KV caches into non-contiguous memory blocks inspired by OS virtual memory paging 4. This eliminates external memory fragmentation and minimizes internal fragmentation (reducing memory waste from over 60% down to near-zero) and enables flexible cross-request cache sharing 4. Building on this, modern engines combine chunked prefill and disaggregated KV transfers over RDMA or NVLink to decouple prefill latency from decode responsiveness 1.

vLLM also ships disaggregated prefilling, still marked experimental. Its documentation is explicit about the trade-off: running prefill and decode in separate instances lets you tune TTFT and inter-token latency separately and controls tail inter-token latency more reliably than chunked prefill, but "Disaggregated prefill DOES NOT improve throughput" 10.

DistServe — disaggregation for goodput

DistServe (listed for OSDI 2024) assigns prefill and decode to different GPUs to remove the interference between them, then co-optimizes each phase's resources and parallelism against the application's TTFT and TPOT targets 6. Against the serving systems it was compared with, it served 7.4× more requests, or met a 12.6× tighter SLO, while keeping more than 90% of requests within their latency limits 6.

Splitwise and Sarathi-Serve — split the machines, or split the prompt

Two research systems take the two main routes. Splitwise splits the phases onto separate machines so each can be provisioned on its own hardware, moving the KV state over the cluster's fast interconnect; its authors report 1.4× higher throughput at 20% lower cost, or 2.35× more throughput for the same cost and power 7. Sarathi-Serve keeps both phases on the same GPUs but splits each prefill into roughly equal chunks and admits new requests without pausing ongoing decodes ("stall-free scheduling"); compared with vLLM under tail-latency limits, it reports 2.6× higher serving capacity for Mistral-7B on one A100 and up to 3.7× for Yi-34B on two A100s 8.

Moonshot AI — Mooncake for Kimi

Mooncake, the serving platform behind Moonshot AI's Kimi, is built around the KV cache: it separates prefill and decode clusters and pools idle CPU, DRAM and SSD on its GPU servers into a disaggregated KV cache store 9. Its authors report up to a 525% throughput increase in some simulated scenarios, and that under real workloads it lets Kimi handle 75% more requests 9.

Discussion

The architectural shift in LLM serving reveals that hardware capacity alone cannot resolve latency contention. Even the fastest accelerators face a structural dichotomy between compute-bound matrix multiplications and memory-bound autoregressive decoding.

While static Prefill-Decode disaggregation provides clean physical isolation, it imposes an infrastructure tax through underutilized GPU instances when request arrival patterns vary. As demonstrated by recent research in unified architectures like TaiChi, the next frontier in AI inference engineering is dynamic, SLO-aware scheduling 1. By treating prefill chunk size and cluster allocation as dynamic control sliders, serving systems can shift latency margins from requests that easily meet their deadlines toward requests at risk of SLO violations 1. When paired with architectural KV compression like MLA and memory management like PagedAttention, inference clusters can achieve both high token throughput and predictable, low-jitter latency 3, 4.

Which technique should you use?

The sources point to a simple order of operations:

If your problem is… Start with Why
Long, repeated prompt prefixes (system prompts, documents) Prompt caching Removes repeated prefill work entirely 2
Streams that stutter when long prompts arrive, on one GPU pool Chunked prefill Keeps decodes running while prefills are processed in chunks 8
Strict tail inter-token latency, and you can run separate pools Prefill-decode disaggregation Isolates the phases; tunes TTFT and ITL separately, but doesn't raise throughput 6, 10
Decode limited by memory bandwidth or KV cache size KV cache compression and paging (MLA, PagedAttention) Shrinks or packs the KV cache each decode step reads 3, 4

Conclusion

Resolving LLM serving latency interference requires a multi-layered engineering approach. At the model architecture level, low-rank attention mechanisms like MLA shrink the physical memory footprint of cached states 3. At the memory runtime layer, virtualized paging mechanisms like PagedAttention maximize GPU memory density 4. At the cluster scheduling layer, chunked prefill and hybrid prefill-decode disaggregation prevent compute-heavy prefill bursts from degrading interactive streaming latency 1. Together, these techniques transform LLM serving from a rigid, monolithic batching process into an elastic, SLO-driven distributed system.

Sources

  1. Wang et al. (2025). Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving. arXiv:2508.01989. https://arxiv.org/html/2508.01989v1
  2. Anthropic. (2024). Prompt caching with Claude. Anthropic Engineering & News. https://www.anthropic.com/news/prompt-caching
  3. Meng et al. (2025). TransMLA: Multi-head Latent Attention Is All You Need. arXiv:2502.07864. https://arxiv.org/html/2502.07864v1
  4. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of ACM SOSP '23 / arXiv:2309.06180. https://ar5iv.labs.arxiv.org/html/2309.06180
  5. DeepSeek-AI et al. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434. https://arxiv.org/abs/2405.04434
  6. Zhong et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024 / arXiv:2401.09670. https://arxiv.org/abs/2401.09670
  7. Patel et al. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677. https://arxiv.org/abs/2311.18677
  8. Agrawal et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arXiv:2403.02310. https://arxiv.org/abs/2403.02310
  9. Qin et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079. https://arxiv.org/abs/2407.00079
  10. vLLM. Disaggregated Prefilling (experimental). vLLM documentation. https://docs.vllm.ai/en/latest/features/disagg_prefill.html

Keep reading

All posts →

Be first in when doors open.

Get early access