Ten Whys: Why Do These Core Techniques Appear in LLM Serving?
Many people use large language models today, but deploying an LLM as a stable, efficient, and cost-effective service is far more complex than simply running it on a GPU.
A real LLM serving system must handle unpredictable requests, variable prompt lengths, unknown output lengths, limited GPU memory, and token-by-token decoding. Because of these challenges, modern LLM infrastructure has developed many techniques that may look complicated at first, but are actually driven by very intuitive system bottlenecks.
In this blog, I explain LLM inference systems through “ten whys.” Each “why” corresponds to a real bottleneck, and each technique exists to address that bottleneck.
1. Why does LLM inference need to be divided into prefill and decode?
LLM inference naturally consists of two stages: prefill and decode.
In the prefill stage, the model processes the entire input prompt at once and generates the corresponding keys and values for the attention layers. This stage usually involves many tokens, large matrix computations, and high GPU parallelism, so it is more compute-bound.
In the decode stage, the model generates the response one token at a time. Each step has relatively small computation, but it must repeatedly access the historical KV cache. Therefore, decode is more sensitive to memory bandwidth and scheduling overhead.
The key insight is that one LLM request is not uniform. Prefill is like large-scale parallel computation, while decode is like many small iterative steps. This difference is the basis for understanding KV Cache, Continuous Batching, Chunked Prefill, and PD disaggregation.
2. Why do we need KV Cache?
During decoding, if there is no KV Cache, the model would need to recompute attention over all previous tokens every time it generates a new token. This creates a large amount of redundant computation.
KV Cache solves this problem by storing the keys and values of historical tokens. During decode, the model only computes the K/V for the newly generated token and reuses the previous K/V from the cache.
Therefore, KV Cache essentially trades GPU memory for computation. It greatly reduces redundant computation in decoding, but it also creates a new challenge: KV Cache grows with context length and batch size, consuming a large amount of GPU memory.
As a result, the bottleneck of LLM serving begins to shift from pure computation to KV Cache storage and management.
3. Why do we need PagedAttention?
KV Cache is useful, but managing it efficiently is difficult because request lengths and output lengths are highly variable.
If the system allocates a large continuous GPU memory region for every request, much memory may be wasted. If it allocates too little, the request may run out of space. This makes KV Cache memory management a serious system problem.
PagedAttention, introduced in vLLM, borrows the idea of paging from operating systems. Instead of storing each request’s KV Cache in a continuous memory region, it splits the cache into fixed-size blocks and uses a mapping table to connect logical token positions with physical KV blocks.
The key insight is that PagedAttention does not change the attention computation itself. It solves the problem of how to efficiently manage dynamically growing KV Cache in LLM serving.
4. Why do we need Continuous Batching?
Traditional batching processes a group of requests together and waits until all requests in the batch finish before starting the next batch.
This works for ordinary deep learning inference, but it is inefficient for LLM generation because different requests generate different numbers of tokens. Short requests may finish early, while long requests continue running. If the batch is static, GPU slots are wasted and new requests cannot enter.
Continuous Batching solves this by scheduling at the granularity of each decode iteration rather than the whole request. After every token generation step, finished requests are removed, their KV Cache and batch slots are released, and new requests are added.
The key insight is similar to restaurant seating: do not wait for the whole table to finish. When one seat becomes free, let a new customer sit down.
5. Why do we need Chunked Prefill?
Continuous Batching improves decode scheduling, but long prompts still create another problem: prefill can be very heavy.
For example, if a request contains tens of thousands of tokens, processing the whole prompt at once may occupy the GPU for a long time. This can delay ongoing decode requests and make token generation less smooth, worsening TPOT and ITL.
Chunked Prefill solves this by splitting a long prompt into smaller chunks. The serving engine can then interleave prefill chunks with decode requests, balancing throughput and latency.
The key insight is that Chunked Prefill is not just about cutting prompts into smaller pieces. It turns a large prefill task into multiple schedulable units, giving the system finer-grained control.
6. Why do we need FlashAttention?
Attention looks like a matrix multiplication problem, but on real GPUs, the bottleneck is often not only computation. Data movement also matters.
Standard attention may require large attention matrices and frequent reads and writes to HBM, especially for long contexts. Since HBM is much slower than on-chip SRAM, excessive data movement can significantly reduce performance.
FlashAttention is based on an IO-aware insight. It uses tiling to keep attention computation in on-chip SRAM as much as possible, reducing repeated HBM access while still computing exact attention.
The key lesson is that optimizing LLM inference is not only about reducing FLOPs. It also requires understanding the hardware memory hierarchy. Sometimes the real bottleneck is not computing too much, but moving too much data.
7. Why do we need Prefix Caching / RadixAttention?
In real applications, many requests share common prefixes.
For example, chatbots often reuse the same system prompt, RAG applications may reuse the same documents, and agent workflows may repeatedly include the same tool descriptions or templates. If every request recomputes these prefixes from scratch, many prefill computations are wasted.
Prefix Caching reuses the KV Cache of shared prefixes. If a new request has the same prefix as a previous request, the system can directly reuse the already computed KV Cache.
SGLang’s RadixAttention further organizes shared prefixes using a radix-tree-like structure, allowing the system to automatically find and reuse KV Cache across different generation calls.
The key insight is that LLM requests should not always be treated as independent. Many real workloads contain repeated structures, and reusing them can reduce TTFT and save GPU resources.
8. Why do we need Speculative Decoding?
Decode is slow because it is autoregressive. The model must generate the first token before generating the second, and the second before the third. This token-by-token dependency limits parallelism.
Speculative Decoding tries to break this limitation. A smaller draft model first predicts several candidate tokens, and the larger model verifies them in parallel.
If the candidates are accepted, the system effectively generates multiple tokens in one large-model forward pass. If they are rejected, the system falls back to the large model’s output.
The key insight is that verifying multiple tokens with a large model may cost roughly the same as generating one token. Therefore, speculative decoding partially turns serial generation into parallel verification.
This technique is especially useful for latency-sensitive scenarios, small batch sizes, and workloads where decode dominates the overall cost.
9. Why do we need Quantization?
LLM inference is expensive mainly because model weights and KV Cache are large.
Large weights consume GPU memory and memory bandwidth during decoding. Large KV Cache limits batch size and context length, especially in long-context serving.
Quantization reduces this cost by representing high-precision data such as FP16 or BF16 with lower-precision formats such as INT8, INT4, or even fewer bits. This can reduce memory usage, lower bandwidth pressure, and improve throughput.
The challenge is to keep model quality stable. Weight-only quantization compresses model weights, activation quantization must handle outliers, and KV Cache quantization directly affects long-context decoding.
The key insight is that quantization is a trade-off: lower precision is used to improve deployment efficiency while trying to preserve model quality.
10. Why do we need PD Disaggregation?
PD disaggregation means Prefill-Decode Disaggregation. It separates prefill and decode into different serving stages.
This is necessary because prefill and decode are fundamentally different workloads. Prefill is compute-bound, highly parallel, and suitable for large computations. Decode is more memory-bound, iterative, and sensitive to per-token latency.
If both stages run on the same GPUs, long prefill tasks may block decode, while decode iterations may interfere with prefill computation. It also becomes difficult to optimize TTFT and TPOT separately.
PD disaggregation addresses this by using one group of GPUs for prefill and another group for decode. After prefill finishes, the generated KV Cache is transferred to the decode nodes.
The key insight is that prefill and decode should be treated as two different services. The prefill cluster can be optimized for throughput and long prompts, while the decode cluster can be optimized for low-latency token generation.
Of course, PD disaggregation is not free. It requires KV Cache transfer, more complex scheduling, network bandwidth, cache placement, and load balancing. Therefore, it is not just a single-node optimization, but a cluster-level architecture optimization for LLM serving.
Summary
This blog explains ten key techniques in modern LLM inference engines through the perspective of “why.” Each technique appears because of a real system bottleneck, and each one reflects an important insight in LLM serving.
In future posts, I will continue exploring the key technologies behind AI infrastructure and the system insights behind them.