This is the second post in our series. Last time, we looked at the “parallelismsalad” of DP, TP, PP, and EP. Today, I want to address the elephant in the room:
Why does serving an LLM feel like systems hell, when “classical” ML felt almost boring?
If you’ve ever deployed a vision model or a recommendation engine, you know the drill: preprocess, one-shot inference, postprocess. It’s predictable. LLMs, however, break every rule of traditional ML serving.
The “Boring” World of Classical ML
Typical “old-school” ML inference flows like this:
- Preprocess: Resize an image, normalize it, tokenize a quick string, or turn a user/item pair into features.
- Forward Pass: One quick run through the model. Output shows up. That’s it.
- Postprocess: Apply softmax, NMS, decode boxes, set thresholds, rank items, or toss in some business logic.
The shocking part: For plenty of real-world models, the model itself isn’t even the slow part. Preprocessing and postprocessing can eat up as much time (or more) as the inference step.
Plus, you get two big perks without trying:
- It’s largely stateless. Each request stands alone. The server doesn’t have to hold onto much from one call to the next. Scaling out horizontally is no big deal.
- Batching is straightforward. Static batching works cleanly: Grab N requests, stack them, do one forward pass, hand back N results.
Want more throughput? Bump up the batch size. Need lower latency? Dial it down. It’s all reliable. This is the mindset most folks carry into LLM serving and it’s precisely what makes LLM serving hit like a gut punch.
Why LLM Serving Stands Apart
LLMs skip the “one forward pass” routine. Instead, they chat back and forth with the GPU, token by token. LLMs are autoregressive, they don’t give you the answer all at once. They generate one token, feed it back into themselves, and repeat. It’s more like: Run the model, grab the next token, tack it onto the input, run again — and loop, maybe for thousands of steps. A single user request isn’t one “task” for your server, it’s a long-lived, stateful job that repeatedly hijacks GPU cycles for seconds or even minutes.
Classical inference is a shot. LLM inference is a loop.
KV Cache: The Optimization That Changes the Problem
In a Transformer, every new token needs to “look back” at all previous tokens. If you recomputed everything at every step, the math would become quadratic and slow to a crawl. To fix this, we use the KV Cache. In prefill, calculate Key and Value tensors for the prompt just once. In decode, add only the new Key and Value for each token and pull from the prior ones. It shifts from “recompute the past every go” to “reuse the past, add one token.”
But this “optimization” creates a massive headache: State.
Your server now has to manage a massive, growing blob of memory for every single active user. Inference has stopped being a math problem and has become a memory allocation problem.
KV Cache Makes Inference a Memory Juggling Act.
KV Cache Turns Serving Into Memory Management
KV cache isn’t some side note. It creates a growing memory demand that expands with: Context length, concurrent sequences, model scale (layers, heads, hidden dims), and generation duration. Suddenly, LLM serving morphs from “GPU crunching” into “simulating a GPU memory manager.”
This leads to two major production killers:
- Fragmentation: Requests vary in prompt and output lengths, leading to uneven KV chunks. Allocate, release, repeat and soon you’ve got “VRAM confetti”: Total memory free, but scattered in unusable fragments.
- Over-Allocation: Systems often book KV space ahead to dodge mid-run reshuffles. That curbs fragmentation but charges you for peak KV even on lighter loads.
What it means in plain words: You handle fewer simultaneous requests than your raw VRAM implies.
This explains why serving frameworks elevate KV cache to core status, not just a tweak.
PagedAttention and Block Based KV Management
One of the key ideas that made practical LLM serving possible is paged or block based KV cache management, popularized by vLLM’s PagedAttention. The idea is similar to virtual memory in operating systems. Instead of requiring one large contiguous chunk of memory for each request’s KV cache, the cache is split into fixed-size blocks or pages. Those blocks can live in different physical locations in VRAM while still appearing logically contiguous to the runtime.
This greatly reduces fragmentation and improves memory utilization, which in turn increases the number of concurrent requests a system can handle. Most modern high performance LLM serving engines use some form of paged or block based KV cache management.
PagedAttention (or similar paged/block KV setups) is a foundational idea that enabled practical modern serving.
Why Batching Becomes Hard
In the old world, you’d wait 5ms, grab 16 requests, run them, and send them back (server side batching). In LLMs, that doesn’t work. If you batch 16 users together, User A might finish their sentence in 5 tokens, while User B wants 500. If you use static batching, the whole batch is held hostage by the longest request. Your GPUs sit idle waiting for that one user to finish.
The solution is Continuous Batching. As soon as User A finishes, their slot is immediately freed for User C to jump in, even while the other 15 users are still mid-generation. It turns your server into a high-speed revolving door. That improves utilization dramatically, but it also introduces another major piece of infrastructure: the scheduler. At that point, the serving stack is no longer just executing model kernels. It is continuously deciding which requests enter, which continue decoding, and how GPU resources should be shared.
Two Different Workloads
Unlike classical ML (one forward pass), LLM inference is actually two different systems problems:
- Prefill: Processing the prompt. It’s compute heavy and loves big batches.
- Decode: Generating tokens. It’s bandwidth heavy and happens one by one.
This is why we track two different latencies: TTFT (Time to First Token) for the prefill, and TPOT (Time Per Output Token) for the decode. If you try to optimize for one, you often break the other (as I mentioned in the previous post).
LLM inference isn’t uniform. It’s dual.
Why Specialized Serving Frameworks Exist
This is also why you cannot simply wrap a PyTorch model in Flask and call it a serving system. LLM serving requires efficient state management, KV cache handling, scheduling, and dynamic batching. That is why specialized frameworks emerged:
- vLLM: The gold standard for high throughput open-source serving. It’s focused on the PagedAttention and continuous batching logic.
- TensorRT-LLM: NVIDIA’s powerhouse. It’s more complex to set up, but it’s designed to squeeze every possible drop of performance out of the kernels themselves.
- SGLang: A newer contender that focuses on fast scheduling and complex structured prompts (like tool calling).
These are only a few examples, but they illustrate the point: LLM serving needs infrastructure that classical ML serving often does not.
so, in short: Classical ML serving is mostly stateless, one-off, batch simple, and often ruled by non model bits. While, LLM serving is state heavy, persistent, looped, and all about taming memory and scheduling while tokens flow.
Bottom line
If classical ML serving is a smooth drive, LLM serving is bumping through the dark: Requests linger, work loops, KV cache shifts targets, batching must flow continuously, and success rides on the scheduler.
If you treat LLM serving like a traditional model, your GPUs will stay idle and your VRAM will stay full. Once you accept that serving is now a memory and scheduling problem, the engineering choices like, PagedAttention and continuous batching, start to make perfect sense.
