A lot of LLM serving work looks like this:
You lower max_model_len and suddenly concurrency improves more than expected.
You raise batch size and throughput goes up, but p95 latency gets worse.
You move to a faster GPU and decode barely improves.
You turn on prefix caching and see a big win in one workload and almost nothing in another.
You try speculative decoding and realize the headline gain depends on whether the model actually accepts the guesses.
At first, this feels like a collection of unrelated tricks.
It isn’t.
Most serving behavior comes from a small set of budgets:
- memory capacity
- memory bandwidth
- compute throughput
- communication overhead
- queueing delay
- reuse rate
That is the “LLM math” I care about. Not elegant transformer theory. The small number of equations and scaling rules that explain why serving systems behave the way they do, and why engines like vLLM, SGLang, and TensorRT-LLM expose the knobs they expose.
If you understand those budgets, a lot of inference tuning stops being guesswork.
The first mistake is treating all tokens the same
Serving systems talk in tokens, but tokens are not interchangeable.
A prompt token being ingested during prefill is not the same kind of work as a generated token during decode. A cached prefix token is not the same as a brand-new prompt token. A token in a long request does not cost the system the same way a token in a short request does.
That sounds obvious, but a lot of bad tuning decisions come from forgetting it.
The rest of this post is really just a way of making that idea concrete.
KV Cache Usually Sets Your Concurrency Budget
The most common serving bottleneck people underestimate is KV cache.
A useful first-order estimate is:
KV bytes ≈ live_tokens × 2 × L × H_kv × D_head × bytes_per_elem
Where:
- live_tokens is the total prompt + generated tokens currently resident across active requests
- L is the number of layers
H_kvis the number of key/value heads- D_head is the head dimension
The above formula is an estimate for dense transformer models with standard KV caching. It applies most naturally to MHA/GQA/MQA style attention. Newer architectures such as MLA, along with some hybrid and MoE designs, can deviate from it because they change what state is stored per token and how that state scales at inference time.
It explains why one long request can crowd out many short ones. It explains why setting max_model_len much higher than your real working point can quietly waste concurrency. It explains why GQA, MQA, and lower precision KV cache are not just model details, they change serving capacity directly by changing bytes per live token.
This is one of the places where people lose time. They keep looking for a smarter scheduler before asking whether they are simply paying too much memory per active token.
Prefill and Decode Want Different Hardware
People say “LLM inference” as if it describes a single bottleneck. In practice, prefill and decode want different things.
A good first order approximation:
throughput_prefill ~ available_FLOPS / work_per_prompt_token
throughput_decode ~ memory_bandwidth / KV_bytes_touched_per_token
The main point is not that prefill is always compute bound and decode is always bandwidth-bound. That is too simplistic.
Prefill is often more compute friendly because many prompt tokens can be processed together. But if input sequence length is short, or batch size is small, or the kernels are underutilized, then prefill can leave a lot of FLOPS on the table. In that regime, saying “prefill is compute-heavy” is technically true but operationally misleading.
Decode is usually where bandwidth pressure becomes much more obvious, because every step keeps reaching back into existing KV state. But even there, the dominant bottleneck can move depending on model size, context length, scheduler behavior, kernel quality, and parallelism strategy.
So the more honest version is:
- prefill is typically more compute efficient
- decode is typically more bandwidth sensitive
That distinction matters because TTFT and TPOT often want different optimizations. It is also the reason chunked prefill, speculative decoding, and prefill/decode disaggregation are all solving real problems rather than being engine specific gimmicks.
Batching Is Really Token Budget Math
A lot of serving advice says “increase batch size for more throughput.” That is true right up until it stops being true.
What you are actually managing is a token budget:
total_active_tokens ≈ prefill_tokens_in_flight + decode_tokens_in_flight
Those tokens compete for KV capacity, scheduler space, latency headroom, compute, and bandwidth. This is why increasing batch can improve one metric while damaging two others.
You may get better utilization. But you may also:
- increase KV pressure
- make long prompts more disruptive
- slow down shorter requests
- reduce how much context length you can safely support
- push TTFT or TPOT in the wrong direction
The important subtlety is that prefill and decode tokens are not operationally identical even if the scheduler reasons about both under one token budget. They stress the system differently.
This is also why chunked prefill is such a practical idea. It is not just a clever scheduler trick. It is a way of preventing one very large prompt from consuming too much of the token budget in one shot.
Where Good Throughput Numbers Start Lying
This is the part teams usually learn the hard way.
A system can look great on average and still feel bad to users.
The decomposition is simple:
latency = service_time + waiting_time
And once utilization gets high, waiting_time can become the whole story. Especially when request sizes vary a lot.
That variance is everywhere in LLM workloads:
- short chat turns
- giant prompts
- short generations
- long generations
- requests with reusable prefixes
- requests that are entirely fresh
When that mix gets ugly, tail latency gets ugly. One large prefill can delay a lot of lighter requests behind it. This is why long prompts tend to hurt p95 TTFT much more than they hurt average throughput.
This is also why “more utilization” is not automatically a win. Sometimes it is just a nice looking benchmark number that comes with a worse product.
High utilization is only part of the story. Tail behavior also depends on service time variance, scheduler policy, and how much large requests can delay small ones.
Once you see serving this way, a lot of scheduler behavior starts to look much less arbitrary:
- chunking heavy prefills
- capping admitted tokens
- treating large jobs more carefully
- separating phases
- intentionally leaving some headroom
Prefix Caching Is Only as Good as Your Actual Reuse
Prefix caching sounds like a universal optimization until you look at the workload.
Get Or Zipori’s stories in your inbox
Join Medium for free to get updates from this writer.Subscribe
Remember me for faster sign in
A useful way to think about it is:
effective_prefill_tokens = total_prompt_tokens - reusable_prefix_tokens
If many requests share long prefixes, effective prefill cost drops a lot. If they do not, prefix caching helps much less than people hope.
This is why the feature shines in some workloads:
- shared system prompts
- repeated chat templates
- stable tool scaffolding
- repeated RAG wrappers with a changing tail
And it is why it disappoints in others.
There is also a more practical point here: prefix reuse is not semantic reuse. Two prompts that mean almost the same thing do not help unless the engine can actually match and preserve the shared prefix structurally.
Granularity matters too. Smaller blocks or pages can improve reuse opportunities. Larger ones may be better for backend efficiency. That trade off is easy to miss if you only think of prefix caching as an on/off feature.
The bigger lesson is that some serving wins come from application shape, not only engine tuning. If your application can preserve long stable prefixes, the engine gets to cash that in.
Speculative Decoding Is an Acceptance Rate Business
The promise of speculative decoding is straightforward: get more progress per expensive target model step.
The catch is also straightforward.
net_gain > 0
only if
accepted_extra_tokens × target_step_cost > draft_overhead + verification_overhead
That is the basic trade.
If the proposed tokens are accepted often enough, you win. If they are not, the extra machinery starts eating the gain.
This is why speculative decoding results vary so much. The outcome depends on:
- acceptance rate
- how cheap the draft path is
- how expensive the target path is
- verification overhead
- workload shape
This is also where MTP belongs.
MTP is in the same family because it tries to improve useful multi token progress per expensive decode step, but it does so through training rather than only through a separate draft model pipeline. In the good cases, that can improve the alignment between what gets proposed and what the main model is willing to accept.
There is a very practical reason people care about this beyond benchmark tokens/sec. If a decode iteration can advance by more than one accepted token, TPOT can drop, E2E latency can improve, and the user can see higher effective tokens/sec.
Still, this is not a free multiplier. Sometimes the gain appears more clearly in throughput than in E2E latency, because verification and orchestration still cost real time.
Parallelism Is Usually Solving a Fit Problem Before It Solves a Speed Problem
When people ask about TP versus PP, the conversation often starts in the wrong place.
They ask: which one is faster?
The better question is: what problem are we trying to solve?
- model fit?
- communication cost?
- interconnect limits?
- latency sensitivity?
- multi-node deployment constraints?
Tensor parallelism reduces per device weight load but adds collective communication. Pipeline parallelism reduces per device layer load but introduces stage transfer and bubble costs.
A simple PP intuition is:
pipeline_efficiency drops when microbatch_count is too small relative to pipeline_depth
So PP is never just “free extra GPUs.” It only works well if there is enough work in flight to keep the stages busy.
This gives a practical rule of thumb:
- if the model fits on one GPU, avoid sharding.
- if it fits within one fast node, TP is often the natural first move.
- if you need to span nodes, PP becomes more interesting because cross node TP communication can get expensive fast.
TP can absolutely be the right answer, especially inside a strong NVLink box. But it is also easy to talk about TP as if it only changes fit. It changes latency too, because communication is now in the step path.
This also helps answer a common question around long prompts.
Very large prompts do not automatically imply PP. They first imply a prefill heavy workload. PP becomes relevant when the worker doing that prefill needs model sharding or when the interconnect makes other strategies unattractive.
That distinction matters. Otherwise teams reach for parallelism when the real issue is workload separation.
Disaggregation Is What You Do When Prefill and Decode Stop Being Good Roommates
This is one of the cleanest modern serving ideas because the logic is simple.
benefit_of_isolation > KV_transfer_cost + orchestration_overhead
If prefill and decode interfere with each other badly enough, separating them helps. If they do not, disaggregation is just extra plumbing.
The pattern where disaggregation makes the most sense is usually the same one people run into in practice: long prompts, moderate outputs, and a strong mismatch between what TTFT wants and what TPOT wants.
That is when a single shared fleet starts behaving like a compromise neither phase actually likes.
This is also where the earlier long prompt question lands.
If prompts are huge, the first question is not “should I use PP?”
The first question is “should prefill be isolated from decode?”
Only after that do you ask how to shard the prefill worker itself.
That is how you end up with an architecture like:
- prefill pool: compute oriented, maybe TP or PP or both
- decode pool: bandwidth and KV oriented
- KV handoff between them
That design is not over engineered by definition. It is justified when the interference you remove is more expensive than the transfer and coordination overhead you add.
That last clause matters. KV handoff, cache layout conversion and the network path are real costs. Disaggregation is attractive when those costs are smaller than the tail latency and efficiency damage of mixing the phases.
Some of the Best Wins Come From Shrinking Bytes Per Token
A lot of serving work eventually collapses into one question:
How many bytes am I paying per active token?
A useful intuition is:
capacity_gain ~ old_bytes / new_bytes
Not exact, but directionally powerful.
If KV moves from FP16 to FP8, token capacity can increase a lot if the backend and quality trade offs are acceptable. If the model uses GQA or MQA, the KV footprint drops structurally. If part of the cache can move to host memory or a tiered cache, effective capacity can increase further, though usually with latency trade offs.
This is why I do not like treating model architecture, precision, and serving as separate conversations. They are all touching the same budget.
This is one of the easiest places to waste time in the wrong order. Teams sometimes reach for clever scheduler changes when the highest ROI move is just reducing bytes per live token.
That said, lower KV precision is not free. You still have to care about quality impact, kernel availability and backend maturity.
Finale
LLM serving is often presented as a collection of tricks and knobs. I think that framing misses the important part. Most of the behavior we care about – throughput, latency, concurrency and cost – comes from a small set of budgets: KV cache capacity, compute throughput, memory bandwidth, queueing, reuse, communication and bytes per token. Once you see those budgets clearly, a lot of serving decisions stop looking arbitrary.
That does not make tuning easy, but it does make it less superstitious. You can look at a workload and ask better questions before you start benchmarking: are you KV bound or queue bound, is decode limited by bandwidth, is prefix reuse actually real, is PP solving the problem or is prefill/decode interference the real issue?
That is the “LLM math” that matters to me. Not because the equations are elegant, but because they let you reason before you guess.
Originally published by Or Zipori in his Medium Blog
