Inference & Serving

Training a model is a one-off cost. Serving it is a cost you pay on every token, for every user, forever. This page covers how an LLM actually generates text on a GPU, why the two phases of generation have opposite bottlenecks, and the stack of techniques (KV caching, continuous batching, paging, prefix reuse, disaggregation, speculative decoding, quantization, parallelism) that modern engines use to turn a fixed GPU budget into as many tokens as possible within a latency target. Interviewers ask about this because it is where model knowledge meets systems engineering: a strong candidate can do the back-of-envelope arithmetic on a whiteboard and explain which knob moves which metric.

1. The autoregressive loop

A decoder-only transformer defines \(p(x_t \mid x_{<t})\). To generate, you run the network on the prompt, get a distribution over the vocabulary for the next position, pick a token (the sampling step), append it, and repeat until an end-of-sequence token or a length limit. In pseudocode:

tokens = tokenize(prompt)
kv = model.prefill(tokens)            # one big parallel pass over the prompt
logits = kv.last_logits
while not done:
    t = sample(logits, temperature, top_p, ...)   # pick next token
    stream_to_user(t)
    logits, kv = model.decode_step(t, kv)          # one token in, one distribution out

Two facts about this loop drive everything on this page.

1. The output is sequential. Token \(t+1\) cannot be computed until token \(t\) is chosen, because it is an input. Within a single sequence you cannot parallelize across output positions (speculative decoding is a clever way around this).
2. Each step only needs one new row of computation, provided you saved the intermediate state (keys and values) of all earlier tokens. That saved state is the KV cache. Without it each step would recompute attention over the entire prefix: \(O(n^2)\) work per step, \(O(n^3)\) per sequence in the attention part. With it, each step is \(O(n)\) for attention plus a constant amount of weight math.

So a request has two very different phases. The prefill (also called the prompt or context phase) processes all \(n\) prompt tokens at once, fills the KV cache and produces the first output token. The decode (generation) phase then runs one step per output token, each step processing a single new token per sequence against the cached state.

2. Prefill vs decode: the arithmetic-intensity argument

The key concept is arithmetic intensity: FLOPs performed per byte moved from memory (HBM). A GPU has a peak compute rate \(F\) (FLOP/s) and a peak memory bandwidth \(B\) (bytes/s). A kernel with intensity \(I\) runs at \(\min(F,\ I \cdot B)\). The crossover \(I^* = F/B\) is the "ridge point" of the roofline. Below it you are memory-bound and FLOPs sit idle. Above it you are compute-bound. Austin+ 2025 (Scaling Book)

For an H100 SXM, NVIDIA lists 3.35 TB/s of HBM3 bandwidth and ~989 dense BF16 TFLOPS (the spec sheet's 1,979 figure is with 2:4 sparsity). NVIDIA H100 So $$ I^*_{\text{H100, BF16}} \approx \frac{989\times 10^{12}}{3.35\times 10^{12}} \approx 295\ \text{FLOPs/byte}. $$ With FP8 compute the ridge doubles to roughly 590. Any kernel doing less than a few hundred FLOPs per byte loaded leaves the tensor cores waiting.

Why a linear layer's intensity ≈ the number of tokens in the batch

Take one linear layer \(Y = XW\) with \(W \in \mathbb{R}^{d \times d}\) in BF16 (2 bytes per value), and \(X\) holding \(T\) tokens (rows). FLOPs: \(2Td^2\). Bytes: the weights \(2d^2\), plus activations \(2Td + 2Td\). When \(T \ll d\) the weights dominate: $$ I \approx \frac{2Td^2}{2d^2 + 4Td} \approx T \quad (\text{for } T \ll d). $$ In plain terms: each weight byte you load gets used once per token in the batch. That gives the whole story:

• Prefill of a 2,000-token prompt has \(T = 2000\), well above 295, so it is compute-bound. Doubling prompt length roughly doubles prefill time (plus a quadratic attention term that matters at long context).
• Decode at batch size \(b\) has \(T = b\) (one new token per sequence). At \(b=1\), \(I \approx 1\): the GPU runs at about 1/300 of its peak FLOPs. Adding more sequences to the batch is almost free until \(b\) approaches the ridge point, because you're paying for the weight read anyway.

Attention during decode is worse: batching doesn't help it

The weight matmuls are shared across the batch, so batching amortizes them. Decode attention is different: each sequence reads its own KV cache (\(n\) cached tokens) to do about \(2 \cdot n \cdot d\) FLOPs per head group. The intensity stays around 1 FLOP/byte for MHA no matter how big the batch (GQA raises it by the group factor, since several query heads share one KV head; MLA raises it further). So at long contexts and large batches, decode time becomes dominated by streaming KV caches, not weights. This is why KV-cache size is a first-class serving concern.

3. Latency metrics, SLOs and the latency–throughput trade-off

Goodput was popularized by DistServe, which defines it as the maximum request rate served within both TTFT and TPOT constraints. Zhong+ 2024 The point is that raw throughput can be gamed: you can push tokens/s up with a giant batch while every user waits 200 ms per token. Always quote throughput at a latency target, ideally a tail percentile (P90/P99), since averages hide stalls.

The latency–throughput curve

Model one decode step as reading the weights \(W\) plus all active KV caches, at bandwidth \(B\), with compute \(2Pb\) FLOPs for \(P\) parameters and batch \(b\): $$ t_{\text{step}}(b) \approx \max\!\left(\frac{W + b \cdot \text{KV}_{\text{avg}}}{B},\ \frac{2Pb}{F}\right), \qquad \text{throughput} = \frac{b}{t_{\text{step}}},\qquad \text{TPOT} = t_{\text{step}}. $$ At small \(b\) the weight read dominates, so \(t_{\text{step}}\) barely moves while throughput grows linearly with \(b\): free throughput. As \(b\) grows, the KV term (and eventually compute) takes over, TPOT rises, and throughput flattens. Beyond that point you're only adding latency. The operator's job is to sit at the knee of this curve, at the largest batch that still meets the TPOT SLO. A memory ceiling also exists: KV capacity caps \(b \times\) context.

Little's law ties it to load: concurrent requests in the system \(= \lambda \times\) mean time in system. If you serve 20 requests/s each lasting 10 s end to end, you hold about 200 sequences in flight, which sets your KV memory need.

4. Back-of-envelope: how fast can decode be?

At batch 1 each decode step must read every weight once (dense model). So the ceiling is $$ \text{tokens/s}_{\,b=1} \;\lesssim\; \frac{\text{memory bandwidth}}{\text{bytes of weights}} . $$ Real systems hit roughly 60–85% of peak bandwidth, and also read the KV cache, so treat this as an upper bound.

Worked example on an H100 SXM (≈3.35 TB/s, 80 GB)

Note the TP rows: splitting weights across \(g\) GPUs multiplies aggregate bandwidth by \(g\), which is why tensor parallelism is the standard latency lever. It isn't perfectly linear because each layer adds two all-reduces whose latency doesn't shrink.

Adding batch: the aggregate throughput estimate

Llama-3-8B BF16 on one H100, batch 64, average context 2,048 tokens. KV per token for this model is 128 KiB (computed in the next section), so KV for the batch is \(64 \times 2048 \times 128\ \text{KiB} \approx 17\ \text{GB}\). Each step reads \(16 + 17 = 33\) GB, so \(t_\text{step} \approx 33/3350\ \text{s} \approx 10\) ms. Compute per step is \(2 \times 8\times10^9 \times 64 \approx 1\) TFLOP, about 1 ms at 989 TFLOPS. Still bandwidth-bound. So:

• TPOT ≈ 10 ms (100 tok/s per user), and aggregate ≈ 64/0.010 ≈ 6,400 output tok/s as a ceiling, perhaps 3,000–4,500 in practice.
• Compare batch 1 at ~200 tok/s: batching bought ~30× throughput for a 2× hit on per-user speed. That is the economics of serving in one line.
• Memory check: 16 GB weights + 17 GB KV + activations and CUDA graphs fit in 80 GB, so you could push the batch further until KV fills HBM or TPOT breaks the SLO.

Prefill estimate

Prefill FLOPs ≈ \(2P\) per token (plus attention). For 8B params, that's 16 GFLOP/token. At 989 TFLOPS and a realistic 40–60% MFU, that's roughly 25k–37k prompt tokens/s per GPU. A 4,000-token prompt takes on the order of 110–160 ms of prefill. For a 70B model on one GPU in FP8 (≈2× BF16 FLOPs), scale by 70/8 and divide by about 2: around 400–600 ms for the same prompt. Those are the TTFT floors before queueing.

5. The KV cache

What is cached and why

In each attention layer, every token produces a query \(q\), key \(k\) and value \(v\). The output for position \(t\) is \(\text{softmax}(q_t K_{\le t}^\top/\sqrt{d})V_{\le t}\). With causal masking, the keys and values of earlier tokens never change once computed, because they depend only on earlier tokens. So you compute them once, store them per layer, and at each decode step compute only the new token's \(q,k,v\), append \(k,v\) to the cache, and attend over the cache. Queries are not cached since each is used only at its own step.

Size formula

The 2 is for K and V. \(L\) is layers, \(n_{kv}\) is the number of KV heads (equal to query heads for MHA, fewer for GQA, 1 for MQA), \(d_{head}\) is head dimension.

Worked examples

The 70B row is the headline: at modest batch × context, KV memory exceeds weight memory, so KV capacity, not compute, decides how many users a GPU can hold. The MHA counterfactual shows why nearly every modern model uses GQA or something stronger.

How to shrink it

6. Batching: static, dynamic, continuous

Batching is the main way to raise arithmetic intensity in decode. The question is when sequences enter and leave the batch.

Continuous batching is now universal in serious engines (vLLM, SGLang, TensorRT-LLM's "in-flight batching", TGI). Anyscale measured large throughput gains over static batching on real workloads, on the order of up to ~20×+ in their benchmark, with most of it coming from not wasting slots and from fitting more sequences via better memory management. Anyscale 2023

Continuous batching exposed the next bottleneck: memory. If each sequence pre-reserves a contiguous KV region sized for the maximum length, you can only fit a handful of sequences and most of the reserved memory is empty. That's what PagedAttention solves.

Scheduling policies worth knowing

• FCFS with admission control: admit a waiting request only if enough free KV blocks exist for its prompt (plus a margin).
• Preemption: when the cache runs out mid-generation, evict a sequence and either swap its blocks to CPU or recompute its KV later (recompute is often cheaper because prefill is efficient). vLLM supports both.
• Prefill-priority vs decode-priority: running new prefills ASAP minimizes TTFT but stalls ongoing decodes (ITL spikes); deferring them protects ITL. Chunked prefill and disaggregation reduce this conflict.
• Cache-aware / priority scheduling: SGLang sorts waiting requests to maximize prefix-cache hits; production systems add priorities and fairness per tenant.

7. PagedAttention (vLLM)

The fragmentation problem

Before vLLM, engines allocated a contiguous KV buffer per sequence, sized to the maximum possible length, because the final length is unknown. That causes three kinds of waste: reservation (slots for tokens not yet generated), internal fragmentation (slots that will never be used because the sequence ends early), and external fragmentation (free gaps between buffers that are too small for a new request). The vLLM authors reported that existing systems wasted 60–80% of KV memory this way, while paging brings waste to under 4%. vLLM blog 2023

The mechanism: virtual memory for the KV cache

PagedAttention borrows the OS idea of paging. Kwon+ 2023

• GPU KV memory is carved into fixed-size physical blocks, each holding the K and V for a fixed number of tokens (16 is a common default) for all layers/heads (implementations differ in layout).
• Each sequence has a block table mapping its logical blocks (token positions 0–15, 16–31, …) to physical block IDs, which need not be contiguous.
• Blocks are allocated on demand as the sequence grows. Waste is limited to the unfilled tail of the last block.
• The attention kernel takes the block table as input and gathers K/V from scattered blocks. This indirection costs a little kernel efficiency, which is more than repaid by the larger batches it allows.

Sharing and copy-on-write

Because blocks are referenced through tables, sequences can share physical blocks. With parallel sampling (n completions of one prompt) the prompt's blocks are stored once with a reference count. With beam search beams share long common histories and fork often. When a sequence needs to write into a block whose reference count is above 1 (typically the partially filled last block), the engine performs copy-on-write: allocate a new block, copy the contents, point this sequence at the copy, decrement the original's count. The vLLM team reported memory savings of up to ~55% for these sampling methods. vLLM blog 2023

The paper's headline was 2–4× throughput over FasterTransformer and Orca at the same latency, mostly from fitting larger batches. Kwon+ 2023 Paged KV is now standard: TensorRT-LLM, SGLang, TGI and others all use block-based KV.

8. Prefix caching and RadixAttention

Many requests share a prefix: the same system prompt, few-shot examples, a long document asked several questions, previous turns in a multi-turn chat, an agent's growing transcript. If the KV for that prefix is still in memory, a new request can skip prefilling it entirely. For agentic and chat workloads this is often the largest single TTFT and cost saving.

• Automatic prefix caching (vLLM): each full KV block is identified by a hash of its tokens and the hash of the prefix before it, so identical prefixes map to the same block. Freed blocks stay in an LRU pool until reused or evicted. vLLM docs
• RadixAttention (SGLang): keeps a radix tree (compressed trie) over token sequences whose edges point at KV memory. A new request walks the tree to find its longest cached prefix; eviction is LRU over leaves; the scheduler orders requests to maximize hits. Zheng+ 2023 (SGLang)

Both approaches reuse at block or token granularity, and only for exact prefixes: any difference early in the prompt (say a timestamp at the top of the system prompt) invalidates everything after it. That's why API providers tell you to put static content first and variable content last, and why they price cached input tokens at a steep discount. DeepSeek reported that on one day in early 2025, 56.3% of input tokens hit their on-disk KV cache. DeepSeek 2025

Cache-aware routing

With many replicas, a prefix cache only helps if the request lands on the replica that holds it. So load balancers become KV-aware: route by longest-prefix match weighed against load. NVIDIA Dynamo and llm-d both implement this (llm-d reads vLLM's KV events to know which replica holds which prefix). NVIDIA Dynamo llm-d Hierarchical caches extend capacity by spilling KV to CPU DRAM, SSD or a distributed store (LMCache, Mooncake Store, Dynamo's KV block manager). LMCache

9. Chunked prefill

With continuous batching, a new request's prefill runs in the same iteration as everyone else's decode step. A 30k-token prompt makes that iteration take hundreds of milliseconds, so every other user sees a stall (an ITL spike). Running prefill and decode in separate iterations has a similar problem: the decodes just wait.

Chunked prefill splits the prompt into chunks (e.g. 512–2,048 tokens) and processes one chunk per iteration alongside the batch's decode tokens, under a per-iteration token budget. SARATHI introduced the idea of "piggybacking" decodes onto prefill chunks: the prefill chunk makes the iteration compute-bound, and the decodes ride along nearly free because their weight reads are already happening. Agrawal+ 2023 Sarathi-Serve made it a "stall-free" scheduler with a token budget derived from the TPOT SLO. Agrawal+ 2024 vLLM V1 and SGLang use chunked prefill by default.

• Pros: bounded iteration time, so ITL stays smooth; higher utilization from mixing compute-bound and memory-bound work.
• Cons: TTFT for the long prompt goes up somewhat (it's spread over several iterations, and each chunk re-reads the KV of earlier chunks); smaller chunks lose some prefill efficiency.

10. Prefill–decode disaggregation

Chunking shares a GPU between phases. Disaggregation goes the other way: run prefill on one pool of GPUs, decode on another, and transfer the KV cache in between.

Why split?

• No interference: decode ITL never stalls behind a long prefill, and prefill TTFT doesn't wait for decode batches.
• Independent tuning: different parallelism, batch sizes, even GPU types per phase. Splitwise argued that prefill wants compute-heavy GPUs while decode can use cheaper, bandwidth-rich or older ones. Patel+ 2023 (Splitwise)
• Independent scaling: the prefill-to-decode GPU ratio follows the workload's input/output length mix. DistServe optimizes this placement for goodput and reported serving 7.4× more requests (or 12.6× tighter SLOs) than colocated baselines in its experiments. Zhong+ 2024

The cost: moving the KV cache

For a 70B GQA model a 4k-token prompt has about 4096 × 320 KiB ≈ 1.3 GB of KV. Over a 400 Gb/s (50 GB/s) RDMA NIC that's ~26 ms; over NVLink much less. That's acceptable next to a prefill of hundreds of ms, especially if transfer is pipelined layer by layer as prefill proceeds. For MLA models the KV is far smaller, which makes disaggregation even more attractive. Over slower networks, or for short prompts, the transfer overhead can outweigh the gains, so colocated with chunked prefill remains a good default for small deployments.

Production examples

• Mooncake (Moonshot AI's Kimi): a KV-cache-centric architecture with a disaggregated KV pool using the cluster's spare CPU DRAM and SSD, plus prediction-based early rejection under overload. Reported handling 75% more requests on real workloads. Qin+ 2024 (Mooncake)
• DeepSeek-V3/R1 serving: prefill and decode on separate deployments, with different expert-parallel degrees (EP32 for prefill, EP144 for decode) and data-parallel attention. DeepSeek 2025
• Open source: SGLang and vLLM both support PD disaggregation; NVIDIA Dynamo and llm-d orchestrate it across engines, with NIXL as a transfer library. LMSYS 2025

11. Speculative decoding

Decode is memory-bound, so a forward pass over \(k+1\) tokens costs about the same as a pass over one token: the weights are read once either way. Speculative decoding exploits this. A cheap drafter guesses the next \(k\) tokens; the target model scores all of them in a single forward pass; you keep the longest prefix the target agrees with, plus one extra token from the target. If guesses are often right, you emit several tokens per expensive pass. Introduced concurrently by Leviathan et al. and Chen et al. Leviathan+ 2022 Chen+ 2023

Why it's lossless: the rejection-sampling rule

Let \(q(x)\) be the drafter's distribution and \(p(x)\) the target's, at some position. The drafter sampled \(x \sim q\). Accept it with probability $$ \Pr[\text{accept } x] = \min\!\left(1, \frac{p(x)}{q(x)}\right). $$ If rejected, sample a replacement from the normalized residual $$ p'(x) = \frac{\max(0,\ p(x) - q(x))}{\sum_{x'} \max(0,\ p(x') - q(x'))}. $$ The probability of ending up with token \(x\) is \(q(x)\min(1, p(x)/q(x)) + \Pr[\text{reject}]\,p'(x) = \min(q(x), p(x)) + \max(0, p(x) - q(x)) = p(x)\). So each emitted token is distributed exactly as the target would have produced. It is not an approximation, and the quality is identical. Under greedy decoding this reduces to "accept while the draft token equals the target's argmax."

Acceptance-rate math

The per-token acceptance probability is \(\alpha = \sum_x \min(p(x), q(x)) = 1 - \text{TV}(p, q)\): one minus the total-variation distance between drafter and target. Assuming each token is accepted independently with probability \(\alpha\), and \(k\) tokens are drafted, the expected number of tokens emitted per target pass is $$ \mathbb{E}[\text{tokens/pass}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}. $$ With \(\alpha = 0.8\), \(k = 4\): \((1 - 0.8^5)/0.2 \approx 3.4\) tokens per pass. With \(\alpha = 0.6\): ≈ 2.3. If one draft step costs a fraction \(c\) of a target step, the wall-clock speedup is roughly \(\frac{1-\alpha^{k+1}}{(1-\alpha)(ck + 1)}\). Leviathan+ 2022 The diminishing returns in \(k\) are visible: at \(\alpha = 0.8\), going from \(k=4\) to \(k=8\) only raises the expectation from 3.4 to 4.3 while doubling draft cost.

(Expected tokens per target pass, ignoring draft cost.) Acceptance is high on predictable text such as code, boilerplate, JSON, and quoting the input; it's lower on creative, high-temperature sampling.

When it helps and when it hurts

• Helps most at small batch / latency-sensitive serving, where the GPU has spare FLOPs. Verifying \(k+1\) tokens is nearly free in compute.
• Helps less, or hurts, at large batch: the batch already makes decode closer to compute-bound, so verifying \(k\) extra tokens per sequence costs real FLOPs, and rejected tokens are wasted work. Engines often adapt \(k\) to load or switch speculation off at high batch.
• It also costs memory (the drafter's weights and KV) and engineering complexity (rolling back KV, tree attention masks).

Variants

Related training-side work: Gloeckle et al. showed multi-token prediction as a pretraining objective can both improve quality and enable self-speculation. Gloeckle+ 2024 In production engines, vLLM's current docs list EAGLE (including EAGLE-3), MTP, draft models, n-gram and suffix decoding among supported methods. vLLM docs

12. Quantization for inference

Quantization stores numbers in fewer bits. For inference it buys three things: less memory (fit a bigger model, or more KV, on a GPU), less bandwidth per token (faster memory-bound decode), and, if the matmul itself runs in low precision, more FLOPs (faster compute-bound prefill). Which of these you get depends on what you quantize.

Why weight-only quantization helps decode so much

At small batch the decode step time is about weight bytes ÷ bandwidth. Going BF16 → INT4 cuts weight bytes by about 3.5–4× (scales add overhead), so the ceiling rises by a similar factor even though the multiply still happens in BF16 after on-the-fly dequantization. The dequant costs ALU work, which is idle anyway in a memory-bound kernel. As batch grows and decode becomes compute-bound, the advantage shrinks and dequant overhead can make W4A16 slower than FP8 W8A8. Rule of thumb: weight-only INT4 for latency at low concurrency (local, edge, small-batch); FP8 (or FP4) W+A for high-throughput serving.

The main methods

• Round-to-nearest (RTN) per channel or per group (e.g. groups of 128 weights share a scale) is the baseline. At 8 bits it's nearly lossless; at 4 bits it degrades noticeably on some models, and at 3 bits badly.
• GPTQ quantizes one layer at a time, column by column, using second-order information: it minimizes the layer's output error \(\|WX - \hat W X\|^2\) on calibration data, and after quantizing each column it updates the not-yet-quantized columns to compensate for the error, using the inverse Hessian \(H = 2XX^\top\). It quantizes 175B-class models in a few GPU hours at 3–4 bits. Frantar+ 2022
• AWQ observes that a small fraction of weight channels (~1%) matter most, identified by large activation magnitudes. Rather than keep them in high precision (bad for kernels), it scales those channels up before quantization (and folds the inverse scale into the previous op), which reduces their relative rounding error. No backprop or Hessian needed, and it generalizes well. Lin+ 2023
• LLM.int8() found that LLM activations have a few outlier feature dimensions with huge magnitude, which wreck naive INT8 activation quantization. It handled them with mixed precision (outlier columns in FP16). Dettmers+ 2022
• SmoothQuant makes W8A8 practical by moving the difficulty from activations to weights with a per-channel scale \(s\): \(Y = (X\,\mathrm{diag}(s)^{-1})(\mathrm{diag}(s)\,W)\), with \(s_j = \max|X_j|^{\alpha} / \max|W_j|^{1-\alpha}\) and \(\alpha \approx 0.5\). The transform is mathematically exact and can be folded into the previous LayerNorm, so it's free at runtime. Xiao+ 2022
• FP8: E4M3 (more precision, used for weights/activations in inference) and E5M2 (more range). Micikevicius+ 2022 Hopper and later run FP8 matmuls at 2× BF16 throughput. FP8 W8A8 with per-channel or block-wise scales is close to lossless for most large models and has become the default production precision for many deployments; DeepSeek-V3 was even trained in FP8.
• FP4: MXFP4 and NVFP4. FP4 (E2M1) has only 16 values, so everything depends on scaling. MXFP4 (the OCP Microscaling spec) uses blocks of 32 elements sharing a power-of-two (E8M0) scale. Rouhani+ 2023 NVIDIA's NVFP4 uses blocks of 16 with an FP8 (E4M3) scale, plus a per-tensor FP32 scale; the finer, non-power-of-two scale fits values better. NVIDIA reports ≤1% accuracy drop going from FP8 to NVFP4 on DeepSeek-R1 benchmarks. NVIDIA 2025
• System co-design like QServe's W4A8KV4 shows the right mix depends on kernels as much as accuracy: INT4 weights with INT8 activations can beat both pure W4A16 and W8A8 at serving batch sizes. Lin+ 2024 (QServe)

Quality trade-offs

13. Parallelism for inference

Training parallelism (see the distributed training page) splits the work to fit and to speed up a giant batch. Inference parallelism is driven by two different goals: fit the model and its KV cache, and hit a latency target.

14. Serving Mixture-of-Experts models

An MoE layer replaces one FFN with \(E\) expert FFNs and a router that sends each token to the top-\(k\). DeepSeek-V3 has 671B total parameters with 37B active per token (256 routed experts, 8 active, plus a shared expert). DeepSeek-V3 2024 This creates a split personality at inference time:

• Memory scales with total params: ~671 GB at FP8, more than one 8×H100 node (640 GB), so you need H200/B200-class nodes or multiple nodes.
• Compute per token scales with active params (37B), which makes prefill cheap relative to a dense model of similar quality.
• Decode bandwidth depends on batch. At batch 1 you read only the ~37B active params. But across a batch, different tokens pick different experts. The expected fraction of experts touched per layer is \(1 - (1 - k/E)^b\). With \(k/E = 8/256\) and \(b = 32\), that's \(1 - 0.969^{32} \approx 64\%\); at \(b = 128\), ~98%. So at serving batch sizes you read nearly all expert weights each step, but each expert only sees about \(bk/E\) tokens: at \(b=128\), just 4 tokens per expert. That's terrible arithmetic intensity.

The fix is to make the global batch enormous and spread experts thin: wide expert parallelism. DeepSeek's decode deployment used EP across 18 nodes (144 GPUs), so each GPU holds only a couple of experts and receives tokens from the whole cluster, reporting roughly 14.8k output tokens/s per H800 node for decode and 73.7k input tokens/s per node for prefill (including cache hits). DeepSeek 2025 SGLang reproduced this style on 96 H100s with PD disaggregation and reported about 22k decode tokens/s per node. LMSYS 2025

MoE-specific serving problems

• All-to-all communication: each MoE layer dispatches tokens to expert owners and gathers results. Libraries like DeepEP provide a high-throughput mode for prefill and a low-latency mode (CUDA-graph compatible) for decode. DeepEP
• Overlap: split the batch into two micro-batches so one computes while the other communicates ("two-batch overlap"); SGLang reported 27–35% throughput gains from it. LMSYS 2025
• Load imbalance: popular experts become stragglers. An expert-parallel load balancer (EPLB) replicates hot experts and re-places them based on observed traffic.
• Capacity and dropping: training sometimes uses capacity factors that drop overflow tokens; inference engines generally avoid dropping, so worst-case expert load matters for memory planning.
• Small deployments: on one node, MoE gives you fast batch-1 decode (few active params) but poor batched efficiency. MoE economics shine at large scale with lots of traffic.

15. Sampling

The model outputs logits \(z\) over the vocabulary. Sampling turns them into one token. The common pipeline applies penalties, then temperature, then truncation, then samples. Order varies by engine, and it matters.

Why beam search is rare for chat

• Quality: maximizing likelihood produces bland, generic and repetitive open-ended text. Holtzman et al. showed human text is not the highest-probability text. Holtzman+ 2019 Beam search still makes sense for constrained tasks such as translation or ASR.
• Cost: \(B\) beams multiply decode compute and KV (paging with copy-on-write softens the memory part), and beams fork and die unpredictably, which complicates batching.
• Streaming: you don't know which beam wins until the end, so you can't stream tokens to the user.
• Modern alternatives for "better answers" use sampling: best-of-N with a reward model, self-consistency, or reasoning models' long chains of thought.

16. Structured and constrained decoding

Applications often need output that parses: JSON matching a schema, a regex, SQL, a function call. Prompting alone fails some fraction of the time. Constrained decoding guarantees the format by masking the logits at each step so only tokens that keep the output a valid prefix of the grammar can be sampled.

FSM approach (regex, simple JSON schemas)

Outlines compiles a regex (JSON schemas are converted to regexes) into a finite-state machine over characters, then precomputes an index mapping each FSM state to the set of vocabulary tokens that are valid from that state (a token may span several characters and FSM transitions). At runtime each step is a lookup plus a mask: \(O(1)\) amortized per token after the one-time compile. Willard & Louf 2023

CFG approach (nested JSON, code, arbitrary grammars)

Recursive structures (arbitrarily nested JSON) need a context-free grammar and a pushdown automaton, whose state includes a stack, so you can't precompute masks for every state. XGrammar splits the vocabulary into context-independent tokens (validity determined by the current grammar position alone, precomputed and cached) and context-dependent tokens (checked at runtime against the stack), uses a persistent stack for fast branching and rollback, and overlaps mask computation on CPU with the GPU's forward pass. It reported up to ~100× speedups over prior approaches and near-zero overhead in end-to-end serving. Dong+ 2024 (XGrammar) It's the default structured-output backend in several engines.

Practical gotchas

• Token boundaries: grammar is over characters, the model emits multi-character tokens. Masks must account for tokens that complete one symbol and start another. Forcing unnatural tokenizations can hurt quality.
• Quality under constraints: forcing a format the model isn't used to can degrade content. Put a reasoning field before the answer field, use field names the model understands, and keep schemas simple.
• Jump-forward: when the grammar allows only one continuation (e.g. a fixed key name), the engine can append those tokens directly without sampling them one at a time (SGLang does this). LMSYS 2024
• Interaction with speculative decoding: draft tokens must also be checked against the grammar; engines handle this unevenly.

17. Long-context serving

Long contexts (128k to 1M+ tokens) stress every part of the stack.

• Prefill becomes quadratic. Per layer, attention costs about \(4n^2d\) FLOPs (scores plus weighted sum, before causal halving), against about \(2Pn\) for the linear layers over the whole model. For a 70B model (\(d = 8192\), \(L = 80\)) the two are equal around \(n \approx 2P/(4dL) \approx\) 50k tokens (roughly 100k with causal masking). Past that, attention dominates and TTFT grows quadratically. FlashAttention keeps it IO-efficient but doesn't change the FLOP count. Dao+ 2022
• KV capacity: one 128k-token Llama-3-70B sequence needs ~42 GB of BF16 KV. A single user can consume most of a GPU, so batch sizes collapse and cost per token rises. Providers often price long context higher for this reason.
• Decode reads the whole KV per step, so TPOT grows linearly with context. At small batch and long context, standard attention kernels underuse the GPU because they parallelize over batch and heads only. Flash-Decoding also splits the KV sequence across thread blocks and combines partial softmaxes, recovering parallelism. Dao+ 2023
• Prefill parallelism: context/sequence parallelism (ring attention) spreads a single long prefill over many GPUs to bound TTFT. Liu+ 2023
• Prefix caching matters even more: re-asking questions about the same 200k-token document should never re-prefill it. Tiered KV storage (GPU → CPU → SSD → remote) makes this feasible.
• Architecture-level fixes: GQA/MLA, sliding-window or local–global layers, attention sinks for streaming Xiao+ 2023, and trained sparse attention.

18. Serving engines

Others you may hear about: LMDeploy, DeepSpeed-FastGen (introduced "Dynamic SplitFuse", a chunked-prefill relative) Holmes+ 2024, Ollama (desktop wrapper building on llama.cpp-style runtimes), MLX (Apple silicon) MLX, and managed inference platforms that wrap these engines.

19. Hardware notes

The pattern to remember: decode speed per user tracks memory bandwidth per byte of model; decode throughput tracks HBM capacity (KV space) and bandwidth; prefill tracks FLOPs. SRAM-based designs (Groq, Cerebras) win on single-stream latency because on-chip bandwidth is orders of magnitude above HBM, but they pay in chip count and capacity. AMD's MI300-series GPUs compete with large HBM capacity (192 GB+) and are supported by vLLM and SGLang.

20. Cost math

Worked example 1: 8B model on one H100

Assume an H100 rents for $2.50/hour (cloud prices vary widely by provider and commitment; verify current rates).

• Output tokens: from section 4, ~3,500 output tok/s realistic at batch 64. That's 12.6M tokens/hour → $2.50 / 12.6 ≈ $0.20 per 1M output tokens.
• Input tokens: prefill ~30k tok/s → 108M tokens/hour → ≈ $0.023 per 1M input tokens, about 9× cheaper. This asymmetry is why API providers price output tokens several times higher than input, and cached input lower still.

Worked example 2: 70B FP8 on an 8×H100 node

• Node at $20/hour. Run 4 replicas with TP=2 (aggregate 6.7 TB/s per replica).
• Per replica at batch 64, 2k average context, FP8 KV (160 KiB/token): KV = 64 × 2048 × 160 KiB ≈ 21 GB. Step bytes ≈ 70 + 21 = 91 GB → \(t_{\text{step}}\) ≈ 13.6 ms → ceiling ≈ 4,700 tok/s. Compute check: \(2 \times 70\text{B} \times 64 \approx 9\) TFLOP per step across ~4 PF of FP8: ~2–3 ms, so still bandwidth-bound.
• Assume ~55% of ceiling: ~2,500 tok/s per replica, 10,000 tok/s per node → 36M tokens/hour → ≈ $0.56 per 1M output tokens at full load.
• Real traffic is bursty. At 40% average utilization the effective cost is ≈ $1.40/M. Utilization is often the biggest cost lever, bigger than kernel tweaks.

For calibration, DeepSeek published that its V3/R1 service averaged ~227 H800 nodes over one day, costing about $87k/day at an assumed $2/GPU-hour, and claimed a theoretical cost-profit margin of 545% at R1 list prices (actual revenue was much lower). DeepSeek 2025

21. Edge and on-device inference

The same roofline applies with much smaller numbers. A phone's memory bandwidth is roughly 50–100 GB/s; Apple's high-end desktop chips reach ~800 GB/s. A 3B model at 4-bit (~1.7 GB) on a phone has a decode ceiling around 30–60 tok/s, and an 8B 4-bit model maybe half that. Batch is almost always 1, so decode is purely bandwidth-bound, which is why aggressive weight-only quantization (4-bit and below) is standard on device.

• Constraints: RAM shared with the OS and apps, thermal throttling during long generations, battery, and app size. Prefill on long prompts is often the slowest part on device, since mobile NPUs and GPUs have modest FLOPs.
• Runtimes: llama.cpp (GGUF), MLX on Apple silicon, MLC LLM (including WebGPU in the browser), Google's LiteRT LiteRT, PyTorch ExecuTorch, and OS-level APIs such as Apple's Foundation Models framework for its on-device model Apple.
• Patterns: small distilled models (1–4B) for on-device tasks, hybrid routing (device first, cloud for hard queries), LoRA adapters swapped per task on a shared base, speculative decoding with a tiny on-device drafter.

Interview question bank