Transformer & LLM Architecture

This page covers how a modern large language model is built: how text becomes tokens and vectors, what attention actually computes (shapes included), why the KV cache dominates serving cost and how MQA, GQA and MLA shrink it, how position is encoded and context extended, what is inside a decoder block, how to count parameters and FLOPs on a whiteboard, how Mixture-of-Experts works, and where hybrids with state-space and linear-attention layers stand in 2026. Interviewers use this material to tell apart people who have called an LLM from people who can reason about its memory, compute and failure modes. That reasoning drives real decisions about context length, serving hardware, fine-tuning and cost.

1. Why transformers won

Before 2017, sequence modelling meant RNNs/LSTMs, sometimes with attention bolted on, and some convolutional models (ByteNet, ConvS2S). The Transformer kept the attention, dropped the recurrence, and stacked self-attention with position-wise feed-forward layers. Vaswani+ 2017 It won on three axes that matter more as scale grows.

The complexity rows follow Table 1 of the original paper. Vaswani+ 2017 The deeper reasons:

• Parallel training. With teacher forcing, a causal transformer computes the loss for all \(n\) positions of a sequence in one forward pass, as a handful of large matrix multiplications. GPUs and TPUs are matmul machines, so you get far more useful FLOPs per dollar than with an LSTM's step-by-step loop.
• Gradient paths. Token 1 can affect token 4,000 through one attention hop. An RNN needs 4,000 multiplicative steps, which is where vanishing gradients come from. Residual connections plus attention give short, well-conditioned paths.
• Scaling behaviour. Transformer loss falls as a smooth power law in parameters, data and compute over many orders of magnitude. LSTMs plateau earlier, especially on later tokens in the context. Kaplan+ 2020 That predictability made it rational to spend very large budgets.
• Generality. Attention carries no built-in assumption of locality (unlike convolutions) or of order (positions are injected separately). The same block works on text, image patches, audio frames and actions.

The cost is that attention is quadratic in sequence length, and at inference you must keep K and V for every past token. Most of the architecture work since 2019 (GQA, MLA, sliding windows, FlashAttention, SSM hybrids) is about paying that cost more cheaply.

2. Tokenization

A model works on integer IDs from a fixed vocabulary \(V\). The tokenizer maps text to IDs and back. It is trained separately, before the model, and then frozen. Changing it later means re-learning the embeddings.

2.1 BPE: the workhorse

Byte-Pair Encoding was adapted from compression to NMT subwords. Sennrich+ 2016 Training it works like this:

1. Start with a base alphabet (characters, or the 256 byte values).
2. Count every adjacent pair of symbols in the training corpus.
3. Merge the most frequent pair into a new symbol and record the merge rule.
4. Repeat until the vocabulary reaches the target size (for example 32K, 128K or 200K).

Encoding new text replays the merge rules in priority order. Frequent words become single tokens. Rare words split into pieces ("tokenization" → ["token", "ization"]). Nothing is ever out-of-vocabulary, because you can always fall back to the base symbols.

2.2 Byte-level BPE

GPT-2 ran BPE over UTF-8 bytes rather than Unicode characters, so the base vocabulary is exactly 256 symbols and any string (emoji, code, binary junk) can be encoded with no <unk>. Radford+ 2019 It also applies a regex pre-tokenizer that first splits text into chunks (words, numbers, punctuation, whitespace runs), so merges never cross those boundaries and you don't get tokens like "dog.". OpenAI's tiktoken (cl100k, o200k vocabularies) and Llama 3's 128K tokenizer follow this design. tiktoken Llama 3 2024

2.3 SentencePiece and Unigram

SentencePiece is a library, not an algorithm. It treats input as a raw character stream with whitespace encoded as a visible symbol (▁), so it needs no language-specific pre-tokenizer. That matters for Chinese and Japanese, which have no spaces. It can train either BPE or Unigram models. Kudo & Richardson 2018

Unigram LM runs BPE in reverse. It starts from a large candidate vocabulary and iteratively prunes the tokens whose removal hurts corpus likelihood least, under a model where each token has an independent probability. To encode, it finds the most probable segmentation with Viterbi. Because it defines a distribution over segmentations, you can sample different splits during training (subword regularization), which improves robustness. Kudo 2018 T5 and many earlier multilingual models used SentencePiece-Unigram. Llama 1/2 used SentencePiece-BPE with byte fallback, and Llama 3 moved to a tiktoken-style byte-level BPE.

2.4 Vocabulary size trade-offs

Vocabulary sizes have grown: 32K (Llama 2, Mistral 7B) → 128K (Llama 3) → about 150K (Qwen) → about 200K (o200k, gpt-oss) → 256K–262K (Gemma). OpenAI 2025 Gemma 3 2025 The trade-off:

2.5 Tokenization failure modes (interview favourites)

• Character-level tasks. "How many r's in strawberry?" The model sees perhaps ["str", "awberry"] and has to have memorized the spelling of each token. Reversal, anagrams and rhyme run into the same issue.
• Arithmetic. Numbers get split inconsistently ("12345" might be ["123", "45"]), which misaligns place value. Llama 3 splits numbers into at most 3-digit chunks, and some tokenizers (Gemma's, for example) split every digit. Gemma 3 2025
• Glitch / under-trained tokens. Tokens that existed in the tokenizer's training corpus but almost never in the model's (Reddit usernames, for example) have near-random embeddings and can trigger bizarre outputs. There are automated ways to detect them. Land & Bartolo 2024
• Whitespace and leading-space sensitivity. "hello" and " hello" are different tokens. Trailing spaces in prompts, odd indentation in code, and prompt templates that leave a dangling space can all hurt quality.
• Token tax. Non-Latin scripts can need several times more tokens for the same content, so users pay more and get less effective context. The multiplier depends on the tokenizer and language. often 2–4×
• Prompt boundary / token healing. If a prompt ends mid-word, the most natural continuation might need a token that straddles the boundary. Constrained decoding and JSON-mode implementations have to handle partial tokens.
• Security. Special tokens such as <|im_start|> must never be produced from user text. If they are, users can forge chat-role boundaries.

3. Embeddings and the output head

The embedding matrix \(E \in \mathbb{R}^{V \times d}\) is a lookup table: token ID \(i\) becomes row \(E_i\), a \(d\)-dimensional vector (\(d\) = 4,096 for Llama-3-8B, 7,168 for DeepSeek-V3). A batch of token IDs of shape [B, n] becomes a tensor [B, n, d], the residual stream, which flows through every layer.

At the top, the final hidden state \(h \in \mathbb{R}^d\) is projected by the unembedding (LM head) \(W_U \in \mathbb{R}^{d \times V}\) to logits, then softmaxed to a next-token distribution:

• Weight tying (\(W_U = E^\top\)) saves \(V d\) parameters. Small models usually tie (Gemma, Llama 3.2 1B/3B). Large models usually don't, because the saving is negligible and untied heads perform slightly better.
• Embeddings are learned, not pre-computed like word2vec. Early layers turn each token embedding into a context-dependent representation, so the vector for "bank" at layer 20 differs between "river bank" and "bank loan".
• Some models scale embeddings by \(\sqrt{d}\) at input (original Transformer, Gemma) so their magnitude matches the rest of the residual stream.
• LLM embeddings vs embedding models: a "sentence embedding" for RAG usually comes from a separately trained encoder (or decoder) with a pooling step and contrastive training. It is not the input embedding table. Raw input embeddings know nothing about context.

4. Scaled dot-product attention, step by step

Notation: batch \(B\), sequence length \(n\), model width \(d\), heads \(h\), head dimension \(d_h = d/h\) (usually 64–128). Take one head and one sequence (drop \(B\)), with input \(X \in \mathbb{R}^{n \times d}\).

4.1 The computation

1. Project: \(Q = XW_Q,\; K = XW_K,\; V = XW_V\), with \(W_\ast \in \mathbb{R}^{d \times d_h}\). Shapes: \(Q, K, V \in \mathbb{R}^{n \times d_h}\).
   Q is "what am I looking for", K is "what do I contain (for matching)", V is "what I hand over if selected".
2. Scores: \(S = QK^\top / \sqrt{d_h} \in \mathbb{R}^{n \times n}\). Entry \(S_{ij}\) is how much query position \(i\) matches key position \(j\).
3. Mask: \(S \leftarrow S + M\), where \(M_{ij} = -\infty\) if \(j > i\) (future) and 0 otherwise. Padding positions are masked the same way.
4. Normalize: \(A = \mathrm{softmax}_{\text{row}}(S)\). Each row is a probability distribution over positions \(\le i\).
5. Mix: \(O = AV \in \mathbb{R}^{n \times d_h}\). Each output row is a weighted average of value vectors.

4.2 Why divide by √d_h?

Suppose the components of \(q\) and \(k\) are independent with mean 0 and variance 1. Then \(q \cdot k = \sum_{i=1}^{d_h} q_i k_i\) is a sum of \(d_h\) terms, each with variance 1, so

With \(d_h = 128\) the raw logits have a standard deviation of about 11. A softmax over logits that spread out is nearly one-hot, so its gradient (\(\partial a_i / \partial s_j = a_i(\delta_{ij} - a_j)\)) is close to zero almost everywhere, and learning stalls. Dividing by \(\sqrt{d_h}\) brings the logits back to unit variance, keeping the softmax in a responsive range at initialization. Vaswani+ 2017 Later, the model can learn to sharpen attention by growing \(W_Q, W_K\). Some recent models add QK-norm (RMSNorm on q and k) to keep logits bounded throughout training, a stability fix used in Gemma 3, Qwen3 and others. Gemma 3 2025 Qwen3 2025

4.3 Causal masking

A decoder predicts token \(t+1\) from tokens \(\le t\). During training we feed the whole sequence at once and get \(n\) predictions in parallel. Without a mask, position \(i\) could look at token \(i+1\) and copy the answer. Setting future logits to \(-\infty\) before the softmax gives them exactly zero weight. The same mask makes KV caching valid: past keys and values never change when new tokens arrive, so at decode time you compute Q, K and V only for the new token and attend over the cached K and V.

4.4 Prefill vs decode

This asymmetry explains why KV-cache size, batching and quantization dominate serving economics (covered on the inference page).

5. Multi-head attention and complexity

Rather than one attention of width \(d\), use \(h\) heads of width \(d_h = d/h\) in parallel, concatenate their outputs and mix them with \(W_O \in \mathbb{R}^{d \times d}\):

The parameter count equals one big head (\(4d^2\) for Q, K, V and O), but each head can learn a different attention pattern. One softmax can only produce one weighted average per position, while \(h\) heads produce \(h\) different averages. Interpretability work has found heads specialized for previous-token lookups, induction (copy what followed the last occurrence of the current token), syntax and more. Elhage+ 2021

5.1 Complexity

The \(n^2\) attention term overtakes the \(nd^2\) projection and FFN terms when \(4n^2 d \gtrsim 24 n d^2\), that is when \(n \gtrsim 6d\). For \(d\) = 4,096 that is roughly 25K tokens. Below that, the matmuls dominate and "attention is quadratic" matters less in practice than people assume. Above it, as at 128K–1M context, attention dominates prefill. Decode is different: per new token, attention costs \(O(nd)\) compute but must read the whole cache, which is a memory-bandwidth problem.

6. MQA, GQA, MLA and the KV cache

At decode time, every layer must keep K and V for all previous tokens. The memory for one sequence is:

Here 2 counts K and V, \(L\) is the number of layers, \(n_{kv}\) the number of K/V heads, \(d_h\) the head dimension, \(n_{\text{ctx}}\) the number of tokens, and \(b\) the bytes per element (2 for BF16, 1 for FP8). Every decode step reads the whole cache. So a smaller cache means larger batches, longer contexts, and more tokens/s per GPU.

6.1 The three sharing schemes

• MHA (multi-head): every query head has its own K and V head, so \(n_{kv} = h\).
• MQA (multi-query): all query heads share a single K/V head, so \(n_{kv}=1\). The cache shrinks by a factor of \(h\). Decoding got much faster, with some quality loss and training instability. Shazeer 2019
• GQA (grouped-query): query heads are split into \(g\) groups, and each group shares one K/V head, so \(n_{kv} = g\). Quality is close to MHA with a speed close to MQA. You can even "uptrain" an MHA checkpoint into GQA by mean-pooling K/V heads and training for about 5% of the original compute. Ainslie+ 2023 GQA with 8 KV heads is the default in Llama 2-70B and Llama 3, Mistral, Qwen, Gemma and gpt-oss.

6.2 Worked memory example

Take Llama-3-70B: \(L = 80\), \(h = 64\) query heads, \(n_{kv} = 8\), \(d_h = 128\), BF16. Llama 3 2024

Model weights in BF16 are about 140 GB. On 2×80 GB GPUs only about 20 GB is left after weights (less after activations and fragmentation). That is enough for one 64K-token sequence or eight 8K-token sequences. This is why the KV cache, rather than weights, often caps concurrency, and why paged attention, prefix caching, KV quantization and cache offload matter so much in production.

For a smaller model, Llama-3-8B (\(L=32, n_{kv}=8, d_h=128\)) needs \(2\cdot32\cdot8\cdot128\cdot2 = 128\) KiB per token, so 1 GiB per 8K-token sequence and 16 GiB at 128K. That is more than the 16 GB of BF16 weights.

6.3 MLA: Multi-head Latent Attention (DeepSeek)

MLA takes a different route. Instead of sharing heads, it compresses K and V into a low-rank latent. DeepSeek-V2 2024

1. Down-project each token's hidden state to a small latent: \(c^{KV}_t = W^{DKV} h_t\), where \(d_c = 512\) is much smaller than \(h \cdot d_h\) (\(128 \times 128 = 16{,}384\) in DeepSeek-V2/V3).
2. Per-head keys and values are up-projections of that latent: \(k_t^{(i)} = W^{UK}_{(i)} c^{KV}_t\), \(v_t^{(i)} = W^{UV}_{(i)} c^{KV}_t\). Only \(c^{KV}_t\) is cached.
3. Absorption trick: \(q^\top k = q^\top W^{UK} c = (W^{UK\top} q)^\top c\). At inference you fold \(W^{UK}\) into the query projection and \(W^{UV}\) into the output projection, then attend directly against the latents without ever rebuilding full K/V.
4. Decoupled RoPE: RoPE's position-dependent rotation would sit between \(W^{UK}\) and \(q\) and break the absorption. MLA therefore adds a separate small key of dimension \(d_R = 64\), shared across heads, that carries RoPE, and caches it alongside the latent.

The cache per token per layer is \(d_c + d_R = 576\) elements, against \(2 \cdot 128 \cdot 128 = 32{,}768\) for full MHA, roughly 57× smaller. DeepSeek reports a 93.3% KV-cache reduction versus their dense 67B model, with quality equal to or better than MHA. DeepSeek-V2 2024 For DeepSeek-V3 (61 layers): \(576 \times 61 \times 2\ \text{B} \approx\) 69 KiB per token, about 8.6 GiB at 128K. A 70B GQA model needs about 40 GiB, and DeepSeek-V3 is a model with 671B total parameters. DeepSeek-V3 2024

7. Positional encodings

Self-attention is permutation-equivariant: shuffle the input tokens and the outputs shuffle the same way. Without position information, "dog bites man" and "man bites dog" look identical. (A causal mask leaks some order implicitly, which is why NoPE, no positional encoding at all, works surprisingly well in decoders.)

7.1 Sinusoidal and learned absolute

The original Transformer added a fixed vector to each input embedding: Vaswani+ 2017

Each pair of dimensions is a sinusoid at a different frequency, from fast (local position) to very slow (coarse position), like the hands of a clock. GPT-2/3 and BERT instead learned a \(n_{\max} \times d\) table. Both are absolute and generalize poorly past the trained length, because there is simply no row for position 2,049.

7.2 RoPE: rotary position embedding

RoPE is the default in essentially every modern open LLM. Su+ 2021 It does not add anything to embeddings. It rotates q and k inside each attention layer. Split the \(d_h\) dimensions into \(d_h/2\) pairs. For pair \(i\), define the frequency \(\theta_i = b^{-2i/d_h}\), with base \(b\) = 10,000 originally. At position \(m\), rotate the pair by angle \(m\theta_i\):

The key property: rotating q by \(m\theta\) and k by \(n\theta\) gives a dot product that depends only on the angle between them, \((m-n)\theta\). In complex notation, \(\langle q e^{im\theta}, k e^{in\theta}\rangle = \mathrm{Re}[q \bar{k} e^{i(m-n)\theta}]\). So attention scores become a function of relative position, while the implementation stays a cheap element-wise operation applied to absolute positions. Nothing is added to the residual stream, and values are not rotated.

• High-frequency pairs (small \(i\)) rotate quickly. They distinguish neighbouring tokens and carry fine local order.
• Low-frequency pairs (large \(i\)) rotate slowly. The wavelength \(2\pi/\theta_i\) can exceed the training context, so those dimensions act almost as position-free semantic channels and coarse long-range position.
• Base \(b\): raising it stretches all wavelengths. Llama 3 uses 500,000 and Gemma 3's global layers use 1,000,000, which helps long-context training. Llama 3 2024 Gemma 3 2025 Increasing the base and then continuing pretraining on long documents ("ABF") is a standard context-extension recipe. Xiong+ 2023

7.3 ALiBi

ALiBi uses no embeddings at all. It adds a fixed, head-specific linear penalty to attention logits, \(S_{ij} \mathrel{+}= -m_h \cdot (i - j)\), with slopes \(m_h\) in a geometric sequence across heads. Distant tokens are penalized linearly, and some heads look far while others look locally. It extrapolated well past training length in the paper's experiments. Press+ 2021 BLOOM and MPT used it, but the field converged on RoPE plus scaling methods, partly because ALiBi's built-in recency bias can hurt retrieval of distant facts.

7.4 Context extension: PI, NTK-aware, YaRN

Suppose a RoPE model trained at length \(L\) is fed positions \(m > L\). Its low-frequency dimensions then see rotation angles never encountered in training, and quality collapses. The fixes all remap angles back into the trained range, given scale factor \(s = L'/L\):

In practice, long context is mostly earned through continued pretraining on long data (with a scaled base or YaRN), not through the scaling trick alone. A model can "accept" 128K tokens and still use the middle poorly. Retrieval accuracy tends to dip for information placed mid-context. Liu+ 2023

8. The decoder block

Each block does two things:

Attention is the only place where information moves between positions. The FFN works on each token independently and holds most of the parameters (about two-thirds in a dense model).

8.1 Pre-LN vs Post-LN

Variants exist: Gemma 2/3 apply RMSNorm both before and after each sublayer ("sandwich"), and some recent models (OLMo 2 reportedly among them) move the norm to the sublayer output inside the residual branch. Both target training stability at scale. QK-norm (Section 4.2) is part of the same stability toolkit.

8.2 RMSNorm

LayerNorm subtracts the mean, divides by the standard deviation, then applies a learned scale and bias. RMSNorm drops the mean-centering and the bias: Zhang & Sennrich 2019

It is cheaper (one reduction instead of two, no bias) and empirically just as good: the re-scaling invariance seems to be what matters, not the re-centering. The paper reports speedups that vary with model and hardware. (roughly 7–64% in their tests) Nearly every modern LLM uses it.

8.3 The FFN and SwiGLU

The classic FFN is \(W_2\,\sigma(W_1 x)\) with \(W_1 \in \mathbb{R}^{d\times 4d}\) and ReLU or GELU. Modern LLMs use a gated linear unit: Shazeer 2020

One projection produces the "content," the other a data-dependent "gate," and they are multiplied element-wise. Because there are three matrices instead of two, the hidden width is cut to about \(\tfrac{8}{3}d\) to keep parameters equal to a \(4d\) FFN. In practice models round it: Llama-3-8B uses \(d_{ff}=14{,}336 = 3.5d\). Gated FFNs consistently gave lower perplexity in Shazeer's comparison; the paper itself (checked against the PDF) says: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." GeGLU (GELU gate) is the same idea and is used by Gemma.

Interpretability work frames FFN layers as key-value memories: rows of \(W_1\) detect input patterns and the matching columns of \(W_2\) write associated content into the residual stream. Geva+ 2021 This is one reason factual knowledge is often localized to mid-layer FFNs.

8.4 The residual stream view

Rather than "layers transform representations," think of a \(d\)-dimensional shared bus. Embeddings write the initial state. Every attention head and every FFN reads from it (through its input projection) and adds a delta back. The unembedding reads the final state. Because everything is additive, the output decomposes into a sum of contributions from each component, which is the basis of circuit-style interpretability and the "logit lens" (unembedding intermediate layers to see predictions form). Elhage+ 2021

• It explains why layers can be skipped or pruned with modest damage, and why early exit and speculative self-drafting are possible.
• It explains why LoRA adapters and steering vectors work: small additive writes to a shared space.
• Heads in different layers compose by reading what earlier heads wrote. Induction heads, for example, combine a previous-token head with a matching head.

9. Counting parameters and FLOPs

9.1 The 12·L·d² rule

For a classic block with MHA and a 4× FFN:

• Attention: \(W_Q, W_K, W_V, W_O\), each \(d \times d\), giving \(4d^2\).
• FFN: \(d \times 4d\) plus \(4d \times d\), giving \(8d^2\).
• Norms and biases: \(O(d)\), negligible.

Sanity check, GPT-3 175B: \(L=96\), \(d=12{,}288\). Then \(12 \times 96 \times 12288^2 \approx 1.74\times10^{11}\), which is 174B. ✓

Modern variant, Llama-3-8B (\(L=32\), \(d=4096\), GQA with 8 KV heads of 128, SwiGLU with \(d_{ff}=14336\), \(V=128{,}256\), untied):

• Attention: \(W_Q, W_O\) are \(d^2\) each. \(W_K, W_V\) are \(d \times 1024 = 0.25d^2\) each. Total \(2.5d^2\).
• SwiGLU: \(3 \times d \times 3.5d = 10.5d^2\).
• Per layer \(13d^2 \approx 218\text{M}\). Times 32 layers gives about 6.98B.
• Embeddings plus head: \(2 \times 128256 \times 4096 \approx 1.05\)B. Total about 8.0B. ✓

GQA shrinks attention parameters and the wider SwiGLU makes up for it, so "\(12Ld^2\)" stays a good first estimate. For small models the embedding table is a large share. A 1B model with a 128K vocabulary and \(d\)=2,048 spends about 260M parameters on embeddings.

9.2 FLOPs per token: 2N and 6N

A matrix-vector product with a \(d_{in}\times d_{out}\) matrix costs \(d_{in} d_{out}\) multiply-adds, which is 2 FLOPs per parameter. So the forward pass costs about \(2N\) FLOPs per token. Backprop computes gradients with respect to both activations and weights, roughly twice the forward cost, so training costs about \(6N\) FLOPs per token. Kaplan+ 2020 The attention-score term adds roughly \(2 L n_{\text{ctx}} d\) per token forward, which is small unless \(n_{\text{ctx}}\) is comparable to \(6d\) or more, as computed earlier.

That last row is the core insight of LLM serving. At batch 1 you use about 140 GFLOPs out of hundreds of TFLOPs available per second. Batching many users shares each weight read across all of them, which is why throughput rises almost linearly with batch size until either compute or KV-cache memory runs out.

10. Encoder-only vs decoder-only vs encoder-decoder

Why decoder-only won for general models: the next-token objective uses every token as a training signal (MLM uses only about 15%), every task can be phrased as text continuation, there is one homogeneous stack to scale and serve, and in-context learning emerges naturally. Encoders remain the right tool for embeddings, rerankers (cross-encoders), and cheap classifiers, which is relevant for RAG design.

11. Mixture of Experts (MoE)

MoE replaces the single FFN in (some or all) blocks with \(E\) parallel expert FFNs plus a small router. Each token runs through only \(k\) experts, so parameter count (capacity, knowledge) is decoupled from FLOPs per token. Shazeer+ 2017 Lepikhin+ 2020

11.1 Routing

For token representation \(x\), router weights \(W_r \in \mathbb{R}^{d\times E}\):

where \(\tilde{g}\) are the selected scores, usually renormalized to sum to 1. Switch Transformer used \(k=1\) Fedus+ 2021, Mixtral uses 8 experts with \(k=2\) Jiang+ 2024, and DeepSeek-V3 uses 256 routed experts with \(k=8\) plus 1 shared expert, with sigmoid scores. DeepSeek-V3 2024 The router is trained end-to-end through the gate values \(\tilde{g}\). The discrete top-k choice is not differentiable, but the weighting is.

11.2 Total vs active parameters

• FLOPs and latency scale with active parameters: DeepSeek-V3 costs about \(2\times37\text{B}\) FLOPs per token, like a 37B dense model.
• Memory scales with total parameters: all 671B must sit somewhere (about 671 GB in FP8), so MoE needs many GPUs even though each token touches only a few experts.
• Quality sits between the two. An MoE generally beats a dense model of its active size, and falls short of a dense model of its total size at equal training tokens.
• Serving is efficient at high batch, because enough tokens arrive per expert to make matmuls efficient. At batch 1 you still read \(k\) experts' weights per token per layer, and across a sequence you read nearly all of them.

11.3 Load balancing: the central problem

Left alone, routers collapse. A few experts get slightly better early, receive more tokens, improve further, and the rest starve. With expert parallelism (experts spread over GPUs) an overloaded expert also becomes a straggler that stalls everyone.

• Auxiliary load-balancing loss (Switch-style): \(\mathcal{L}_{aux} = \alpha \cdot E \sum_{i=1}^{E} f_i P_i\), where \(f_i\) is the fraction of tokens dispatched to expert \(i\) and \(P_i\) the mean router probability for \(i\). The minimum is at a uniform distribution. \(\alpha\) is small (about 0.01). Fedus+ 2021 The downside is that this loss's gradients fight the language-modelling objective.
• Capacity factor: each expert processes at most \(C \cdot \tfrac{T k}{E}\) tokens per batch, for \(T\) tokens. Overflow tokens are dropped (they skip the FFN via the residual). \(C\) of 1.0–1.25 trades wasted padding against dropped tokens. Many recent systems avoid dropping entirely.
• Router z-loss penalizes large router logits for numerical stability. Zoph+ 2022
• Aux-loss-free balancing (DeepSeek): add a per-expert bias \(b_i\) to the scores only for the top-k selection, not for the gate weights. After each step, decrease \(b_i\) if expert \(i\) was overloaded and increase it if underloaded. Balance is enforced by a control loop instead of a gradient, so there is no interference with the main loss. Wang+ 2024 DeepSeek-V3 uses this plus a tiny sequence-level auxiliary loss, and drops no tokens. DeepSeek-V3 2024

11.4 Fine-grained and shared experts

DeepSeekMoE made two influential changes. Dai+ 2024 (1) Fine-grained experts: split each expert into smaller ones and activate proportionally more (for example 256 small experts with top-8 instead of 16 large ones with top-1). That gives combinatorially more expert combinations and sharper specialization at the same FLOPs. (2) Shared experts: one or more experts that every token uses. They absorb common knowledge (syntax, frequent patterns) so routed experts don't each have to duplicate it. Practice varies: DeepSeek-V3, Kimi K2 and Llama 4 use a shared expert, while Qwen3 and gpt-oss do not.

11.5 Systems realities

• Expert parallelism places different experts on different GPUs. Each MoE layer then needs two all-to-all exchanges (dispatch tokens to expert hosts, gather results back). Interconnect bandwidth becomes the bottleneck, which is why DeepSeek-V3 limits each token to experts on at most 4 nodes and overlaps communication with compute. DeepSeek-V3 2024
• Fine-tuning MoEs is touchier: routers can drift and experts can collapse on narrow data. Many teams freeze the router or keep balancing active during fine-tuning.
• Experts don't specialize along human-legible topics. Specialization is often syntactic or token-level.

12. Long-context techniques

Two costs grow with context: prefill compute (\(O(n^2)\)) and KV-cache memory and bandwidth (\(O(n)\) per layer). The techniques below attack one or both.

12.1 Sliding-window and local/global interleaving

Sliding-window attention (SWA) lets each token attend only to the last \(W\) tokens, so the cache is capped at \(W\) per layer. Information still propagates further through depth: after \(L\) layers the receptive field is about \(L \times W\). Mistral 7B used \(W\)=4,096 across 32 layers. Jiang+ 2023 Pure SWA can't do precise long-range retrieval, so modern models interleave local and global layers:

• Gemma 2: alternating local (4,096) and global layers, 1:1. Gemma 2 2024
• Gemma 3: 5 local : 1 global, local window 1,024. KV overhead at 32K falls from about 60% (global-only) to under 15%. Gemma 3 2025
• gpt-oss: alternating dense and banded-window layers, with a window of just 128 tokens. OpenAI 2025

12.2 Sparse attention

Fixed sparse patterns came first: strided and fixed patterns in Sparse Transformers Child+ 2019, and local plus a few global tokens in Longformer Beltagy+ 2020 and BigBird (which adds random links) Zaheer+ 2020. The modern wave uses learned, content-dependent sparsity designed for GPU kernels:

• NSA (DeepSeek, 2025): combines compressed coarse tokens, top-k selected blocks and a sliding window, and is trainable end-to-end. Yuan+ 2025
• DSA (DeepSeek-V3.2): a lightweight FP8 "lightning indexer" scores all previous tokens, then each query attends to only the top 2,048 key-value entries. Core attention drops from \(O(L^2)\) to \(O(Lk)\), although the indexer itself stays quadratic but cheap. It is built on MLA and was trained by continuing from V3.1-Terminus. DeepSeek 2025 GLM-5 adopted MLA plus DSA in 2026.
• DeepSeek-V4 (2026): interleaves Compressed Sparse Attention (KV compressed 4× along the sequence, then top-k selection) with Heavily Compressed Attention (128× compression, dense). At 1M tokens it reports about 10% of V3.2's KV memory. HF 2026

12.3 Linear attention

Replace \(\mathrm{softmax}(qk^\top)\) with a kernel \(\phi(q)\phi(k)^\top\). Matrix multiplication is associative, so \((\phi(Q)\phi(K)^\top)V = \phi(Q)(\phi(K)^\top V)\). The second form costs \(O(n d^2)\) instead of \(O(n^2 d)\). In causal form it becomes an RNN with a matrix-valued state: Katharopoulos+ 2020

The state \(S \in \mathbb{R}^{d_k \times d_v}\) has constant size, so there is no growing KV cache. The weakness: everything is summed into a fixed-capacity memory, so recall of specific earlier tokens degrades, and vanilla linear attention underperformed softmax. Modern variants add gates/decay (forget old content) and the delta rule (overwrite the value stored under key \(k\) rather than adding to it): Yang+ 2024

12.4 Attention sinks

Trained LLMs put a surprising amount of attention mass on the first few tokens, whatever their content. The softmax must sum to 1, so heads with "nothing to attend to" dump their weight on a consistently visible position. If you evict those tokens in a sliding-window cache, perplexity explodes. StreamingLLM keeps the first ~4 "sink" tokens plus a rolling window, which allows stable generation over millions of tokens (though without recall beyond the window). Xiao+ 2023 gpt-oss builds this in as a learned per-head bias in the softmax denominator, which lets a head attend to "nothing." OpenAI 2025 Vision transformers show the same phenomenon on background patches, and adding dedicated "register" tokens fixes it. Darcet+ 2023

12.5 Systems-level tools

• FlashAttention: exact attention, tiled in SRAM, so memory is \(O(n)\). Not a sparsity method, but it made 32K–128K training practical. Dao+ 2022
• Ring / context parallelism: shard the sequence across devices and pass K/V blocks around a ring, overlapping communication with compute. Liu+ 2023
• KV-cache compression at inference: quantization (FP8/INT4), token eviction (H2O/SnapKV-style heuristics), and offload to CPU or SSD (covered on the inference page).

13. Alternatives and hybrids: SSMs, RWKV, linear-attention hybrids

13.1 State-space models and Mamba

A (discretized) linear state-space model runs a recurrence \(h_t = \bar{A} h_{t-1} + \bar{B} x_t,\; y_t = C h_t\). Classic SSMs (S4) used fixed, input-independent \(A, B, C\), which allows training as a long convolution, but they could not decide what to remember. Mamba makes \(B, C\) and the step size \(\Delta\) functions of the input ("selective"), so the model can choose to ignore or store a token. That breaks the convolution trick, so Mamba uses a hardware-aware parallel scan. Gu & Dao 2023 Inference uses a constant-size state per layer and \(O(1)\) time per token. Mamba-2 restricts \(A\) to a scalar times the identity, which exposes a duality with (masked) linear attention and makes training matmul-friendly. Dao & Gu 2024

13.2 RWKV

RWKV is a linear-attention-style RNN with learned per-channel time decay. It trains in parallel like a transformer and runs as an RNN at inference. Peng+ 2023 Later versions (RWKV-7 "Goose") add delta-rule-like dynamic state updates, converging with the DeltaNet family. Peng+ 2025 It is community-driven and has strong efficiency results, but it is not used in frontier models.

13.3 What the evidence says

A controlled 8B comparison on the same data (up to 3.5T tokens) found pure Mamba/Mamba-2 match transformers on many tasks but lag on copying, in-context learning (5-shot MMLU, phonebook lookup) and long-context reasoning. A hybrid of 43% Mamba-2, 7% attention and 50% MLP layers beat the transformer on all 12 standard tasks evaluated. Waleffe+ 2024 This explains the field's direction: a fixed-size state is cheap but lossy, and a few full-attention layers restore exact retrieval.

13.4 Hybrid models in production

14. Architectures of notable open models

The table shows the design space has converged. Nearly everything is a pre-norm decoder with RMSNorm, SwiGLU/GeGLU, RoPE and GQA or MLA. The differences are in attention sparsity, MoE configuration and vocabulary. Values come from the cited reports; cells marked with a dashed underline are from memory or secondary sources.

15. Multimodal input basics

15.1 Vision Transformer (ViT)

A ViT treats an image as a sequence of patches. Dosovitskiy+ 2020

1. Split an \(H\times W\times 3\) image into \(P\times P\) patches. A 224×224 image with \(P\)=16 gives \(14\times14 = 196\) patches.
2. Flatten each patch (\(16\cdot16\cdot3 = 768\) values) and apply a linear projection to width \(d\). This is equivalent to a stride-\(P\) convolution.
3. Add position embeddings (learned 2-D, or 2-D RoPE in newer encoders) and optionally a [CLS] token.
4. Run a standard bidirectional transformer encoder.

Token count grows with resolution squared. An 896×896 image at \(P\)=14 is 4,096 patches, which is why VLMs pool or compress visual tokens.

15.2 Vision encoder + projector + LLM (late fusion)

• Vision encoder: usually a CLIP or SigLIP ViT pretrained with image-text contrastive learning, so its features are already language-aligned. Radford+ 2021
• Projector: LLaVA used a single linear layer Liu+ 2023 and LLaVA-1.5 a 2-layer MLP Liu+ 2023b. Others use cross-attention resamplers that compress to a fixed number of tokens. Gemma 3 condenses SigLIP outputs to 256 tokens per image. Gemma 3 2025
• Training recipe: (1) alignment, where only the projector is trained on image-caption pairs with both towers frozen; (2) visual instruction tuning, where the projector and LLM (fully or with LoRA) are trained on multimodal conversations.
• Dynamic resolution: Qwen2-VL processes images at native resolution with variable token counts and uses M-RoPE (separate temporal, height and width rotary components). Wang+ 2024 Other models tile high-resolution images into crops plus a thumbnail.

15.3 Early fusion

Early-fusion models put image tokens (discrete VQ codes or continuous patch embeddings) into one backbone from the start of pretraining, rather than bolting a vision encoder onto a finished LLM. Examples are Chameleon Chameleon Team 2024, Llama 4 Meta 2025, and in 2026 Kimi K2.5 and Qwen3.5. Raschka 2026 The benefit is deeper cross-modal integration. The cost is training from scratch with mixed data, and the open question of how to balance modalities.

Interview question bank