Pre-training

Pre-training is the step that turns a randomly initialised transformer into a base model. You run next-token prediction over trillions of tokens of filtered text and code, and almost all of the model's knowledge and raw capability comes from it. Interviewers ask about it because it pulls together statistics (the objective and scaling laws), data engineering (the pipeline is now where most of the quality gains come from), numerical optimisation (optimisers, schedules, stability) and cost reasoning (FLOPs, GPU-hours, MFU). This page teaches each piece well enough for you to reason about it out loud and back it up with numbers.

1. The objective: next-token prediction

Cross-entropy, the autoregressive factorisation, and teacher forcing

A decoder-only language model defines a distribution over a sequence by the chain rule:

Pre-training minimises the average negative log-likelihood (NLL) of the training tokens:

This is the cross-entropy between the empirical data distribution and the model. Minimising it is the same as minimising \(\mathrm{KL}(p_{\text{data}} \,\|\, p_\theta)\) plus the data's own entropy, which is a constant. So the loss can never go below the irreducible entropy of text. That floor is the \(E\) term in the scaling laws below. Training uses teacher forcing: with a causal mask, all \(T\) positions of a sequence are predicted in parallel from the ground-truth prefixes. One forward pass therefore gives \(T\) training signals. This is a big reason why pre-training is so efficient on GPUs.

Practical details interviewers sometimes check:

• Packing. Documents are concatenated with an end-of-sequence token and cut into fixed-length windows (e.g. 4k or 8k tokens), so no compute goes to padding. Some recipes add an intra-document attention mask so tokens can't attend across document boundaries. Llama 3 did this, and it matters more at long context Grattafiori+ 2024.
• Loss in nats or bits. Frameworks report natural-log loss (nats per token). Divide by \(\ln 2\) to get bits.
• Multi-token prediction (MTP). Extra heads also predict tokens \(t{+}2, t{+}3,\dots\). This densifies the signal and allows speculative decoding later. DeepSeek-V3 used an MTP auxiliary objective DeepSeek-AI 2024. It is an add-on to next-token prediction, not a replacement.

Perplexity and bits-per-byte

Perplexity is \(\mathrm{PPL} = \exp(\mathcal{L})\): the effective number of equally likely choices per token. A loss of 2.0 nats per token gives PPL ≈ 7.4. Perplexity is tokenizer-dependent. A tokenizer with a larger vocabulary packs more text into each token, so per-token loss goes up even if the model is just as good per character. You cannot compare Llama and GPT perplexities directly.

Bits-per-byte (BPB) fixes this by normalising total information by the raw UTF-8 byte count:

Example: loss 2.0 nats per token with about 4 bytes per token (typical for English with a BPE vocabulary of roughly 100k) gives \(2.0/0.693/4 \approx 0.72\) BPB. The Pile popularised BPB as a tokenizer-agnostic metric Gao+ 2020. BPB is also a direct compression rate: a model at 0.7 BPB, combined with arithmetic coding, would compress that text to about 9% of its raw size.

Why the objective produces general capabilities, and its limits

• Implicit multitask learning. Web text contains Q&A, translations, tutorials, code with comments and worked problems. Predicting all of it means learning all of those tasks. GPT-3 showed that this yields in-context few-shot learning at scale Brown+ 2020.
• Smooth loss, uneven skills. Validation loss improves as a smooth power law. Individual benchmarks can look flat and then jump (see the emergence debate in §3).
• Limits. The objective rewards imitating the data distribution, including its errors. It has no notion of helpfulness, truthfulness or following instructions. Those come from post-training (A4). It also spends equal effort on every token, whether trivial or crucial. That motivates data selection, weighting tokens by importance, and MTP-style objectives.

2. Data: sources, pipeline, mixtures

Since about 2023 the clearest consensus in open research has been that data quality and composition are worth more than most architecture tweaks. DCLM's controlled experiments found model-based filtering to be the single most important curation step. A 7B model trained on their filtered pool reached 64% MMLU on 2.6T tokens, roughly matching Llama 3 8B on MMLU with about 6.6× less compute Li+ 2024.

Sources

The filtering pipeline, step by step

1. Extraction

Common Crawl ships raw HTML (WARC) plus pre-extracted text (WET). WET text keeps lots of menus and boilerplate. FineWeb found that running its own extraction on WARC with trafilatura produced noticeably better models than using WET files, even though it costs more compute Penedo+ 2024. Math and code need special care: naive extraction destroys LaTeX and indentation.

2. Language identification

A fastText classifier assigns a language and a confidence to each document. Documents below a confidence threshold are dropped. FineWeb kept English documents with score above about 0.65 Penedo+ 2024. Multilingual pipelines route each language to its own filters, because English-tuned heuristics such as stop-word lists over-filter other languages.

3. Heuristic filters

These are cheap, document-level rules built from statistics of "good" text. The canonical sets are the Gopher rules Rae+ 2021 and the C4 rules Raffel+ 2019:

• Document length bounds (e.g. 50 to 100k words) and mean word length in a sane range.
• Symbol-to-word ratio (too many # or …), and fraction of lines starting with bullets or ending in ellipses.
• Must contain some common stop words (catches word salad and keyword lists).
• Repetition: fraction of duplicated lines, paragraphs and n-grams inside the document (catches templated spam).
• C4-style: keep only lines ending in terminal punctuation, drop pages with "lorem ipsum" or curly braces, drop pages that hit a bad-word list. The bad-word rule is now known to remove a disproportionate amount of dialectal and minority-community text.

RefinedWeb showed that web data alone, if filtered and deduplicated hard enough, can match curated corpora Penedo+ 2023.

4. Deduplication

Duplicates waste compute, increase memorisation and regurgitation, and can leak eval data. Deduplicating made models better and reduced verbatim memorised output Lee+ 2021. There are three levels:

MinHash mechanics. For a random hash function \(h\), the probability that two sets have the same minimum hash equals their Jaccard similarity \(s = |A\cap B|/|A\cup B|\). With \(b\) bands of \(r\) rows, the chance that two docs become candidates is

which is an S-curve with threshold near \((1/b)^{1/r}\). FineWeb used 5-gram shingles and 112 hashes split into 14 buckets of 8 (threshold ≈ 0.75 similarity) Penedo+ 2024. You tune \(b, r\) to trade false positives against false negatives.

5. Model-based quality classifiers

This is the biggest single win of the 2024 era. You train a cheap classifier to score "educational" or "high-quality" text and keep the top slice:

• FineWeb-Edu. Llama-3-70B-Instruct rated a few hundred thousand web pages on a 0–5 educational-value scale. A small regression head on text embeddings was trained on those labels, and the corpus was filtered at score ≥ 3. That kept about 1.3T tokens out of FineWeb's ~15T, and gave large gains on knowledge and reasoning benchmarks such as MMLU and ARC Penedo+ 2024 (dataset).
• DCLM-Baseline. A fastText bigram classifier trained with positives from instruction-style data (OpenHermes-2.5 and ELI5 answers) and negatives from random web pages. It keeps roughly the top 10%. Surprisingly, this simple classifier beat fancier options such as perplexity filtering and embedding-based scorers in their tests Li+ 2024.
• Nemotron-CC. Uses an ensemble of classifiers and LLM rephrasing of lower-quality documents instead of discarding them, to keep more unique tokens for long training runs Su+ 2024.
• Llama 3 used Llama-2-based quality classifiers plus separate code and reasoning classifiers Grattafiori+ 2024.

The trade-off: aggressive classifier filtering raises benchmark scores per token but shrinks the pool and its diversity. It can also bias toward "textbook English" and away from conversational, creative or long-tail knowledge. When you plan to train for 15T+ tokens you cannot keep only the top 10%, so labs bucket data by quality and upsample the better buckets.

6. Toxicity, PII, and safety filtering

Toxicity classifiers and URL blocklists (adult sites, known malware or spam domains) remove harmful content. Regex and NER-based scrubbers mask emails, phone numbers, IPs and IDs. Over-filtering for toxicity can hurt the model's ability to recognise toxicity and can remove dialects, so some labs filter lightly and handle behaviour in post-training.

7. Decontamination

Remove training documents that overlap with benchmark test sets, usually by n-gram matching (GPT-3 used 13-gram overlap Brown+ 2020). It is never perfect. Paraphrased, translated or reformatted copies slip through, and benchmark answers spread across the web after release. For interviews, the takeaway is that published scores of models trained on recent web crawls are partly inflated by contamination. Strong evaluation therefore uses held-out, post-cutoff or private sets (A8).

Data mixtures and mixture optimisation

After filtering you have domains (web, code, math, books, multilingual and so on) and must pick sampling weights. The weights do not have to match natural proportions. High-value domains are upsampled (seen for several epochs) and low-value ones downsampled. Llama 3 reports a final mix of roughly 50% general knowledge, 25% math and reasoning, 17% code, 8% multilingual tokens Grattafiori+ 2024.

Ways to choose weights:

• Hand-tuning plus ablations. Still the most common. Train small models on candidate mixes and compare.
• DoReMi. (1) Train a small reference model on a default mix. (2) Train a small proxy model with Group-DRO, which keeps raising the weight of domains where the proxy's loss is far above the reference's (high "excess loss" means the domain is learnable but not yet learned). (3) Average the domain weights over training and use them to train the big model. A 280M proxy produced weights that sped up an 8B model's training on The Pile by about 2.6× to baseline accuracy Xie+ 2023.
• RegMix / data-mixing laws. Train many tiny models on random mixtures, fit a regression (or a functional "mixing law") from mixture weights to validation loss, then pick the predicted best mix Liu+ 2024 Ye+ 2024.

Caveat: the best mix depends on the target you optimise, such as average validation loss versus a code benchmark. It also depends on scale and total token budget, because with more tokens small high-quality domains get repeated too often. Mixtures found with small proxies transfer imperfectly. This is an active research area.

Curriculum and annealing

Strict easy-to-hard curricula have mixed evidence for LLMs. What clearly works is end-of-training emphasis on high-quality data: when the learning rate decays toward zero, the model "settles" into whatever it is seeing, so you switch to your best data then.

• Llama 3 annealed on a small set of high-quality code and math while decaying the LR to zero. This lifted GSM8K and MATH noticeably for the 8B model (less for 405B). They also used short annealing runs as a cheap way to measure the value of a new dataset Grattafiori+ 2024.
• MiniCPM put high-quality and SFT-style data into the decay phase of its WSD schedule Hu+ 2024. OLMo 2 formalised a "mid-training" stage with a curated high-quality mix and averaged ("souped") several annealed runs OLMo+ 2024.
• Qwen3 used three stages: about 30T general tokens, then about 5T tokens of STEM, code, reasoning and synthetic data, then a long-context stage at 32k tokens Yang+ 2025.

Synthetic data in pre-training

• Textbook-style generation. phi-1 (1.3B) was trained on filtered code plus GPT-3.5-generated "textbooks" and exercises, and was very strong on HumanEval for its size Gunasekar+ 2023. The phi line kept leaning on synthetic data Abdin+ 2024.
• Rephrasing the web. An instruction model rewrites web documents in styles such as "like Wikipedia" or "as Q&A". Mixing rephrased and real data sped up pre-training by about 3× in WRAP's setup Maini+ 2024. Nemotron-CC and Kimi K2 use rephrasing to get more useful, unique tokens out of limited high-quality data Su+ 2024 Kimi 2025.
• Reasoning traces and code-execution data are increasingly mixed into mid-training, which blurs the line with post-training.
• Risks. Training recursively on model output alone causes "model collapse", where the tails of the distribution vanish Shumailov+ 2023. Collapse is largely avoided when synthetic data is added to real data rather than replacing it Gerstgrasser+ 2024. Other risks: low diversity, the generator's errors and biases, contamination (the generator may have memorised benchmarks), and licensing terms for generator outputs.

Data repetition limits (epochs)

High-quality text is finite, so how much is repeated data worth? Muennighoff+ trained 400 models up to 9B parameters and 900B tokens. With a fixed compute budget, up to about 4 epochs of repeated data gave a negligible loss penalty compared with unique data. With more repetition the value of extra compute falls off, and it becomes nearly worthless around 16+ epochs (their fitted "half-life" of repeated data). They also found that adding code and loosening filters were reasonable ways to stretch scarce data Muennighoff+ 2023. In practice, small premium domains (Wikipedia, textbooks, math) are commonly seen for several epochs, while web data is seen about once.

How many tokens modern models are trained on

The pattern: frontier-scale open models sit in the 15–40T token range, and small models are trained far past Chinchilla-optimal. Closed frontier labs generally do not disclose token counts, so any numbers quoted for them are estimates.

3. Scaling laws

Training compute: \(C \approx 6ND\)

For a dense transformer with \(N\) non-embedding parameters, the forward pass costs about \(2N\) FLOPs per token: each weight is used in one multiply and one add. The backward pass costs about twice the forward, because it computes gradients with respect to both activations and weights. That gives

This ignores attention-score FLOPs, roughly \(6 \cdot n_{\text{layers}} \cdot T \cdot d_{\text{model}}\) extra per token (counting forward and backward), where \(T\) is the context length. They are a few percent at 4k context for big models but become significant at long context or for small models Kaplan+ 2020. For MoE models use the active parameters per token: DeepSeek-V3 costs like a ~37B dense model per token, not 671B. Activation recomputation (checkpointing) adds another forward pass, about 8ND, but that counts toward hardware utilisation, not model FLOPs (see MFU in §8).

Kaplan et al. 2020

OpenAI fit power laws to models up to about 1.5B parameters Kaplan+ 2020:

with \(\alpha_N \approx\) 0.076, \(\alpha_D \approx\) 0.095 and \(\alpha_C \approx\) 0.050. Key claims:

• Loss depends mainly on scale (\(N\), \(D\), \(C\)). Shape choices such as depth versus width barely matter within a wide range.
• Big models are more sample-efficient. For a fixed compute budget you should train a very large model and stop well before convergence. The optimal size grows as \(N_{\text{opt}} \propto C^{\approx 0.73}\) and data only as \(D \propto C^{\approx 0.27}\).

This guided the GPT-3 era: 175B parameters trained on only 300B tokens.

Chinchilla (Hoffmann et al. 2022)

DeepMind trained over 400 models from 70M to 16B parameters, with the cosine schedule length matched to each run's token budget. They estimated the compute-optimal allocation three ways: (1) the lower envelope of training curves, (2) IsoFLOP profiles, and (3) fitting a parametric loss Hoffmann+ 2022:

Here \(E\) is the irreducible entropy of text, the second term is the penalty for a finite model, and the third is the penalty for finite data. Their Approach-3 fit was \(E \approx 1.69,\ A \approx 406.4,\ B \approx 410.7,\ \alpha \approx 0.34,\ \beta \approx 0.28\) (values as published; see replication caveat below). Minimising \(L\) subject to \(C = 6ND\) gives

and all three approaches put both exponents near 0.5: parameters and tokens should grow equally with compute. The memorable rule is about 20 tokens per parameter. Chinchilla (70B, 1.4T tokens) used the same compute as Gopher (280B, 300B tokens) and beat it across the board.

Why Kaplan and Chinchilla disagreed

Replication caveat: Besiroglu+ refit Chinchilla's Approach 3 from data reconstructed out of the paper's figures. They found the published parametric fit was inconsistent with the other two approaches and had implausibly narrow confidence intervals. Their corrected fit agrees with ~20 tokens per parameter Besiroglu+ 2024. The headline rule stands, but don't over-trust the exact published constants. DeepSeek also found that the optimal ratio depends on data quality: higher-quality data shifts the optimum toward bigger models DeepSeek-AI 2024.

Inference-aware scaling: why modern models are "over-trained"

If a model will serve \(D_{\text{inf}}\) tokens over its lifetime, total cost is about \(6ND_{\text{train}} + 2ND_{\text{inf}}\). A smaller model trained longer reaches the same loss at a somewhat higher training cost but a much lower serving cost. Sardana+ formalised this. With large inference demand, the optimal choice is smaller-and-longer than Chinchilla, and quality kept improving up to extreme ratios of 10,000 tokens per parameter in their runs Sardana+ 2023.

Worked example with the Chinchilla fit (illustrative only):

About 1.2× the training compute buys a model that is 8.75× cheaper to serve at about the same loss. That is the Llama 3 8B strategy in one line Grattafiori+ 2024. The fit is known to be imprecise at extreme ratios (Sardana+ found standard fits misestimate the value of extra tokens there), so treat this as a direction rather than a precise prediction.

Emergent abilities: real or a measurement artefact?

Wei+ catalogued tasks where performance stays near chance and then rises sharply past some scale, such as multi-digit arithmetic and some BIG-Bench tasks Wei+ 2022. Schaeffer+ argued that many such jumps come from discontinuous metrics like exact-match accuracy. If per-token accuracy improves smoothly, the probability of getting all \(k\) tokens of an answer right is \(p^k\), which looks like a sudden jump. Under continuous metrics (token edit distance, log-likelihood of the correct answer) the curves become smooth and predictable Schaeffer+ 2023.

Balanced view as of 2026: the underlying capability usually improves smoothly with loss, but usable task success can still cross thresholds sharply. That matters in practice: an agent that succeeds 30% of the time versus 90% is a different product. So "emergence" is partly a metric artefact and partly a genuine property of composite tasks. Labs predict downstream metrics by mapping compute → loss → task metric with a fitted sigmoid, rather than extrapolating accuracy directly. Llama 3 used this two-step method to predict the 405B model's ARC-Challenge score from much smaller runs Grattafiori+ 2024.

Fitting scaling laws and using them for hyperparameters

How it's done in practice:

1. Pick an IsoFLOP grid. For about 5–8 compute budgets (e.g. 1e18 to 1e21 FLOPs), train several model sizes at each budget, with the LR schedule set to the run length. Note the minimum-loss model size at each budget, fit a parabola in \(\log N\), and fit a power law of \(N_{\text{opt}}\) against \(C\).
2. Or fit the parametric form \(E + A N^{-\alpha} + B D^{-\beta}\) with a robust loss (Chinchilla used Huber on log-loss with L-BFGS). Bootstrap to get confidence intervals. These are often wider than people expect.
3. Extrapolate 1–2 orders of magnitude, not 5. GPT-4's final loss was reportedly predicted from runs using about 1,000–10,000× less compute OpenAI 2023.
4. WSD makes this cheaper. One long constant-LR run with branched cooldowns produces many token budgets from a single run, instead of one cosine run per budget Hägele+ 2024.

Hyperparameter scaling laws. DeepSeek LLM fit the optimal batch size and learning rate as power laws in compute, roughly \(B_{\text{opt}} \propto C^{0.33}\) and \(\eta_{\text{opt}} \propto C^{-0.125}\) (their fitted exponents). They found a broad near-optimal basin and used the fits to set the 7B and 67B runs DeepSeek-AI 2024. As compute grows, the optimal batch gets larger and the optimal LR gets smaller. This complements μP (§4), which makes the optimal LR invariant to width by construction.

4. Optimisation

AdamW: mechanics and memory

For each parameter, with gradient \(g_t\) Kingma+ 2014 Loshchilov+ 2017:

• First moment \(m\): momentum. It smooths noisy minibatch gradients.
• Second moment \(v\): a per-coordinate scale. Dividing by \(\sqrt{v}\) normalises each update to about unit size, so rarely updated or small-gradient parameters (embeddings of rare tokens) still move. This makes Adam far less sensitive to gradient scale than SGD.
• Decoupled weight decay (the "W"): decay is applied directly to the weights rather than added to the gradient. If it is added to the gradient, Adam's normalisation rescales it per coordinate, which is not what you want.
• Typical LLM settings: \(\beta_1 = 0.9\), \(\beta_2 = 0.95\) (lower than the 0.999 default, which helps stability at large scale because \(v\) adapts faster after a gradient-scale shift), \(\epsilon = 10^{-8}\), weight decay 0.1, gradient clip 1.0. Peak LR falls with model size: about \(3\times10^{-4}\) at 7B and about \(1.5\times10^{-4}\) at 70B for Llama 2, and 8e-5 for Llama 3 405B.

Memory. With BF16 mixed precision and Adam, a common accounting is 16 bytes per parameter: BF16 weights (2) + BF16 or FP32 grads (2–4) + FP32 master weights (4) + FP32 \(m\) (4) + FP32 \(v\) (4). For 70B that is about 1.1 TB before activations. No single GPU can hold it, so you need ZeRO/FSDP sharding (A3). Adam's two states are the motivation for memory-lean optimisers such as Adafactor (factored second moment), 8-bit Adam, and Muon (one state).

Learning-rate schedules: warmup, cosine, WSD

• Warmup (linear, typically a few hundred to a few thousand steps). At initialisation Adam's \(v\) estimates are poor and the loss landscape is sharp. A large LR then causes divergence or wasted early steps. Warmup lets statistics settle and the network move into a flatter region Goyal+ 2017.
• Cosine decay to about 10% of peak (or to zero) over the planned run. It is robust and was the standard through 2023. Its drawback: you must fix the total steps in advance, and an intermediate checkpoint is not a finished model because its LR is still high.
• WSD (warmup–stable–decay): hold a constant LR for most of training, then decay quickly (linear, cosine or "1−sqrt") over the last ~10–20%. It matches or beats cosine Hu+ 2024 Hägele+ 2024. Advantages: you can extend training indefinitely, branch cooldowns for scaling-law fits, and put high-quality data into the decay phase (annealing). DeepSeek-V3 used a closely related schedule: constant, then cosine decay, then a short low-LR tail DeepSeek-AI 2024.

Batch size and the critical batch size

Larger batches average away gradient noise, but past a point they stop reducing the number of steps needed. McCandlish+ model this with the gradient noise scale \(B_{\text{noise}} = \operatorname{tr}(\Sigma)/|G|^2\), the ratio of per-example gradient variance to squared true-gradient norm. Below \(B_{\text{crit}} \approx B_{\text{noise}}\), doubling the batch roughly halves the steps (perfect scaling). Above it, returns diminish and you waste compute McCandlish+ 2018. They describe the trade-off as

where \(S\) is steps and \(E\) is examples processed. Key facts:

• The noise scale grows as loss falls: near the optimum the true gradient is small relative to noise. So the critical batch size rises during training. Hence batch-size ramp-up: Llama 3 405B went from about 4M to 8M to 16M tokens per batch as training progressed (schedule as reported) Grattafiori+ 2024.
• Recent work finds critical batch size scales mainly with data size (training duration) rather than model size Zhang+ 2024.
• Batch size and LR interact: bigger batches allow bigger LRs, up to a limit. Systems constraints (enough data parallelism to keep 10k+ GPUs busy) push batches up. Optimisation efficiency pushes them down. Typical frontier batches are a few million to tens of millions of tokens.

μP and hyperparameter transfer

Under standard parametrisation (SP) the optimal LR shrinks as width grows, so a sweep at 100M parameters doesn't tell you the LR for 10B, and sweeping at 10B is unaffordable. Maximal Update Parametrisation (μP) rescales init, per-layer LR and output multipliers so that every layer's activations and updates stay \(\Theta(1)\) as width \(\to \infty\). The optimal hyperparameters then become (approximately) width-invariant Yang+ 2022.

Workflow ("μTransfer"): tune LR, init scale and multipliers on a narrow proxy (e.g. width 256 to 1024), then scale width with the same values. The paper transferred hyperparameters from a ~40M-parameter proxy to GPT-3 6.7B and beat the published model, with tuning costing only about 7% of a pre-training run Yang+ 2022 (mup library). Caveats: classic μP covers width, while depth transfer needs extra residual-branch scaling (an active research line). Transfer across token budget and batch size isn't guaranteed. Many labs combine μP-style parametrisation with empirical scaling-law fits for LR and batch size.

Newer optimisers: Muon, Shampoo, SOAP

Muon (MomentUm Orthogonalized by Newton–Schulz), introduced by Keller Jordan in 2024 (blog post). For each 2-D hidden weight matrix:

It replaces the momentum matrix with its nearest semi-orthogonal matrix, so all singular values are set to 1. This uses about 5 Newton–Schulz iterations of matmuls instead of an SVD. The intuition: raw gradient updates for transformer weights are dominated by a few directions (low effective rank). Orthogonalising boosts the "rare" directions, giving a better-conditioned update with steepest descent under the spectral norm. Embeddings, the LM head and gains/biases still use AdamW. Memory: one momentum buffer instead of two.

• Moonshot added weight decay and per-matrix update-RMS matching so that Muon "just works" with AdamW-tuned LRs. They reported about 2× compute efficiency over AdamW in their scaling experiments, and trained Moonlight (16B MoE, 5.7T tokens) with it Liu+ 2025.
• Kimi K2 (1T total parameters) used MuonClip, which is Muon plus "QK-clip": rescale the query and key projection weights when attention logits get too large. It pre-trained on 15.5T tokens with no loss spikes Kimi 2025. Essential AI also reported practical efficiency gains, with Muon staying data-efficient at large batch sizes Essential AI 2025.
• Counterpoint: a careful benchmark with equal tuning effort found that claimed 1.4–2× speedups over well-tuned AdamW shrink with scale, to about 1.1× at 1.2B parameters. The fastest optimisers (Muon, SOAP) were all matrix-preconditioned Wen+ 2025.

Shampoo keeps left and right preconditioners for each weight matrix, \(L = \sum G G^\top\) and \(R = \sum G^\top G\), and updates with \(L^{-1/4} G R^{-1/4}\). That is a Kronecker-factored approximation of full-matrix AdaGrad Gupta+ 2018. It is expensive: matrix inverse roots, computed only every \(k\) steps, plus extra memory. SOAP runs Adam in Shampoo's eigenbasis (updated occasionally). This is more stable, has fewer hyperparameters, and showed meaningful step and wall-clock savings over AdamW at the 360M–660M scale Vyas+ 2024. Muon can be seen as Shampoo with the preconditioner accumulation turned off.

5. Training stability

Large runs fail in two main ways: loss spikes, where the loss jumps by a lot over tens of steps and sometimes recovers, and divergence, where it never recovers. Both get more frequent with scale, higher LR and lower precision. The 175B OPT logbook is a frank record of dozens of restarts, LR cuts and hardware failures Zhang+ 2022. PaLM reported roughly 20 spikes despite gradient clipping. Its fix was to restart from a checkpoint about 100 steps before the spike and skip a few hundred data batches, which suggests spikes come from specific batches interacting with a particular parameter state Chowdhery+ 2022.

Known failure modes and their fixes

OLMo 2 is a good recent open case study. It tracked down spikes and fixed them with a combination of QK-norm, z-loss, changed norm placement, no weight decay on embeddings, and data filtering of repeated n-grams. The result was much smoother loss curves OLMo+ 2024. DeepSeek-V3 and Kimi K2 both report runs with no irrecoverable spikes and no rollbacks DeepSeek-AI 2024 Kimi 2025.

Mixed precision (overview)

• BF16 (8 exponent bits, 7 mantissa bits) has the same range as FP32, so unlike FP16 it needs no loss scaling. Matmuls run in BF16 with FP32 accumulation. Master weights, optimiser states and reductions stay in FP32. It has been the default since about 2021.
• FP8 (E4M3 for activations and weights, E5M2 for gradients) doubles tensor-core throughput on Hopper and later GPUs. It needs careful per-tensor or fine-grained scaling. DeepSeek-V3 trained with FP8 at scale using tile/block-wise scaling and higher-precision accumulation DeepSeek-AI 2024.
• FP4-class formats for training are emerging on Blackwell-generation hardware. They are early and less proven.

Details on precision, memory sharding and parallelism are on the next page (A3: Distributed training).

6. Stages of a modern training run

Why train short first, then extend?

Attention cost per token grows with sequence length, and most documents are short anyway. Training at 4k–8k for most tokens is far cheaper. Context is then extended near the end:

• RoPE scaling. Raise the rotary base frequency (Llama 3 used base 500,000), use position interpolation Chen+ 2023, or use YaRN's frequency-dependent interpolation Peng+ 2023. These keep the rotation angles the model saw in training in range at longer positions. See A1 for RoPE.
• Continued training on long data. Llama 3 extended from 8K to 128K in about six stages over roughly 800B tokens, moving to the next stage only once short-context performance had recovered and needle-in-a-haystack retrieval was solved at the current length Grattafiori+ 2024. Qwen3 used hundreds of billions of tokens at 32k Yang+ 2025.
• Data lessons. Code repositories and books are good natural long-document sources. Mixing in high-quality short data keeps short-task ability intact. Evaluate on real downstream long-context tasks, because perplexity and simple needle tests can be misleading Gao+ 2024.

Mid-training

"Mid-training" became a common term in 2024–25 for the phase between general pre-training and post-training. It is a lower-LR, quality-heavy stage that upweights math, code, reasoning traces, synthetic textbooks and sometimes instruction-formatted data. It raises benchmark scores far more per token than general pre-training and prepares the base model for RL. Examples: OLMo 2's mid-training mix plus checkpoint souping OLMo+ 2024, and Qwen3's ~5T-token reasoning stage Yang+ 2025. The trade-off: a heavy instruction-like mix can make "base" model comparisons unfair (benchmark-tuned base models), and contamination risk goes up.

7. Checkpointing and evaluation during pre-training

Checkpointing

• What to save: model weights, optimiser states (\(m, v\) are about 2/3 of checkpoint size with Adam), LR scheduler step, data-loader position and shuffle seed, and RNG states. With all of these you get bit-for-bit or at least statistically identical resumption. A 70B checkpoint with Adam states is about 1 TB.
• Why it matters at scale: with thousands of GPUs, failures are routine. Llama 3 405B saw ~419 unexpected interruptions in a 54-day snapshot, most from GPU and HBM faults, which is roughly one every few hours Grattafiori+ 2024.
• How often: trade lost work against checkpoint overhead. The Young/Daly rule of thumb is interval \(\approx \sqrt{2 \cdot t_{\text{ckpt}} \cdot \text{MTBF}}\). Use sharded, asynchronous checkpoints: each rank writes its own shard to fast storage while training continues, then it is copied to object storage. Keep a sparse set of permanent milestones for analysis (Pythia released 154 checkpoints per model for research Biderman+ 2023).

Evaluation during the run

8. Compute and cost back-of-envelope

The method

1. FLOPs: \(C = 6ND\) (dense) or \(6N_{\text{active}}D\) (MoE). Add about 5–15% for attention at long context if needed.
2. Effective throughput per GPU: peak dense FLOP/s at your precision × MFU. MFU (model FLOPs utilisation) is the model FLOPs actually achieved divided by peak. It excludes recomputation, unlike HFU (hardware FLOPs utilisation), which counts recompute. Well-tuned large dense runs reach about 35–45% BF16 MFU. Llama 3 405B reported 38–43% on 16k H100s Grattafiori+ 2024.
3. GPU-hours = \(C\) / (effective FLOP/s × 3600).
4. Wall-clock = GPU-hours / #GPUs, plus 10–20% for failures, restarts, evals and checkpoints.
5. Dollars = GPU-hours × $/GPU-hour (owned clusters or long reservations are typically ~$2–3 per H100-hour, versus higher on-demand prices).

Worked example: 70B dense on 15T tokens on H100s at 40% MFU

Sanity checks against reported numbers:

• Llama 3.1 405B: \(6 \times 4.05\text{e}11 \times 1.56\text{e}13 \approx 3.8\text{e}25\) FLOPs. The reported cumulative compute was about 30.8M H100-hours (model card), which implies about 35% effective utilisation once overheads are included. That is consistent with the paper's 38–43% MFU during steady-state training.
• Llama 3.1 70B: the model card lists about 7.0M H100-hours, which implies a lower effective utilisation (about 25%). This shows that overheads, long-context stages and smaller-model parallelism inefficiencies matter. Treat 40% MFU as a good-case assumption.
• DeepSeek-V3: \(6 \times 3.7\text{e}10 \times 1.48\text{e}13 \approx 3.3\text{e}24\) FLOPs in 2.788M H800-hours DeepSeek-AI 2024. That works out to about 330 TFLOP/s effective per GPU, roughly a third of BF16 dense peak (lower relative to FP8 peak). The widely quoted "~$5.6M" figure is just GPU-hours × $2. It excludes research, ablations and prior runs.

9. Small-model training tricks

Distillation-based pre-training

Instead of one-hot next-token targets, the student minimises KL divergence to a larger teacher's full next-token distribution Hinton+ 2015:

• Why it helps: the teacher's soft distribution carries much more information per token than a single sampled label, including which alternatives are plausible and how uncertain the prediction is. It acts like a lower-variance gradient. Small models trained this way beat same-size models trained on raw data with the same number of tokens.
• Who does it: Gemma 2's 2B and 9B were pre-trained with distillation from a larger model, on far more tokens than compute-optimal Gemma Team 2024. Gemma 3 extended distillation across its family Gemma Team 2025. NVIDIA's Minitron pruned a 15B model (width and depth) and recovered quality with distillation using up to about 40× fewer training tokens than training from scratch Muralidharan+ 2024.
• When it's worth it: Distillation Scaling Laws found distillation beats standard training mainly when the student's compute budget is limited, and when a teacher already exists or will be reused across many students. With enough student compute, supervised training catches up. An overly strong teacher is not always better (the "capacity gap") Busbridge+ 2025.
• Engineering cost: an online teacher adds a forward pass, about \(2N_{\text{teacher}}\) FLOPs per token, which can exceed the student's own training cost. Offline alternatives store the teacher's top-k logits per token. That costs storage, and the truncated distribution loses some signal.

Other tricks that matter for small models

• Heavy over-training on curated data. SmolLM2 (1.7B) trained on about 11T tokens with a multi-stage mix that rebalanced web, code, math and new curated datasets between stages Allal+ 2025. Qwen3 small models saw the full 36T-token pipeline.
• Tied input and output embeddings. With a 128k–256k vocabulary, embeddings can be a large share of a 1B model's parameters. Tying saves memory at a small quality cost.
• Pruning plus distillation from an existing larger model (Minitron, and reportedly Llama 3.2 1B/3B) is cheaper than training from scratch.
• Upcycling. Initialise an MoE from a dense checkpoint by copying the MLP into experts Komatsuzaki+ 2022, to reuse sunk compute.
• Synthetic and textbook-quality data gives a disproportionate benefit at small scale, since small models can't afford to memorise noise (phi series).

Interview question bank