Post-training & Alignment

Pretraining gives you a model that has read the internet and can continue any document. Post-training turns that model into an assistant: one that follows instructions, holds a conversation, reasons step by step, uses tools, declines harmful requests and doesn't refuse harmless ones. This page covers the whole stack: supervised fine-tuning, reward models, RLHF with PPO, DPO and its relatives, RL with verifiable rewards, AI feedback, parameter-efficient fine-tuning (LoRA/QLoRA), distillation, model merging and safety training. Interviewers ask about it because nearly every applied AI role touches some part of it. You might be fine-tuning an open model, building a preference dataset, choosing between LoRA and full fine-tuning, or debugging a model that got worse after tuning. And the theory (the KL-regularized objective, the DPO derivation) is a favourite whiteboard topic.

Why post-train? Base models vs assistants

A pretrained ("base") model is a document completer. Ask it "What is the capital of France?" and it may answer, or it may continue with four more quiz questions, because that is what such text looks like on the web. The base model already holds the knowledge and much of the skill. What it lacks is a consistent persona and interface: knowing that a user turn ends and an assistant turn begins, that it should answer helpfully and stop, that it should follow formatting instructions, and that some requests should be declined.

The canonical demonstration was InstructGPT: humans preferred outputs from a 1.3B-parameter model fine-tuned with SFT and RLHF over outputs from the 175B GPT-3 base model, despite the 100× size gap. Ouyang+ 2022 Post-training is extremely leveraged: it uses a tiny fraction of pretraining compute (historically well under ~1–2%, though that share has grown a lot with RL-heavy reasoning training) and changes perceived quality dramatically.

The superficial alignment hypothesis

LIMA fine-tuned a 65B LLaMA on just 1,000 carefully curated prompt–response pairs, with no RLHF. Its responses were judged equal or preferred to GPT-4's in a substantial fraction of cases. Zhou+ 2023 The authors proposed the superficial alignment hypothesis: almost all knowledge and capability is learned in pretraining, and alignment mostly teaches which sub-distribution of formats and styles to use when talking to users. Supporting evidence: URIAL showed that base models aligned purely in context (a few stylistic examples plus a system prompt, no weight updates) get surprisingly close to their SFT+RLHF counterparts. Token-distribution analysis showed that aligned and base models disagree mainly on stylistic tokens ("Sure", "However", discourse markers), not on content tokens. Lin+ 2023

Where the hypothesis breaks

• Reasoning via RL. Large-scale RL on verifiable tasks (DeepSeek-R1, OpenAI o-series) produces large gains on math and code benchmarks, long chains of thought, self-verification and backtracking. DeepSeek-AI 2025 That is more than format. Whether it is new capability or better elicitation of latent capability is debated: one study found RLVR raises pass@1 but the base model catches up and even overtakes at large pass@k, suggesting RL mostly sharpens the distribution towards solutions the base model could already sample. Yue+ 2025 Treat this as contested.
• Long-tail skills and robustness. Tool use, long-horizon agentic behaviour, calibrated refusals, multi-turn consistency and instruction hierarchy all need far more than 1,000 examples. Production post-training sets now run to millions of examples plus RL environments.
• Small data also elicits reasoning. In the same spirit as LIMA, LIMO showed a few hundred curated long-reasoning examples can unlock strong math reasoning in a capable base model. Ye+ 2025 The lesson isn't "data doesn't matter". It is "the base model sets the ceiling, and post-training decides how much of it you reach".

The modern post-training pipeline, end to end

The 2022 InstructGPT recipe was three steps: (1) SFT on human demonstrations, (2) train a reward model on human rankings, (3) optimize the policy against the RM with PPO. Today's recipes keep the skeleton but differ in four main ways. Preference learning is often done with DPO-family losses instead of PPO. A verifiable-reward RL stage targets math, code and instruction-following constraints. Much of the data is synthetic (generated by the model itself or a stronger teacher, then filtered). And the whole thing runs in iterative rounds.

Published recipes worth knowing

Why iterative rounds?

Preference and RL data go stale. A reward model trained on outputs of model v1 is less accurate on outputs of v2, whose distribution has shifted (and v2 may have found the RM's blind spots). Each round, labs therefore (a) sample fresh responses from the current best model, (b) collect new preferences or judge scores on those, (c) retrain or refresh the RM, and (d) run SFT/DPO/RL again. Rejection sampling (best-of-N according to an RM or verifier, then SFT on the winners) is the cheapest way to turn a better RM into a better policy, which is why Llama 2/3 lean on it. This is also why the "offline vs online" distinction blurs: iterative DPO with fresh on-policy samples each round is "semi-online".

Supervised fine-tuning (SFT)

SFT is ordinary next-token prediction on curated (prompt, response) conversations. Three engineering details trip people up: the chat template, loss masking and packing. Above all of them sits the question of data.

Chat templates

A chat model sees a conversation serialized into a single token stream with special tokens marking roles and turn boundaries. The exact template is model-specific, and using the wrong one at inference silently degrades quality. Libraries ship templates with the tokenizer (for example Jinja templates in Hugging Face tokenizer.apply_chat_template). HF docs A schematic ChatML-style example:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What's 17 * 23?<|im_end|>
<|im_start|>assistant
17 * 23 = 391.<|im_end|>

The end-of-turn token matters: the model must learn to emit it, or generation never stops. A common bug is a template that never puts the EOS/end-of-turn token in the loss.

Loss masking on prompt tokens

The SFT loss is computed only on assistant tokens:

System and user tokens are still in the context (the model attends to them) but get label -100 (ignored). Why: (1) you want the model to learn to respond, not to imitate users, (2) prompt tokens are often noisy, repetitive or adversarial, (3) without masking, long prompts dominate the gradient. In multi-turn data you typically train on every assistant turn, or only the final one if earlier turns came from a weaker model. Some work finds training on prompts slightly helps when responses are very short, so it is a tunable choice, but masking is the default.

Packing

SFT examples vary a lot in length. Padding each batch to the longest example wastes compute (often 30–50% of tokens on short-heavy chat data). Packing concatenates several examples into one fixed-length sequence. Done naively, tokens in example B can attend to example A ("cross-contamination"). The correct approach resets position IDs per example and uses a block-diagonal attention mask, implemented efficiently with variable-length attention kernels (for example FlashAttention's varlen interface). Krell+ 2021 Note that packing changes the effective per-example loss weighting: summing over tokens across a packed batch weights long examples more than a per-example mean would. Some frameworks therefore normalize per example.

Data quality over quantity

• Quality and diversity dominate. LIMA's 1,000 examples were hand-picked for diversity and a consistent "helpful assistant" style. Thousands of near-duplicates add little. Deduplicate (exact plus near-duplicate via MinHash or embeddings) and cluster prompts to balance topics.
• Consistency of style. Mixed styles (some terse, some verbose, some with markdown) produce an inconsistent model. Many teams rewrite all responses into one house style.
• Correctness. SFT faithfully imitates errors. One wrong-but-confident answer pattern, repeated, teaches hallucination. Verify math and code answers by execution where possible.
• Don't teach knowledge the model lacks. Fine-tuning on facts the base model doesn't know can increase hallucination: the model learns to produce confident answers it can't ground. This is one reason SFT data is often generated by the model itself plus filtering rather than written by external experts.
• Decontamination. Remove any prompt that overlaps with evaluation sets (n-gram overlap). Tülu 3 found and removed contamination in several popular public datasets. Lambert+ 2024

Synthetic data generation

Most SFT data is now synthetic. The main patterns:

SFT hyperparameters and magnitudes

Typical full fine-tuning SFT uses a learning rate around 1e-5 to 2e-5 for 7–70B models (much lower than pretraining peak LR), 1–3 epochs (more epochs overfit and hurt diversity), cosine or linear decay with short warmup, and global batch sizes of tens to a few hundred sequences. LoRA uses learning rates roughly 10× higher (see below). Watch for loss dropping very low on the train set, which usually means memorization of a small dataset rather than better behaviour.

Reward models

A reward model (RM) turns human (or AI) judgements into a differentiable scalar you can optimize. The standard architecture is the SFT model with its LM head replaced by a linear head that outputs one number \(r_\phi(x,y)\), read at the last token.

Pairwise data and the Bradley–Terry loss

Humans are bad at absolute scores ("rate this 1–10") and much more consistent at comparisons ("which is better?"). So data is collected as pairs: for a prompt \(x\), show two (or more) responses and record which is preferred, \(y_w \succ y_l\). The Bradley–Terry model assumes each response has a latent quality score and

Training maximizes the likelihood of the observed preferences:

Only differences matter, so the RM is defined up to a per-prompt constant (people often normalize rewards before RL). With \(K\) ranked responses you get \(\binom{K}{2}\) pairs per prompt. InstructGPT ranked 4–9 responses and batched all pairs from one prompt together to avoid overfitting. Ouyang+ 2022 Llama 2 added a margin \(m(r)\) inside the sigmoid for "significantly better" pairs. Touvron+ 2023 RMs typically reach only ~65–75% agreement with held-out human preferences on hard chat data. Inter-annotator agreement itself is often in the same range, so preference data is noisy by nature.

Collecting preference data in practice

• Sampling responses: use the current policy (on-policy) and a mix of other models for diversity. Vary temperature. Responses that are too similar give low-information pairs.
• Guidelines: annotators need a written rubric (helpfulness, honesty, harmlessness and their priority order). Most disagreement comes from unclear guidelines, not careless raters.
• Multi-attribute: separate RMs or heads for helpfulness and safety (Llama 2) avoid a single scalar that has to trade them off implicitly.
• Cost: expert comparisons cost dollars each. AI feedback (see RLAIF) costs fractions of a cent, which is why most open preference sets (for example UltraFeedback-style) are LLM-judged.

Reward overoptimization (Goodhart's law)

The RM is a proxy fitted on a finite, off-policy dataset. A policy optimized hard against it finds inputs where the proxy is wrong: verbosity, sycophancy, confident tone, lists and headers, flattering the user, or outright gibberish that the RM happens to score highly. Gao et al. measured this with a synthetic setup: a large "gold" RM labels data, a smaller proxy RM is trained on those labels, and a policy is optimized against the proxy. As KL distance from the initial policy grows, proxy reward keeps rising while gold reward rises, peaks, then falls. Gao+ 2022 They fit functional forms in \(d=\sqrt{\mathrm{KL}(\pi\|\pi_\text{init})}\): roughly \(R_\text{bon}(d)=d(\alpha-\beta d)\) for best-of-N and \(R_\text{RL}(d)=d(\alpha-\beta\log d)\) for RL. Larger RMs and more RM data push the peak further out.

A well-documented specific case is length bias. RMs tend to prefer longer answers, and a large share of measured RLHF "improvement" can come from length alone. Singhal+ 2023 This is why AlpacaEval introduced a length-controlled win rate. Dubois+ 2024 Mitigations: KL penalty, RM ensembles, length penalties or normalization, refreshing the RM on on-policy samples, early stopping on a held-out gold metric, and constraining the output format.

Outcome vs process reward models

In practice PRMs have proven most useful for search and reranking at inference time. For RL training at scale, DeepSeek-R1 reported PRMs were hard to define, label and protect from reward hacking, and used outcome rewards instead. DeepSeek-AI 2025 Work on PRM pitfalls (for example noisy Monte Carlo labels) continues. Zhang+ 2025 Implicit PRMs, which derive step rewards from an outcome-trained model, are an active direction. Cui+ 2025

Generative reward models and LLM-as-judge

Instead of a scalar head, ask an LLM to judge in natural language: "Which response is better and why?", or "Is this solution correct? Answer yes/no", and use the probability of "yes" as the reward. Generative verifiers trained with next-token prediction (optionally reasoning with CoT before the verdict) outperform discriminative RMs on reasoning tasks and can spend more inference compute to judge better. Zhang+ 2024 Liu+ 2025 LLM judges correlate well with human preference on chat (MT-Bench reported >80% agreement with humans for GPT-4, similar to human–human agreement) but have known biases: position, verbosity and self-preference. Zheng+ 2023 A 2025 trend extends RL beyond verifiable domains by having a judge score responses against instance-specific rubrics ("rubrics as rewards"). Gunjal+ 2025 OpenAI described "rule-based rewards" for safety behaviour: an LLM grades responses against explicit propositions ("refuses without judgemental language") instead of holistic preference. Mu+ 2024

RLHF with PPO

The KL-regularized objective

Everything in preference optimization starts from this objective:

\(\pi_\text{ref}\) is the frozen SFT model. The KL term does three jobs. It keeps the policy where the RM is accurate, which limits overoptimization. It preserves the fluency and diversity of the SFT model, which prevents mode collapse onto one high-reward template. And it is a regularizer against forgetting. \(\beta\) sets the trade-off: small \(\beta\) means more reward and more hacking, large \(\beta\) means the policy barely moves. Typical values are around 0.01–0.1, sometimes adapted to hit a target KL.

In implementation the KL is applied per token as a reward-shaping term. The RM score arrives at the last token:

Generation is treated as an episodic MDP: state = prompt + tokens so far, action = next token, episode ends at EOS.

PPO in one paragraph

PPO is a policy-gradient method. Its gradient pushes up the log-probability of actions with positive advantage \(A_t\) (better than expected) and down for negative ones. To reuse a batch of samples for several gradient steps without the policy drifting too far, it clips the importance ratio \(\rho_t=\pi_\theta/\pi_{\theta_\text{old}}\): Schulman+ 2017

Advantages come from a learned value function (critic) \(V_\psi(s_t)\) via Generalized Advantage Estimation (GAE), which blends multi-step returns to balance bias and variance. The critic is usually initialized from the RM or SFT model and trained with a regression loss on returns.

Why it's expensive and unstable

• Memory: four model copies. For a 7B model with mixed-precision Adam (~16 bytes/param for weights + grads + optimizer state), actor ≈ 112 GB and critic ≈ 112 GB, plus two frozen bf16 copies (≈14 GB each). That is roughly 250 GB before activations and the KV cache for generation, so even a 7B run needs multi-GPU sharding. Tricks: LoRA adapters on a shared base (the reference is "base with adapters off"), sharing actor/critic trunks, offloading the frozen models.
• Generation is the bottleneck: autoregressive sampling of long responses dominates wall-clock time. Modern systems run a separate inference engine (vLLM/SGLang-style) for rollouts and sync weights to the trainer. That brings off-policy lag and extra engineering (see A5).
• Many interacting hyperparameters: β/KL target, clip ε, GAE λ and γ, value-loss coefficient, LR for actor and critic, number of PPO epochs, batch and minibatch sizes, reward normalization and whitening, sampling temperature. Small implementation details (advantage normalization, reward clipping, EOS handling) matter a lot.
• Critic difficulty: predicting expected final reward from partial text is hard. A bad critic gives noisy advantages, and noisy advantages give unstable updates.
• Reward hacking: the online loop actively searches for RM exploits. KL can collapse (policy barely moves) or explode (policy diverges into degenerate text).

Direct Preference Optimization (DPO)

DPO removes the reward model and the RL loop entirely. Rafailov+ 2023 The derivation is short and a favourite interview question.

Derivation intuition (three steps)

Step 1 is a standard result: maximizing \(\mathbb{E}_\pi[r] - \beta\mathrm{KL}(\pi\|\pi_\text{ref})\) over distributions gives a Gibbs/Boltzmann reweighting of the reference. Z(x) is intractable (a sum over all possible sequences), which is why you can't just compute \(\pi^*\). The trick in step 3 is that you never need it. The resulting loss:

The quantity \(\hat r_\theta(x,y)=\beta\log\frac{\pi_\theta(y|x)}{\pi_\text{ref}(y|x)}\) is the implicit reward. That is the paper's subtitle: "your language model is secretly a reward model". The log-probability of a sequence is just the sum of token log-probs, so the loss needs four forward passes per pair (policy and reference on chosen and rejected). The reference log-probs can be precomputed once.

What the gradient does

It raises the likelihood of chosen and lowers rejected, weighted by how badly the implicit reward currently mis-orders the pair. Pairs already ranked correctly by a wide margin contribute almost nothing. That weighting is what stops it degenerating like naive "unlikelihood" training.

The role of β

β is the same KL-strength parameter as in RLHF. Small β (for example 0.01–0.05) lets the policy move far from the reference and fit preferences aggressively, with more risk of overfitting and verbosity. Large β (0.3–0.5) keeps it close. β = 0.1 is the common default. In DPO, β also scales the margin at which the sigmoid saturates, so it interacts with learning rate. DPO learning rates are small (around 5e-7 to 5e-6 for full fine-tuning), and 1–3 epochs is typical. It overfits quickly.

Known failure modes

• Both likelihoods go down. DPO only constrains the margin. Commonly the log-prob of the chosen response also decreases, just less than the rejected one, and probability mass moves to responses that are in neither. Remedies: add an SFT/NLL term on chosen responses (as in RPO-style and Llama 3 variants), or use a higher β.
• Length exploitation. Chosen responses are often longer, and DPO learns "longer = better". Length-regularized DPO explicitly penalizes this. Park+ 2024
• Off-policy mismatch. If the preference pairs come from other models, the policy is learning to rank text it would never generate. The implicit reward generalizes poorly to its own outputs. This is the core argument for online/iterative variants.
• Overfitting with deterministic preferences. If \(y_w\) always beats \(y_l\), the BT likelihood is maximized by pushing the margin to infinity. The KL regularization effectively disappears because there is no explicit KL term in the loss. This motivates IPO.

Variants

Other names you may hear: SPIN (self-play against your previous SFT outputs) Chen+ 2024, XPO (exploration bonus for online DPO) Xie+ 2024, and Zephyr's "distilled DPO" (dDPO), where an open 7B model was aligned with DPO on GPT-4-judged UltraFeedback pairs. Tunstall+ 2023

Offline vs online preference learning

This was one of the most-discussed questions of 2024. The evidence has converged reasonably well:

• A careful study found that well-tuned PPO beats DPO across dialogue and code tasks. It identified the key PPO ingredients (advantage normalization, large batch, EMA reference update) and showed DPO suffers from distribution shift when preference data doesn't match policy outputs. Xu+ 2024
• DeepMind compared online and offline methods under controlled KL budgets and found online consistently better. Offline data coverage and quality alone didn't explain the gap. On-policy sampling itself matters. Tang+ 2024
• Meanwhile, offline DPO remains the workhorse because it is cheap, stable, needs no generation infrastructure and is easy to iterate on data. Llama 3, Tülu 3 and OLMo 3 all use it, followed (in Tülu, OLMo and Qwen) by online RL.

RLAIF and Constitutional AI

Human preference labels are slow, expensive, inconsistent and hard to scale to expert domains. RLAIF replaces human labellers with an LLM judge. On summarization and dialogue tasks, Google found that RLAIF reached quality comparable to RLHF as rated by humans. "Direct RLAIF", which asks the LLM for a score during RL instead of training an RM, also worked. Lee+ 2023

Constitutional AI (Anthropic) uses a short list of natural-language principles (a "constitution") to drive harmlessness training with little human harm labelling: Bai+ 2022

1. Supervised phase (SL-CAI): sample responses to red-team prompts from a helpful-only model. Ask the model to critique its response against a randomly drawn principle, then revise it. Repeat. Fine-tune on the revised responses.
2. RL phase (RL-CAI): sample pairs of responses. Ask a model which better satisfies a principle (optionally with chain of thought). Train a preference model on these AI labels (mixed with human helpfulness labels). Run RL against it.

Key outcome: a model that is both less harmful and less evasive. It explains its objections instead of giving flat refusals. The broader idea, specifying behaviour with written principles and having models apply them, now runs through industry practice: OpenAI's rule-based rewards and deliberative alignment (models reason over a written safety specification in their chain of thought), Guan+ 2024 and Anthropic's constitutional classifiers for jailbreak defence. Sharma+ 2025

GRPO and RL with verifiable rewards (brief)

For tasks with checkable answers (math with a known final answer, code with unit tests, instruction-following with programmatic constraints such as "answer in under 50 words, in JSON"), you can skip the learned RM and use a verifier: reward 1 if correct, else 0, plus maybe a format reward. Verifiable rewards are much harder to hack than learned RMs, though not impossible (models do learn to game weak unit tests).

GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath, removes PPO's critic. Shao+ 2024 For each prompt, sample a group of \(G\) responses (for example 8–64), score them, and use the group-normalized reward as every token's advantage:

It then applies a PPO-style clipped objective with a KL penalty to the reference (added to the loss, not the reward). The "baseline" is the group mean, a Monte Carlo estimate of the value of the prompt. You save the critic's memory and avoid its instability. You pay with more samples per prompt, and you get no learning signal from prompts where all \(G\) responses score the same (all right or all wrong). Follow-ups such as Dr. GRPO identify biases from the std and length normalization Liu+ 2025, and DAPO adds tricks like decoupled clipping and dynamic sampling. Yu+ 2025 Full treatment, including rollout infrastructure, credit assignment and RL scaling, is in A5: RL for LLMs.

Comparison: SFT vs PPO vs DPO vs KTO vs GRPO

Parameter-efficient fine-tuning: LoRA, QLoRA, DoRA

LoRA: the math

Full fine-tuning updates every weight matrix \(W\in\mathbb{R}^{d\times k}\). LoRA's hypothesis is that the update \(\Delta W\) needed for adaptation has low intrinsic rank. So freeze \(W_0\) and learn a low-rank factorization: Hu+ 2021

• Initialization: \(A\) random Gaussian, \(B = 0\), so \(\Delta W=0\) at step 0 and training starts exactly at the pretrained model.
• Parameter count: \(r(d+k)\) instead of \(dk\). For a 4096×4096 projection with \(r=16\): \(16\times 8192 = 131\text{k}\) vs \(16.8\text{M}\), about 0.8%. Across a 7B model with LoRA on all linear layers at \(r=16\), that is roughly 40M trainable parameters (about 0.5%).
• α and the α/r scaling: α is a constant that scales the update. Dividing by r means that if you change rank, the effective update magnitude (and thus the right learning rate) stays roughly stable. Common heuristics are α = r or α = 2r. rsLoRA argues the scale should be \(\alpha/\sqrt{r}\) for stable learning at high ranks. Kalajdzievski 2023
• Memory savings: no gradients or Adam states for frozen weights. On GPT-3 175B the LoRA paper reported about 10,000× fewer trainable parameters and about 3× lower GPU memory than full fine-tuning. Hu+ 2021 The frozen weights still have to be held (in bf16, or 4-bit for QLoRA), and activations still dominate at long sequence lengths.
• Merging: after training, compute \(W = W_0 + \tfrac{\alpha}{r}BA\) and you have a plain dense model with zero extra inference latency. Or keep adapters separate and hot-swap many per-customer adapters on one base model (multi-LoRA serving, supported by engines such as vLLM).

Which modules, which rank?

The original paper adapted only the attention \(W_q, W_v\) projections. Later evidence says to apply LoRA to all linear layers, especially the MLP. Thinking Machines' "LoRA Without Regret" study found attention-only LoRA clearly underperforms MLP-only LoRA at matched parameter counts. With LoRA on all layers it matched full fine-tuning sample efficiency on small-to-medium post-training datasets, as long as the dataset didn't exceed the adapter's capacity. Their optimal LoRA learning rate was consistently about 10× the full-FT learning rate. For RL, even rank 1 matched full fine-tuning, because policy-gradient feedback carries very few bits per episode. They also found LoRA tolerates very large batch sizes less well than full FT. Schulman+ 2025 (Thinking Machines) Typical ranks: 8–64 for SFT and style adaptation, higher (128–256+) for learning substantial new skills or domains.

QLoRA

QLoRA keeps the frozen base in 4-bit and trains LoRA adapters in bf16, which made fine-tuning a 65B model possible on a single 48 GB GPU. Dettmers+ 2023 Three ingredients:

1. NF4 (4-bit NormalFloat): pretrained weights are roughly zero-centred normal. NF4 places its 16 quantization levels at quantiles of a standard normal, so each bin holds about the same number of weights. The paper calls this information-theoretically optimal for normally distributed data. Weights are quantized in blocks (block size 64) with a per-block absmax scale.
2. Double quantization: the per-block scales themselves (32-bit, one per 64 weights, about 0.5 bits/param) are quantized to 8-bit, saving roughly 0.37 bits/param (~3 GB on a 65B model).
3. Paged optimizers: use NVIDIA unified memory so optimizer states page to CPU RAM during memory spikes (long sequences), avoiding OOM crashes.

In the forward pass, each 4-bit block is dequantized to bf16 on the fly for the matmul. Gradients flow through the frozen quantized weights into the adapters. The costs are some extra compute from dequantization (QLoRA training is often ~20–40% slower than bf16 LoRA) and a small quality loss. You also usually merge the adapter into a bf16 (not 4-bit) copy of the base for deployment, or serve base + adapter.

DoRA and other variants

Full fine-tuning vs LoRA: what LoRA "learns less and forgets less"

Biderman et al. compared LoRA and full FT on code and math, in both continued pretraining and instruction tuning. LoRA underperformed full FT on the target domain, especially for continued pretraining on large datasets. But it better preserved the base model's performance outside the target domain and kept more output diversity. They also found that full FT learns weight perturbations with rank 10–100× higher than typical LoRA configurations. Biderman+ 2024 A related study found LoRA and full FT can reach similar accuracy yet differ structurally: LoRA introduces "intruder dimensions" (new high-ranking singular vectors) that correlate with worse generalization beyond the training task, especially in continual learning. Shuttleworth+ 2024

Distillation

Distillation trains a smaller student to mimic a larger teacher. It is the main reason small open models are good, and it is now built into flagship pipelines (Qwen3's strong-to-weak distillation, DeepSeek-R1's distilled models).

Reasoning distillation

DeepSeek fine-tuned Qwen2.5 and Llama 3 models (1.5B–70B) with SFT only, on reasoning traces from R1 (~800k samples). The distilled models beat what the same small models achieved with large-scale RL of their own. DeepSeek-AI 2025 The takeaway, widely replicated since: for small models, distilling a strong reasoner beats doing RL from scratch. RL is how you push the frontier with the big model, and distillation is how you spread the result to smaller models. Caveats: teacher terms of service may forbid training on outputs. Distilled models can inherit a teacher's verbosity and its failure modes. And SFT on long CoT can teach the form of reasoning (hedging, "wait, let me re-check") without full robustness.

Model merging

Merging combines the weights of several fine-tuned models that share the same base, with no further training. It works surprisingly well because fine-tunes from the same pretrained initialization tend to stay in a shared, linearly connected low-loss basin. Model soups showed that averaging several fine-tunes of one model improves accuracy and robustness at no inference cost. Wortsman+ 2022 Llama 3 averaged models across runs and hyperparameter settings at each post-training stage. Llama Team 2024

Catastrophic forgetting and regularization

Fine-tuning on a narrow distribution moves weights away from what made the base model general, so capabilities that aren't exercised degrade: other languages, coding, calibration, safety, even instruction following. This is "alignment tax" when it happens during RLHF and "catastrophic forgetting" in general. InstructGPT observed regressions on public NLP benchmarks after RLHF and mitigated them by mixing pretraining gradients into PPO ("PPO-ptx"). Ouyang+ 2022

Safety training

Safety post-training teaches a model to decline genuinely harmful requests (weapons uplift, CSAM, targeted harassment), to handle sensitive-but-legitimate ones carefully (medical, security research), to resist manipulation, and to respect an instruction hierarchy (system > developer > user > tool outputs). Wallace+ 2024

How it's done

• Data: red-teaming prompts (human red teamers, plus automated attack generation such as PAIR Chao+ 2023), paired with ideal responses: refusals that are helpful about why and offer safe alternatives, or safe completions.
• Training signals: SFT on safe responses, a safety RM or safety head (Llama 2), constitution- or rule-based AI feedback (CAI, RBRs), and deliberative alignment, where the model learns to reason explicitly over a written policy before answering. Guan+ 2024
• Borderline data: it is just as important to include benign prompts that look dangerous ("how do I kill a Python process?") with helpful answers.

Over-refusal

Safety training pushes the model to refuse based on surface features. XSTest is a suite of 250 safe prompts that superficially resemble unsafe ones (plus unsafe contrast prompts). It exposed significant exaggerated refusals in safety-tuned models of the time. Röttger+ 2023 Helpfulness and harmlessness are in tension. Labs measure both and tune the trade-off (separate RMs, rule-based rewards that penalize unnecessary refusals, borderline data).

Jailbreaks and robustness

• Attack families: role-play and persona prompts, many-shot prompting, encodings and translations, multi-turn escalation, prompt injection through tools and documents, and optimized adversarial suffixes. GCG found gibberish suffixes via gradient search that transferred across models, including closed ones. Zou+ 2023
• Safety is shallow: much of alignment lives in the first few output tokens (the refusal prefix). Prefilling or forcing a compliant start often bypasses it. Training on "recovery" examples, where the model returns to a refusal after a harmful start, deepens it. Qi+ 2024
• Fine-tuning removes safety: fine-tuning an aligned model on a handful of harmful examples (or even benign data) can strip safety behaviour. Qi et al. jailbroke GPT-3.5 via the fine-tuning API with 10 examples for under $0.20. Qi+ 2023 Narrow fine-tuning (for example on insecure code) can even produce broadly misaligned behaviour ("emergent misalignment"). Betley+ 2025 This matters for anyone offering fine-tuning or releasing open weights.
• Defence in depth: beyond training the model, deploy input and output classifiers. Anthropic's constitutional classifiers, trained on synthetic data generated from a constitution, withstood thousands of hours of red teaming against universal jailbreaks with modest over-refusal cost. Sharma+ 2025 Representation-level methods such as circuit breakers reroute harmful internal representations. Zou+ 2024

Evaluating chat models (pointer)

Post-training is only as good as its evals. The short version (full treatment in A8):

• Capability benchmarks (math, code, knowledge, instruction following with verifiable constraints such as IFEval) catch regressions from tuning.
• Preference-style evals: pairwise human votes (Chatbot Arena), LLM-judge win rates (MT-Bench, AlpacaEval with length control, Arena-Hard). Watch for judge biases and length gaming. Zheng+ 2023 Dubois+ 2024
• Safety evals: attack success rate on red-team sets, over-refusal (XSTest), and jailbreak robustness.
• Product evals: task-specific held-out sets built before training, plus online A/B tests. Always decontaminate training data against evals.

Interview question bank