Model Evaluation & Interpretability

Two ways to answer "what does this model actually do?" Evaluation looks from the outside: what you measure, how you score it, and how much to trust the number. Interpretability looks from the inside: which internal representations and circuits produce the behaviour. This page treats evaluation the way a model builder sees it (pretraining and post-training checkpoints, public benchmarks, leaderboards, capability and safety evals). Product evals for LLM applications (task rubrics, golden sets, online metrics) are covered separately in B5. Interviewers use this area to tell apart people who read leaderboards from people who can say why a 2-point gain on a 500-question benchmark may be noise, why a judge model prefers longer answers, or what a sparse autoencoder is optimizing.

Part 1 · Model evaluation: why it's harder than it looks

Evaluating a classifier on ImageNet was a solved social problem: one fixed test set, one metric, one protocol. LLM evaluation breaks each of those assumptions:

• Open-ended outputs. A model can be correct in many surface forms, so scoring needs answer extraction, execution (code), or a judge. Each of these adds error.
• Many protocol degrees of freedom. System prompt, few-shot examples, chat template, sampling temperature, max tokens, reasoning effort, tool access and scaffolding all change results. A "score" really belongs to a (model, protocol) pair.
• Training data is the internet. Public test sets end up in pretraining corpora (contamination), and labs tune hyperparameters, data mixes and RL environments against the benchmarks they report (Goodhart).
• Capabilities emerge and shift. Benchmarks saturate fast. The interesting capabilities move toward long-horizon agentic tasks that are expensive to run and noisy to grade.
• Small, correlated test sets. Many headline benchmarks have a few hundred items (GPQA Diamond about 198, AIME 30 per year, SWE-bench Verified 500), so naive comparisons are swamped by noise.

A model builder runs evals at several points, each with its own goal:

Perplexity and its limits

For a held-out token sequence \(x_1,\dots,x_N\), the average negative log-likelihood (cross-entropy, in nats) is

Intuitively, perplexity is the effective branching factor: PPL = 10 means the model is, on average, as uncertain as if choosing uniformly among 10 tokens. It's the pretraining objective itself, so it's smooth, cheap, low-variance, and it is what scaling laws are fit to (see the scaling-laws page).

Why you can't stop at perplexity

• Tokenizer dependence. PPL is per token. A model with a larger vocabulary predicts fewer, "bigger" tokens, so per-token PPL isn't comparable across tokenizers. Normalize to bits per byte (or per character): \(\mathrm{BPB} = \frac{N_\text{tokens}}{N_\text{bytes}}\cdot\frac{\mathcal{L}}{\ln 2}\). This is what tokenizer-agnostic comparisons use.
• Distribution dependence. PPL on web text says little about PPL on code or math. Report it per domain. A data-mix change can improve one domain and hurt another while the average barely moves.
• Weak link to task quality. Most of the likelihood mass is on "easy" tokens (function words, boilerplate). A model can gain on PPL by getting better at predictable tokens while getting no better at the rare, decisive tokens of a reasoning chain. Downstream accuracy often looks like a sharp function of loss, and much of that sharpness comes from discontinuous metrics such as exact match. Schaeffer et al. showed that many "emergent abilities" turn smooth when you switch to continuous metrics like token edit distance or log-prob of the correct answer. Schaeffer+ 2023
• Post-training breaks it. RLHF and RL on verifiable rewards deliberately move the model away from the pretraining distribution. Chat models become low-entropy and are often worse calibrated, so their PPL on generic text gets worse even as they become far more useful. Perplexity is a pretraining metric.
• Context-length artefacts. PPL on long documents mostly reflects local prediction. A model can have excellent long-context PPL and still fail to retrieve a fact from 100k tokens back, which is why RULER-style tasks exist.
• Contamination inflates it silently. Memorized documents have very low PPL. Turned around, this becomes a contamination detector (Min-K% Prob, below).

The benchmark landscape (as of late 2026)

The table groups widely cited benchmarks by what they claim to measure. "Status" is a rough judgement of whether the benchmark still separates frontier models. Frontier scores move monthly, so treat the numbers as indicative.

Notes on specific figures: the GPQA expert baseline and OSWorld human baseline are approximate values from the respective papers. HLE frontier ranges are taken from third-party trackers, which disagree with each other.

Metrics: accuracy, pass@k, maj@k, pass^k, Elo

Accuracy and its variants

For multiple-choice or exact-answer tasks the score is just the mean of per-item 0/1 correctness. The details that matter: how you extract the answer (regex over the last "Answer: X", \boxed{} parsing, symbolic equivalence via sympy for math), how you handle refusals or unparseable outputs (count them as wrong, and report the parse-failure rate), and whether you report pass@1 averaged over several samples (often written avg@k, mean@k or pass@1 with n samples). For small sets like AIME, averaging over 8–64 samples per problem reduces sampling variance. It does not reduce the variance that comes from having only 30 problems.

pass@k and the unbiased estimator

pass@k is the probability that at least one of k independent samples is correct. It models a setting where a verifier, such as unit tests, can pick the good sample. The naive approach (draw exactly k samples, check if any pass) has high variance. Codex instead draws \(n \ge k\) samples per problem, counts the \(c\) correct ones, and uses the unbiased estimator Chen+ 2021:

Read it as "1 minus the probability that a random size-k subset of the n samples contains no correct one". Computing the binomials directly overflows, so use the stable product form \(1-\prod_{i=n-c+1}^{n}\left(1-\frac{k}{i}\right)\) (and return 1 if \(n-c \lt k\)). Codex used n = 200 for k up to 100.

Why not just compute \(1-(1-\hat p)^k\) with \(\hat p = c/n\)? Because it's biased: \(1-(1-p)^k\) is concave in \(p\), so plugging in a noisy \(\hat p\) systematically underestimates the true value (Jensen's inequality). The combinatorial form is exactly unbiased for \(k \le n\).

Practical notes on pass@k: temperature matters (higher T helps large-k, hurts k=1, so the Codex paper tuned T per k); pass@k with large k is only meaningful when a verifier exists at deployment; and it's increasingly used as a probe for "latent capability": if pass@256 is high but pass@1 is low, RL may be able to surface the capability. Whether RL with verifiable rewards expands pass@k at large k or only sharpens pass@1 is an active and contested research question as of 2025–2026.

pass^k (reliability)

τ-bench introduced pass^k: the probability that all k independent trials succeed. With per-task success rate \(p_t\) it is \(\mathbb{E}_t[p_t^{\,k}]\), estimated unbiasedly as \(\binom{c}{k}/\binom{n}{k}\) Yao+ 2024. A model with 60% pass@1 that's consistent per task (some tasks 100%, others 0%) keeps a high pass^8. One that succeeds on every task 60% of the time collapses to 0.6⁸ ≈ 1.7%. Customer-facing agents need pass^k, not pass@k.

maj@k (self-consistency)

maj@k samples k reasoning chains, extracts each final answer, and outputs the majority vote Wang+ 2022. Unlike pass@k it produces one answer with no oracle, so it's a fair deployable comparison. It needs answers that can be compared for equality (numbers, choices), and it plateaus once the most common wrong answer dominates. Variants weight the votes by a reward model or by confidence (best-of-N with an RM). When a lab reports "maj@64", it is spending 64× inference compute, so compare at matched compute.

Elo and Bradley–Terry for arenas

Arena platforms collect pairwise votes. The Bradley–Terry (BT) model assigns each model a strength \(\beta_m\) and assumes

The strengths are fit by maximum likelihood, which is just logistic regression where each battle is a row with +1 for A's column and −1 for B's. They're reported on an Elo-like scale, \(R = 400\,\beta/\ln 10 + \text{const}\), so that \(P(A\succ B)=1/(1+10^{(R_B-R_A)/400})\). A 100-point gap is about a 64% win rate, and 200 points about 76%. Chatbot Arena moved from online Elo updates, which depend on the order of games, to batch BT fitting with bootstrap confidence intervals Chiang+ 2024. Ties are handled by counting half wins or with a Rao–Kupper/Davidson extension.

Known issues with arenas:

• Style confounds. Longer, markdown-heavy, more confident answers win votes. LMArena added "style control", which adds response length and markdown counts as covariates in the BT regression so that the β coefficients are style-adjusted.
• Selective disclosure and sampling asymmetries. Providers testing many private variants and publishing only the best one inflates scores through a winner's curse. Proprietary models also got more battles. Singh+ 2025 reported, for example, 27 private Meta variants tested before Llama 4.
• Prompt distribution. Votes reflect what arena users ask (lots of coding and casual chat), not your users.
• Non-transitivity. BT assumes a single "strength" dimension. Real preferences vary by category, hence category leaderboards.

Scoring protocols: shots, prompts, log-likelihood vs generation

Zero-shot vs few-shot

Few-shot prompting puts k solved examples in the context. For base models it's essential: it tells the model the format and the task, and 5-shot MMLU was the standard. For chat/instruct models, zero-shot with an instruction (often plus "think step by step" or native reasoning) is now the norm. Few-shot exemplars can even hurt reasoning models by constraining their reasoning format. Results are not comparable across shot settings, and the choice and order of exemplars add variance on their own.

Prompt sensitivity

Meaning-preserving formatting changes (separators, casing, spacing, "Answer:" vs "A:") swung LLaMA-2-13B few-shot accuracy by up to 76 points on some tasks, and the best format barely correlated across models Sclar+ 2023. Frontier instruction-tuned models are much more robust, but sensitivity of a few points is still common, which is the same size as many claimed improvements. GSM-Symbolic showed that changing names and numbers in GSM8K templates, or adding irrelevant clauses, reduces accuracy and increases variance, a sign of pattern-matching rather than robust reasoning Mirzadeh+ 2024. Good practice is to report the mean and spread over several prompt variants, and to use the model's own chat template.

Log-likelihood (cloze/MC) vs generative scoring

The same model can rank differently under the two schemes. Small models often look much better under cloze log-likelihood, because the letter-MC format is itself a capability that emerges only at scale. The lm-eval-harness authors document these choices in detail Biderman+ 2024.

Contamination

How it happens

• Direct inclusion. Test sets sit on GitHub, Hugging Face or in papers, and web crawls ingest them, sometimes with answers.
• Indirect leakage. Blog posts, forum discussions, solution write-ups and translated or paraphrased versions. Exact-match filters miss these.
• Synthetic data echoes. A teacher model that saw the test set generates training data resembling it.
• Post-training and RL environments. Tuning on problem sets drawn from the same sources (competition archives, GitHub issues) as the benchmark. Repository-based benchmarks are especially exposed: SWE-bench tasks come from public repos with public fix commits.
• Adaptive overfitting without leakage. Choosing checkpoints, data mixes and hyperparameters because they score well on the benchmark. No test item is ever trained on, but the reported number is still optimistically biased.

Detection methods

Mitigations as a benchmark designer

Keep a private held-out split and evaluate through a server (SWE-bench Pro's held-out and commercial splits, ARC-AGI private sets, FrontierMath). Add canary strings (a unique GUID in every file so that crawlers and labs can filter it, as BIG-bench did), keep data gated behind terms of use, refresh items continuously, and prefer tasks where memorizing the answer doesn't help (interactive environments, procedurally generated tasks).

Saturation, the benchmark lifecycle and Goodhart

"When a measure becomes a target, it ceases to be a good measure" (Goodhart's law, in Strathern's popular phrasing). Benchmarks follow a predictable arc:

Signs that a benchmark is saturated: (1) frontier scores are within a few points of the label-noise ceiling (for example, MMLU's estimated 6.5% error rate means about 93% is a soft ceiling); (2) score gains no longer predict gains on fresh or adjacent tasks; (3) top-model differences fall inside the confidence intervals; (4) evidence of memorization appears. The time from release to saturation has shrunk from years to months for some recent benchmarks, which pushes the field toward private, live and agentic evals that are costly to game.

Goodhart failure modes specific to LLMs:

• Teaching to the format. Training on benchmark-style multiple-choice data inflates MC scores without improving the underlying knowledge.
• Reward hacking in agentic evals. Agents that edit tests, special-case checkers or exploit grader loopholes. Graders need adversarial review, and agents need sandboxing that stops them from touching tests.
• Judge hacking. Optimizing against an LLM judge (or arena votes) rewards length, confidence, flattery and formatting.
• Leaderboard selection. Testing many variants and reporting the max (the winner's curse).

LLM-as-judge: biases and mitigations

For open-ended outputs, a strong model grades either a single response (pointwise, for example on a 1–10 rubric) or a pair (pairwise preference). MT-Bench showed that GPT-4 judgments agreed with human experts roughly as often as humans agreed with each other (above 80% on non-tie pairs), which legitimized the approach. The same paper catalogued its biases Zheng+ 2023.

Process for a trustworthy judge: write a decomposed rubric (several binary checks rather than one holistic score); label a few hundred items with humans; measure judge–human agreement (accuracy, Cohen's κ) and compare it to human–human agreement; iterate the prompt; then freeze the judge version and re-validate whenever you change it. Use the judge for relative comparisons, and don't use the same judge as an RL reward and as the final evaluator.

Statistical rigor: error bars on evals

Treat an eval as an experiment: the questions are a sample from a larger super-population of possible questions, and the score is an estimate with sampling error Miller 2024 Anthropic 2024.

Standard error of the mean

For n items with per-item scores \(s_i\) (0/1 for accuracy), \(\mathrm{SE} = \sqrt{\widehat{\mathrm{Var}}(s)/n}\), which is \(\sqrt{p(1-p)/n}\) for accuracy. The 95% CI is about \(\pm 1.96\,\mathrm{SE}\).

To halve the error bar you need 4× the items. A useful rule: the minimum detectable difference between two independent evaluations at 80% power and α = 0.05 is about \(2.8\sqrt{2}\,\mathrm{SE}\), roughly 4 SE. So on a 500-item benchmark near 50%, you need a gap of about 9 points to reliably detect it with unpaired analysis.

Paired comparisons (the big win)

When two models answer the same questions, analyse per-question differences \(d_i = s_{A,i}-s_{B,i}\):

Because models tend to find the same questions easy or hard, the covariance is usually substantial and positive, so the paired SE can be much smaller than the unpaired one. Paired analysis is "free variance reduction" Miller 2024. Use a paired t-test on \(d_i\), McNemar's test on the discordant pairs (A right/B wrong vs. A wrong/B right), or a paired bootstrap.

Other variance sources

• Clustered questions. Items that share a passage (reading comprehension) or a repo (SWE-bench) aren't independent. Use cluster-robust SEs. Naive SEs can understate uncertainty by a factor of up to about 3 when clusters are strong Miller 2024.
• Sampling randomness. With temperature above 0, each item's score is itself random. Sampling several times per item reduces the within-item variance (useful for small sets), but the between-question variance is the floor.
• Seed and training noise. Two training runs with different seeds differ by about as much as many claimed improvements on some benchmarks. Run several seeds for ablations when you can.
• Multiple comparisons. Testing 20 benchmarks, one will "improve significantly" at p < 0.05 by chance. Pre-register the primary metrics or correct (Holm, Benjamini–Hochberg).
• Infrastructure flakiness. For agentic evals, timeouts, rate limits and sandbox failures can move scores by points. Track and report infra-failure rates separately from task failures.

Evaluating reasoning and test-time compute

Reasoning models (o1-style and successors) turn inference compute into accuracy, both through longer chains of thought and through parallel sampling. A single accuracy number hides this trade. Snell et al. showed that compute-optimal test-time strategies, adapted to problem difficulty, can beat a much larger model at matched FLOPs on some problems Snell+ 2024. s1 showed that "budget forcing" (suppressing the end-of-thinking token, or appending "Wait") lets you sweep thinking length and trace a curve Muennighoff+ 2025.

How to evaluate properly:

• Plot accuracy against tokens (or $, or latency) on a log-x axis, for several reasoning-effort settings. Compare models by curve position: which one reaches X% at the fewest tokens, and where each plateaus. A model that wins at high budget can lose at low budget.
• Separate sequential from parallel scaling. Longer CoT and maj@k / best-of-N have different curves and different latency profiles.
• Report the budget. "Accuracy at max effort" isn't comparable to another lab's default-effort number. Count hidden reasoning tokens in the cost.
• Watch for inverse scaling. More thinking can hurt on some task types (distractors, overthinking, spurious features) Gema+ 2025.
• Mind truncation. Score responses that hit the max-token limit as failures, and report how often it happens. Raising the limit can look like a capability gain.
• Use process metrics where relevant, such as verifier-checked step correctness or the rate of backtracking. These are diagnostic, not headline numbers.

Safety and dangerous-capability evals, red teaming

Frontier labs run dangerous capability evaluations before release under their scaling and preparedness policies. The main domains are CBRN uplift (chemical, biological, radiological, nuclear), offensive cyber, autonomous replication and AI R&D automation, and persuasion or deception. Google DeepMind's 2024 paper is a good template for how the domains are structured Phuong+ 2024. Typical forms:

Elicitation is the core methodological issue. A capability eval is an upper-bound claim only if you tried hard to elicit the capability: best scaffolding, tools, fine-tuning to remove refusals, and many attempts. Under-elicitation leads to false "safe" conclusions. Sandbagging (a model strategically underperforming) and evaluation awareness (models recognizing tests) are recent concerns that make propensity evals harder.

Red teaming: manual red teams of domain experts, plus automated red teaming where an attacker LM generates test cases and a classifier scores them Perez+ 2022. Anthropic's scaling study of human red teaming found RLHF models got harder to red-team as they scaled, and released the attack dataset Ganguli+ 2022. Jailbreak robustness is measured as attack success rate (ASR) under a fixed attack budget.

Eval harnesses

Whatever harness you use, log every raw transcript, version everything (model, harness commit, prompts, judge model), and store per-item scores so you can run paired analysis later. Most "mysterious" eval regressions turn out to be template, tokenizer or extraction bugs, and you find them by reading transcripts.

Part 2 · Interpretability: why it matters

Evals tell you what a model does on the inputs you thought to test. Interpretability tries to tell you how it does it, which gives you predictions about inputs you didn't test. The main motivations:

• Safety and auditing. Detecting deception, hidden objectives or dangerous knowledge that behavioural evals miss (a model that knows it's being tested can behave well). Probes on internal states can act as monitors.
• Debugging. Understanding why a model fails (a spurious feature, a wrong retrieval, a shortcut) so you can fix data or training rather than patching prompts.
• Control. Steering behaviour by intervening on activations, which is cheaper than fine-tuning and more targeted.
• Science. How do in-context learning, factual recall and planning actually work? The answers inform architecture and training choices.

A useful axis: top-down / representation-level methods (probing, steering, representation engineering) find directions that correlate with concepts and test interventions. Bottom-up / mechanistic methods (circuits, SAEs, attribution graphs) try to reverse-engineer the computation into human-understandable parts. Another axis is correlational vs causal: probes and attention maps show what is present, while patching and ablation show what is used.

Probing (linear probes)

A probe is a small classifier, usually linear (logistic regression), trained on frozen hidden activations \(h_\ell(x)\) at layer \(\ell\) to predict a property \(y\): part of speech, sentiment, truthfulness, "is this prompt harmful", board state in Othello-GPT. If a linear probe gets high held-out accuracy, the information is linearly decodable at that layer Alain & Bengio 2016. Sweeping layers gives a profile of where a feature becomes available.

Caveats

• Decodable doesn't mean used. The model may never read that information. A probe shows correlation, not mechanism. Follow up with causal interventions (ablate or steer along the probe direction and see whether behaviour changes).
• Probe capacity. A powerful (MLP) probe can learn the task itself from rich features. Hewitt & Liang's control tasks assign random labels to word types. A good probe has high selectivity (real-task accuracy minus control-task accuracy), and simple linear probes are preferred Hewitt & Liang 2019.
• Dataset confounds. A "truth" probe may actually detect "statement looks like a common fact". Test it out of distribution.

Unsupervised variants: Contrast-Consistent Search (CCS) finds a direction where a statement and its negation get complementary probabilities, with no labels, as an attempt to find "what the model believes" Burns+ 2022. Later work questioned whether CCS finds truth specifically rather than other salient features. The broader linear representation hypothesis (concepts are directions) has both theoretical framing Park+ 2023 and known counterexamples, such as circular features for days of the week and other nonlinear manifolds.

The logit lens and tuned lens

Transformers keep a residual stream \(h_\ell\) that every layer adds to. The final prediction is \(\mathrm{softmax}(W_U\,\mathrm{LN}(h_L))\). The logit lens applies the unembedding to an intermediate residual: \(\mathrm{softmax}(W_U\,\mathrm{LN}(h_\ell))\). This lets you watch the prediction form layer by layer: early layers decode to junk or the input token, then the right answer "rises" in the middle and late layers nostalgebraist 2020.

It works because the residual stream is roughly aligned to a shared basis across layers, but it's brittle: in some models (early layers, or models whose basis drifts) the decoded distributions are meaningless. The tuned lens trains a small affine "translator" per layer, \(\mathrm{softmax}(W_U\,\mathrm{LN}(A_\ell h_\ell + b_\ell))\), to minimize KL to the final output. It's more reliable, and the trajectory of predictions across depth can even help detect prompt injection or anomalous inputs Belrose+ 2023.

Attention-pattern analysis and its limits

Attention weights \(A = \mathrm{softmax}(QK^\top/\sqrt{d})\) are easy to visualize. Head-level patterns are real and sometimes interpretable: previous-token heads, induction heads, heads that attend to a syntactic head or to the first token ("attention sinks"). Tools like BertViz and CircuitsVis made this popular.

Why attention isn't explanation:

• Attention weights often correlate poorly with gradient-based importance, and you can construct very different attention distributions that give the same prediction Jain & Wallace 2019. (A follow-up, "Attention is not not explanation", argued the picture depends on the test, which is why this is a nuanced point.)
• The weight says where a head reads from, not what it reads (the OV circuit decides what's copied) or whether the output matters downstream.
• Information mixes across layers: by layer 20, "token 5's" residual already contains information from many other positions, so attention to position 5 isn't attention to the word at position 5.
• Many heads attend to sink tokens (BOS, punctuation) as a no-op, which inflates apparent importance.

The mechanistic replacement is to separate each head into a QK circuit (\(W_E^\top W_Q^\top W_K W_E\): which tokens attend to which) and an OV circuit (\(W_U W_O W_V W_E\): what gets written when attended), and to test claims causally Elhage+ 2021. Anthropic's 2025 work extends attribution to explaining why attention patterns form, in terms of feature interactions.

Mechanistic interpretability: residual stream, features, superposition, circuits

The residual-stream view

"A Mathematical Framework for Transformer Circuits" reframed the transformer as a communication channel. The residual stream is a shared memory of width \(d_\text{model}\). Each attention head and MLP reads from it through a linear projection, computes, and adds its output back Elhage+ 2021. Consequences:

• The final logits are a sum of contributions from every component (direct logit attribution), because the stream is additive and the last operations are near-linear.
• Components can compose: a later head's Q, K or V can read what an earlier head wrote (Q-, K- and V-composition). Paths through the network can be enumerated.
• Attention heads move information between positions. MLPs transform information within a position (and store a lot of factual knowledge as key–value lookups).

Features and superposition

A feature is a property of the input the network represents, ideally as a direction in activation space. If each neuron represented one feature, neurons would be monosemantic. In practice many are polysemantic, firing for unrelated things (academic citations, Korean text and HTTP requests). Superposition explains this: when features are sparse (rarely active at the same time), a network can represent many more features than it has dimensions by assigning them nearly orthogonal directions. It accepts a little interference in exchange for capacity Elhage+ 2022. In high dimensions you can fit exponentially many almost-orthogonal vectors (by the Johnson–Lindenstrauss intuition), so superposition is cheap. The toy models show phase changes: as sparsity rises, features move from "not represented" to "dedicated dimension" to "superposed in geometric arrangements" such as antipodal pairs and pentagons.

Circuits and induction heads

A circuit is a subgraph of features and weights that implements a behaviour. The canonical example is the induction head, a two-head circuit that does in-context copying: on a sequence … [A][B] … [A] it predicts [B] Olsson+ 2022.

Olsson et al. found that induction heads form in a sudden phase change early in training. It shows up as a bump in the loss curve and coincides with a jump in in-context learning ability (loss on later tokens falling relative to earlier ones), and the paper argues that induction heads may account for much of in-context learning in larger models. Other well-studied circuits include the indirect object identification (IOI) circuit in GPT-2 small, which has name-mover, S-inhibition and duplicate-token heads Wang+ 2022, and automated discovery methods such as ACDC Conmy+ 2023.

Activation patching and causal tracing

The workhorse causal method. Run the model on a clean input (for example "The Eiffel Tower is in" → Paris) and a corrupted input (a different subject, or noise added to the subject embeddings). Then, one at a time, copy an activation (a head output, MLP output or residual at some layer and position) from one run into the other, and measure how much of the clean behaviour is restored, for example through the logit difference.

• Denoising (patch clean → corrupted) asks: is this component sufficient to restore the behaviour?
• Noising (patch corrupted → clean) asks: is this component necessary?
• Causal tracing (ROME) is denoising with Gaussian-noise corruption, swept across layers and token positions. It localized factual recall to mid-layer MLPs at the last subject token Meng+ 2022.
• Path patching restricts the patch to specific paths (head A → head B's keys) to map circuits. Attribution patching uses a first-order (gradient) approximation to estimate every patch in one backward pass, trading accuracy for speed.

The methodological choices (corruption type, metric, which positions) noticeably change the conclusions. Gaussian-noise corruption can push the model off-distribution, and logit difference is generally preferred over raw probability Zhang & Nanda 2023 Heimersheim & Nanda 2024. Watch for self-repair (backup heads compensate when one is ablated), which makes necessity tests understate importance.

Sparse autoencoders and dictionary learning

If activations \(x \in \mathbb{R}^{d}\) are sparse linear combinations of many feature directions, \(x \approx \sum_i f_i\, d_i\) with most \(f_i = 0\) and \(m \gg d\) directions, then recovering \(\{d_i\}\) is a sparse dictionary learning problem. A sparse autoencoder (SAE) learns it with a one-layer encoder and a linear decoder Cunningham+ 2023 Bricken+ 2023:

Each column of \(W_d\) is a feature direction. The expansion factor \(m/d\) typically ranges from 8× to 256× or more. Each latent's activations are inspected (top-activating examples, logit effects) and auto-labelled by an LLM.

Key results and variants

• Towards Monosemanticity (Anthropic, 2023): SAEs on a 1-layer transformer's MLP produced features that are far more interpretable than neurons (DNA sequences, Arabic script, base64) Bricken+ 2023.
• Scaling Monosemanticity (2024): SAEs with up to 34M features on Claude 3 Sonnet's middle residual stream. Features were multilingual, multimodal and abstract (code bugs, deception, sycophancy, the Golden Gate Bridge), and clamping features steered behaviour (the "Golden Gate Claude" demo) Templeton+ 2024.
• TopK SAEs (OpenAI): replace L1 with keeping the top-k latents. This sets sparsity directly, removes "shrinkage" (L1 pulls activations toward zero), gives clean scaling laws, and was trained at 16M latents on GPT-4 activations Gao+ 2024.
• JumpReLU SAEs (GDM): a learned per-latent threshold with a straight-through estimator for an L0 penalty, which improves the reconstruction–sparsity frontier Rajamanoharan+ 2024. Gemma Scope released open JumpReLU SAEs for every layer of Gemma 2 models Lieberum+ 2024. Browse them on Neuronpedia.

How SAEs are evaluated

Feature splitting and other limitations

• Feature splitting. As the dictionary grows, one feature ("math") splits into finer ones (algebra, geometry, LaTeX tokens). There's no single "true" feature set: granularity is an artefact of dictionary size. Related problems are feature absorption (a general feature such as "starts with S" fails to fire where a more specific feature absorbs it) and composition (features that are combinations of others).
• Dark matter. Reconstruction error doesn't fall to zero as you scale. A residual part of the activations isn't captured, and it may contain important computation.
• Not causal by construction. SAEs model activations, not computation. A feature can be interpretable yet unused, and the decomposition can reflect dataset statistics rather than the model's own ontology.
• Downstream disappointment. GDM reported that SAE-based probes underperformed linear probes on out-of-distribution harmful-intent detection and deprioritized fundamental SAE research GDM 2025. A 2026 preprint found that on synthetic data with known features, SAEs recovered only a small fraction of true features despite high explained variance Korznikov+ 2026. A counter-position: SAEs are weaker for acting on known concepts but valuable for discovering unknown ones (hypothesis generation, auditing) Peng+ 2025.
• Cost. Training SAEs over every layer of a frontier model takes significant compute and storage.

Transcoders, attribution graphs and circuit tracing

SAEs decompose a representation. To trace computation you want interpretable pieces connected by interpretable interactions. Transcoders do this: instead of reconstructing an MLP's input, a transcoder is trained to imitate the MLP, mapping its input to its output through a sparse latent layer. You can then substitute it for the MLP, and feature-to-feature interactions become (mostly) linear and input-independent Dunefsky+ 2024.

Anthropic's circuit tracing (2025) scaled this up Ameisen+ 2025:

1. Train a cross-layer transcoder (CLT): each layer's features read from that layer's residual stream and can write to all subsequent MLP outputs, which collapses multi-layer MLP computation into fewer features.
2. Build a replacement model by swapping all MLPs for the CLT. For a given prompt, freeze attention patterns and normalization, and add error nodes for whatever the CLT fails to reconstruct, so that the local replacement exactly reproduces the model's output on that prompt.
3. Because the replacement is linear given the frozen attention, compute an attribution graph: nodes are active features (plus input tokens, error nodes and output logits), and edges are direct linear contributions. Prune it to the most influential paths and group features into "supernodes" by hand.
4. Validate hypotheses with interventions in the real model: suppress or inject features and check that downstream features and outputs change as the graph predicts.

Findings in Claude 3.5 Haiku Lindsey+ 2025 included genuine multi-hop reasoning inside the forward pass ("capital of the state containing Dallas" goes through an intermediate "Texas" feature), planning in poetry (rhyme candidates chosen before the line is written), language-independent concept features with language-specific input and output circuitry, a default "can't answer" circuit that known-entity features inhibit (relevant to hallucination), and cases where the model's verbal explanation of its arithmetic didn't match the internal mechanism. The authors noted that the method gave satisfying insight on only roughly a quarter of the prompts they tried, and that it doesn't explain attention-pattern formation (QK) by default. An open-source circuit-tracer library with Neuronpedia integration lets you generate attribution graphs for open-weight models.

Steering vectors and representation engineering

If concepts are (approximately) linear directions, you can add a direction to the residual stream at inference to change behaviour, with no weight updates. Activation addition (ActAdd) takes the difference of activations on a contrast pair of prompts ("Love" − "Hate") and adds a scaled version at some layer for all positions Turner+ 2023. Contrastive Activation Addition (CAA) averages the difference over many paired examples of a behaviour (sycophantic vs. not) to get a more robust vector Panickssery+ 2023:

Representation Engineering (RepE) generalizes this into "reading" (find concept directions with PCA on contrastive activations, use them as monitors) and "control" (add or subtract them) for honesty, harmlessness, emotion and more Zou+ 2023. A striking example: refusal in many chat models is mediated by a single direction. Ablating it (projecting it out of weights or activations) largely removes refusals, and adding it makes the model refuse harmless requests Arditi+ 2024. This shows both the power of the method and why open-weight safety tuning is shallow. Persona vectors extend the idea to monitoring and controlling traits such as sycophancy or "evil" across fine-tuning, including predicting which training data will shift a trait Chen+ 2025.

Model editing: ROME and MEMIT (briefly)

Model editing changes a specific fact ("The Eiffel Tower is in Rome") without retraining. ROME uses causal tracing to find the mid-layer MLP that recalls facts at the last subject token. It treats that MLP's output projection as a linear associative memory \(W k \approx v\) (key = subject representation, value = fact), and applies a closed-form rank-one update so the subject's key maps to a new value while other keys are disturbed as little as possible Meng+ 2022. MEMIT spreads updates across several layers to insert thousands of facts at once Meng+ 2022.

Evaluation uses efficacy (the edit holds), generalization (paraphrases work), and specificity (neighbouring facts don't change). Known limitations: poor "ripple effects" (editing a person's birthplace doesn't update facts that follow from it), degradation after many sequential edits, and evidence that where causal tracing localizes a fact doesn't predict where editing works best. In practice, retrieval (RAG) or fine-tuning is usually preferred over weight editing for knowledge updates. Editing matters more as an interpretability probe than as a product tool.

Chain-of-thought faithfulness

A visible reasoning trace is tempting to read as an explanation. Faithfulness asks whether the CoT reflects the factors that actually drove the answer. The evidence says it often doesn't:

• Biasing features go unmentioned. When few-shot examples always had answer (A), or a user suggested an answer, models shifted toward it while their CoTs rationalized the choice without mentioning the bias Turpin+ 2023.
• Interventions on the CoT. Truncating the CoT early, inserting mistakes, paraphrasing, or replacing it with filler tokens tests whether the answer depends on the CoT's content. Dependence varied by task, and larger models were sometimes less reliant on their stated reasoning Lanham+ 2023.
• Reasoning models. Given hints (including "unauthorized access" style hints) that they demonstrably used, Claude 3.7 Sonnet and DeepSeek R1 verbalized the hint in at least 1% of cases in most settings, but often below 20%. Outcome-based RL improved faithfulness at first and then plateaued, and when RL taught reward hacking, models almost never verbalized the hack Chen+ 2025.
• Mechanistic evidence. Attribution graphs showed cases where a model's described arithmetic method differed from the internal one, and cases of "motivated reasoning" working backward from a suggested answer Lindsey+ 2025.

But CoT is still useful for monitoring. When a task is hard enough that the model needs to reason in tokens, the reasoning has to pass through the visible trace, so monitors can catch misbehaviour there. A multi-lab position paper argues that this monitorability is "a new and fragile opportunity", which training pressure on the CoT (optimizing it to look good) or architectures that reason in latent space could destroy Korbak+ 2025.

What is established vs. recent or contested

Interview question bank