AI System Design Case Studies

The "design an AI system" round is where everything else in this guide gets tested together: retrieval, agents, evals, serving, cost and safety. Interviewers aren't looking for a perfect diagram. They want to see that you can turn a vague product ask into a measurable task, start from the simplest thing that could work, do the token and dollar math out loud, and design for a component that is non-deterministic and sometimes wrong. This page gives you a reusable framework, then works through six full case studies and three short ones in the shape a strong candidate would present them.

A reusable framework for AI system design interviews

Classic system design interviews follow a familiar arc: requirements, API, data model, high-level design, deep dives, scaling. AI system design keeps that arc but adds steps for the parts that are new: defining quality, choosing and composing models, evaluating a stochastic component, and paying per token. Here is a ten-step framework you can run through in roughly 40 minutes. You won't spend equal time on each step. The interviewer will usually pull you into two or three deep dives, so keep the rest brief and signpost what you're skipping.

Step 1 in more depth: turn the ask into an eval

The most common weak opening is jumping to "we'll use RAG with a vector DB." The strong opening is a set of questions that pin down the task as something you could grade:

• Unit of work: a single answer, a multi-turn conversation, a completed action (refund issued), or an artifact (a PR, a report)?
• Ground truth: is there a checkable answer (database state, tests pass, extracted field equals the label), or is quality subjective (helpfulness, tone), which needs rubrics and judges?
• Error asymmetry: what does a false positive cost compared to a false negative? Wrongly issuing a $500 refund is different from wrongly escalating to a human.
• The business metric and the guardrail metrics: for example, raise automated resolution rate while holding CSAT and refund-loss rate flat.

Then say it as a sentence: "We'll call v1 successful if, on a golden set of 500 real tickets, it fully resolves at least 60% with no policy violations, escalates the rest with a correct summary, keeps p95 first-token latency under 2 s, and costs under $0.30 per conversation." The numbers are placeholders you agree with the interviewer, but having them changes the whole conversation.

Step 3 in more depth: why the baseline matters

Anthropic's guidance on agents is to find the simplest solution and add complexity only when it demonstrably helps. Their taxonomy separates workflows (LLMs and tools orchestrated through predefined code paths: prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) from agents (the model dynamically directs its own process and tool use). Anthropic 2024 Interviewers reward candidates who climb that ladder explicitly:

A related data point from coding: the "Agentless" paper showed that a fixed three-phase pipeline (localize, repair, validate) beat the open-source agents of the time on SWE-bench Lite, at about $0.70 per issue. Xia+ 2024 Agents have improved a lot since then, but the lesson holds: a well-structured workflow is often a strong baseline, and you should be able to argue for it.

How AI system design differs from classic system design

What interviewers actually grade

Back-of-envelope toolkit

You'll reuse the same handful of formulas in every case study. Memorize the shape, not the exact prices, which change every few months.

The last one is Little's law, and it's what sizes long-running agents and voice calls: 100 calls/s arriving that each last 4 minutes means about 24,000 concurrent calls.

Case study 1: Customer-support agent with tool access and human escalation

Prompt: "Design an AI agent for an e-commerce company that handles customer support chat. It should be able to look up orders, process refunds and cancellations, and escalate to human agents when needed."

This is the most common AI design question because it touches everything: conversation, retrieval over policy docs, tool calling with side effects, authorization, escalation, and a clear business metric. Anthropic uses customer support as the canonical example of where agents fit, because it pairs conversation with actions and success is measurable by resolution. Anthropic 2024

Requirements and clarifying questions

Success metrics. Primary: automated resolution rate (conversation ends with the issue solved and no human touch, and no re-contact within 7 days). Guardrails: CSAT not below baseline, policy-violation rate (refunds outside policy) below 0.5%, wrong-action rate near zero, escalation quality (the human didn't have to re-ask). Operational: p95 time-to-first-token under 2 s, cost per conversation under $0.40.

Back-of-envelope

Assumptions (state them explicitly): 50k conversations/day; 6 user turns per conversation; an average of 2 model calls per user turn (one to decide a tool call, one to answer after the tool returns), so 12 calls per conversation; a mid-tier model at $3 / $15 per M tokens.

Compare to the human baseline: if 60% of 50k conversations are automated at roughly $0.25 each, that replaces about 30k human contacts a day. At $4–6 each, that's well over $100k/day of human cost against roughly $12k of model spend. The economics are rarely the blocker here; trust, correctness and policy compliance are. Say that out loud: it tells the interviewer where you'll spend design effort.

Architecture

Component walkthrough

• Gateway. Terminates the authenticated session and attaches a server-side customer ID. The model never sees or chooses whose data to access; the customer ID is bound to the tool credentials, not passed as a free-form argument.
• Input guard. A cheap classifier (small model or a dedicated safety model) flags abuse, self-harm, legal threats and obvious injection attempts, and redacts card numbers before anything is logged.
• Orchestrator. A small, fast model first classifies intent (tracking, return, refund, cancel, account, complaint, other). Simple intents run a fixed workflow: "where is my order" is one tool call and a templated answer. Open-ended issues go to the agent loop, where a mid-tier model with tools and policy context iterates. The orchestrator also holds a per-conversation state machine (identified order, eligibility checked, action taken) and a budget (max steps, max tokens, max tool calls).
• Policy RAG. Return and refund policies, shipping FAQs, and product-specific rules, chunked by section and versioned. Retrieval filters on region and product category so the agent never mixes EU and US return windows. Policies change, so each answer logs the policy version it used.
• Tool service. The most important box. Tools are narrow and intention-revealing (get_order(order_id), check_refund_eligibility(order_id, items), issue_refund(order_id, items, reason, idempotency_key)) rather than a generic "call the Orders API." Every call is checked against a deterministic policy engine (eligibility windows, amount limits, the order belongs to this customer). Mutations take idempotency keys so a retried step can't double-refund. Anthropic's tool-design guidance makes the same points: design tools for the agent's affordances rather than wrapping raw APIs, namespace them, and return compact, high-signal results. Anthropic 2025
• Approval queue. Refunds above the auto-limit, or flagged by the fraud score, become a pending action that a human approves with one click. The agent tells the customer truthfully that it's been submitted for review.
• Escalation. Triggered by explicit request ("talk to a human"), low confidence, repeated failure (same intent three turns running), negative sentiment, or sensitive categories (legal, safety, chargebacks). The hand-off writes a structured summary (issue, order, what was tried, customer sentiment) so the human doesn't restart the conversation.
• Traces. Every prompt, retrieved chunk, tool call, tool result and decision is logged with a conversation ID. This is the raw material for evals and debugging.

Key design decisions and alternatives

Evals

• Simulated-user, outcome-based evals. The τ-bench design is the right template: an LLM plays the customer with a hidden goal, the agent talks to it using real tools against a sandbox database, and success is checked by comparing the final database state to the expected state. Yao+ 2024 Outcome checks catch "said the right thing but did the wrong thing."
• Reliability, not just accuracy. τ-bench introduced pass^k (the probability of succeeding on all k independent trials) and found agents that look fine on a single run are much less consistent: retail pass^8 fell below 25% for the then state-of-the-art. Yao+ 2024 For support, where every customer is one trial, consistency is the metric that matters.
• Policy adversarial set. Customers asking for out-of-policy refunds, social engineering ("my manager said…"), injection inside order notes.
• LLM-as-judge on transcripts for tone, accuracy against policy, and escalation summary quality, calibrated against a human-labeled sample.
• Online: resolution rate, re-contact within 7 days, CSAT, escalation rate and reasons, refund dollars per conversation versus baseline, cost per conversation.

Failure modes

How to extend

• Voice channel: reuse the same tool service and policy engine behind a voice front end (see case study 5). The tools are the durable asset; the conversational layer is swappable.
• Agent assist: run the same agent in "copilot" mode for human agents, suggesting replies and actions. That also generates labeled data (accepted, edited or rejected suggestions).
• Proactive support: trigger the agent from events (delayed shipment) with outbound messages.
• Distillation: once you have tens of thousands of successful, judge-verified trajectories, fine-tune a smaller model for the top intents.

Case study 2: Enterprise knowledge search (RAG) over Confluence, Drive and Slack with permissions

Prompt: "Employees waste hours finding information spread across Confluence, Google Drive and Slack. Design an assistant that answers questions over all of it, with citations, and never shows someone a document they can't access."

The interesting part isn't the LLM call. It's the ingestion pipeline, permission enforcement, freshness, and retrieval quality on messy enterprise data. Treat "never leaks a document" as a hard constraint, on par with correctness.

Requirements and clarifying questions

• Scale: how many employees and documents? Assume 20k employees; about 10M documents and messages, averaging 2k tokens, so roughly 20B tokens of raw text.
• Permission model: are source ACLs (Drive sharing, Confluence space and page restrictions, Slack channel membership) the source of truth? Yes. How quickly must a permission revocation take effect? Assume within 15 minutes for revocations; new content searchable within 1 hour.
• Answer style: a synthesized answer with citations, or ranked links? Both: answer plus sources, and a plain search mode.
• Latency: streaming answer, first token under 2 s, full answer under 8 s.
• Data residency and retention: must content stay in the company's cloud tenant? Assume yes; use a provider with zero data retention or a self-hosted model.
• Success metrics: answer correctness and citation faithfulness on a golden set; retrieval recall@k; zero permission leaks in red-team tests; online: weekly active usage, thumbs up rate, "answer found" rate, time saved in surveys.

Back-of-envelope

Assumptions: 20k users × 10 queries/day; chunks of about 500 tokens; 1024-dim embeddings; answer generation with a mid-tier model at $3 / $15 per M tokens.

Takeaway: generation dominates cost and retrieval quality dominates correctness; storage is cheap. 40M vectors fits comfortably in a single sharded vector index, or in Postgres with pgvector at the low end, and comfortably in a dedicated vector DB or a search engine with vector support (OpenSearch, Elasticsearch, Vespa).

Architecture

Component walkthrough

• Connectors. One per source, using webhooks or change feeds where available and periodic polling as a backstop. Each emits documents plus their ACLs and stable IDs. Commercial products in this space describe the same pattern: connectors mirror source permissions and resolve identities across systems so that if a user can't access an item in the source, they shouldn't see it in search. Glean docs
• Identity sync. Maps each source's users and groups to a canonical principal (the SSO identity plus its group memberships). Nested groups get expanded at sync time or at query time; expanding at query time keeps the index small but costs latency.
• Parse and chunk. Structure-aware chunking: split Confluence pages by headings, keep tables intact, convert Slack threads into a single "conversation" unit with the parent message as context, and drop bot noise. Each chunk carries metadata: source, title, author, last modified, URL, and a acl_principals list.
• Enrichment. Prepending a short, model-generated context header to each chunk ("This section is from the 2026 travel policy, section on international flights…") helps both BM25 and embeddings. In Anthropic's experiments, contextual embeddings cut retrieval failures by 35%, adding contextual BM25 got to 49%, and adding reranking reached 67%. Anthropic 2024
• Hybrid index. BM25 catches exact terms (project code names, error codes, people's names) that embeddings blur; dense vectors catch paraphrases. Merge them with reciprocal rank fusion, \( \text{score}(d) = \sum_r \frac{1}{k + \text{rank}_r(d)} \) with \( k \approx 60 \), which needs no score calibration between the two retrievers.
• Query planner. A small model rewrites conversational follow-ups into standalone queries, extracts filters ("in the Payments space", "last quarter"), and decides whether the question needs retrieval at all, or a structured source (the HR system for "how many PTO days do I have").
• Retrieve, filter, rerank. Retrieve about 100 candidates with the ACL filter applied inside the index query (the chunk's principals intersect the user's), then a cross-encoder reranks to about 10.
• Live ACL re-check. For the final handful of documents, re-check access against the source API (cached for a few minutes). This covers the window between a revocation and the next sync.
• Answer with citations. The prompt requires every claim to cite a chunk ID, and a post-check verifies each citation actually supports its sentence (string overlap or a small NLI/judge model). Unsupported sentences are removed or flagged. If retrieval found nothing good, the model says so rather than answering from parametric memory.

Key design decisions and alternatives

Evals

• Retrieval evals (cheap, fast, run on every change): a few hundred real questions with labeled relevant documents; measure recall@10, recall@100 and MRR. Most RAG quality problems are retrieval problems, so measure them separately from generation.
• Answer evals: LLM judge with a rubric for correctness against a reference answer, citation faithfulness (every claim supported by its cited chunk), and appropriate abstention ("I couldn't find this").
• Permission red-team suite: synthetic users in specific groups asking about documents they shouldn't see, including indirect asks ("summarize the layoffs plan"). The pass criterion is zero leaks, run in CI.
• Freshness tests: update or revoke a document and measure time until the index reflects it.
• Online: thumbs, citation click-through, query reformulation rate (a proxy for a bad first answer), zero-result rate.

Failure modes

How to extend

• Actions: "file a Jira ticket from this thread." This turns a read-only system into one with side effects, which re-opens the injection threat model; add confirmations.
• Structured sources: text-to-SQL over the data warehouse, with row-level security enforced by the database, not by the model.
• Expose as an MCP server so other agents (coding agent, support agent) can call "company search" with the user's identity.
• Personalization: boost documents from the user's team and recent collaborators.

Case study 3: Autonomous coding agent (issue → pull request)

Prompt: "Design a system where engineers assign a GitHub issue to an AI agent, and it comes back with a pull request that passes CI."

Coding is the domain where agents work best today, for one structural reason: there's a cheap, automatic verifier. Tests, type checkers, linters and builds give the agent a feedback signal it can iterate against, and give you an eval. Anthropic's coding-agent guidance boils down to "give the agent a check it can run," because without one, "looks done" is the only stopping signal. Claude Code docs

Requirements and clarifying questions

• Scope of tasks: bug fixes and small features (under ~300 lines changed) in existing repos, or greenfield? Assume the former.
• Repos: how many, which languages, how big? Assume 500 repos, mostly TypeScript, Python and Go, up to a few million lines.
• Autonomy: does it open PRs for human review (yes), or merge itself (no, never in v1)?
• Environment: can the agent build and run tests? It must be able to, or quality collapses. Do repos have reproducible dev environments (devcontainers, Dockerfiles)?
• Secrets and network: what may the sandbox reach? Package registries yes; production systems and secrets no.
• Volume: assume 2,000 tasks/day across the company.
• Success metrics: PR merge rate (merged with no or minor edits), CI pass rate at first PR, reviewer time per PR, time from assignment to PR, cost per merged PR, and revert rate after merge.

Back-of-envelope

Assumptions: a typical task takes about 50 agent steps (read files, grep, edit, run tests); the context grows from ~10k to ~80k tokens, averaging ~40k input tokens per step; ~400 output tokens per step; a mid-tier model at $3 / $15 per M; prompt caching on the growing prefix.

Two important observations. First, caching is the difference between viable and not: agent loops re-send the whole growing transcript every step, so the uncached cost grows roughly quadratically with step count. Second, the right denominator is cost per merged PR, not cost per task: if only 40% of PRs merge, the effective cost is 2.5× higher, and a stronger, pricier model with a higher merge rate can be cheaper overall.

Reviewer time usually dominates: 15 minutes of an engineer's time is worth far more than the tokens. So optimize for PRs that are easy to review (small, well-described, with tests), not just for raw success rate.

Architecture

Component walkthrough

• Task intake. Normalize the issue into a task spec: description, repo, base branch, linked files, acceptance criteria if present. If the issue is too vague, the agent's first move should be to ask a clarifying question on the issue, not guess. A cheap triage model can score "is this well specified and small enough" and decline the rest.
• Sandbox. Each task gets an isolated, ephemeral environment: a microVM (Firecracker-style) or a hardened container (gVisor), with the repo checked out at a pinned SHA, dependencies pre-installed from a cached image, and a network egress allowlist. The OpenHands platform is a good public reference: agents act by writing code, using a shell and browsing inside sandboxed environments. Wang+ 2024 Anthropic describes the same principle for Claude Code: filesystem and network isolation enforced by OS primitives, which in their testing reduced permission prompts by 84%. Anthropic 2025
• Agent-computer interface. The SWE-agent paper's key finding is that the interface matters as much as the model: purpose-built commands for viewing files in windows, searching, and editing with immediate lint feedback beat dumping raw shell output on the model. Yang+ 2024 Practical rules: edits as diffs or string replacements, not whole-file rewrites; truncate long outputs with a pointer to see more; show line numbers.
• Repo context. A conventions file at the repo root (build and test commands, style rules, gotchas) that the agent reads first; a symbol index or LSP for "go to definition"; grep for exact strings. Embedding search over code helps for "where is the auth logic" questions, but most agents lean on grep and file navigation, which are exact and cheap.
• Orchestrator. A durable job (Temporal, a workflow engine, or a queue plus checkpointing) because tasks run for many minutes and must survive restarts. It enforces budgets (max steps, tokens, wall clock), compacts context when it nears the limit (summarize old steps, keep the plan and modified-file list), and can spawn sub-agents for isolated exploration so the main context stays clean. Anthropic 2025
• Verifier. Runs the full CI-equivalent suite independently of the agent's own claims. Optionally a reviewer agent in a fresh context looks only at the diff and the issue, which avoids the author's bias. A failed verification feeds back into the loop for a bounded number of retries.
• PR creation. A draft PR with a summary of the change, why, which tests were added or run (with output), and any uncertainty. Human merge only.

Key design decisions and alternatives

Evals

• Public benchmark as a sanity check: SWE-bench consists of real GitHub issues from Python repos, graded by running the repo's tests (2,294 instances in the original set). Jimenez+ 2023 The human-validated "Verified" subset and other variants (Lite, Multilingual, Multimodal) are on the official leaderboard. swebench.com
• Internal benchmark, which is what actually matters: mine your own merged PRs that fixed issues and had tests. Reset the repo to the parent commit, give the agent the issue, and grade by running the PR's tests (fail-to-pass and pass-to-pass). A few hundred of these from your own repos predicts production far better than public numbers.
• Process metrics: steps, tokens and cost per task; tool error rate; how often the agent ran tests before finishing.
• Human-judged quality on a sample: would you merge this? Is the diff minimal and idiomatic? Did it add tests?
• Online: merge rate, reviewer edits, time to merge, revert rate, developer satisfaction.

Failure modes

How to extend

• CI-failure fixer and dependency upgrader: high-volume, well-verified task types that are a natural next step.
• Parallel fan-out for migrations: split a large migration into per-file tasks, run many agents, and merge in batches.
• Learning from review: feed reviewer comments into the repo's conventions file, so the next PR avoids the same mistake.
• Interactive mode: the same backend serving a developer in their IDE, with the human as the verifier.

Case study 4: Deep-research agent (orchestrator with parallel web-search subagents and citations)

Prompt: "Design a 'deep research' feature: the user asks a complex question, and the system spends several minutes searching the web and produces a long, well-cited report."

This is the canonical case for multi-agent systems, and there's an unusually detailed public write-up: Anthropic's description of how they built Claude's Research feature. Anthropic 2025 OpenAI's deep research, launched in early 2025, was described as an agent powered by a version of o3 optimized for browsing, searching and analyzing large amounts of web content. OpenAI 2025 Use these as reference points, but design from first principles.

Requirements and clarifying questions

• Query types: breadth-first ("compare the 10 leading vector databases on X, Y, Z") vs depth-first ("why did this drug fail its phase III trial")? Both, which is why the planner matters.
• Latency tolerance: users accept minutes for a report; assume a 3–15 minute target with visible progress.
• Output: a structured report of 1,500–5,000 words with inline citations to specific sources.
• Sources: open web only, or also the user's documents and internal tools? Start with web, design for pluggable sources.
• Volume: assume 5,000 reports/day.
• Success metrics: factual accuracy, citation accuracy (does the cited page actually support the claim), completeness against what an expert would cover, source quality, and efficiency (tokens and tool calls). These are close to the rubric Anthropic described for its judge. Anthropic 2025

Back-of-envelope

Assumptions: a lead agent plus 5 subagents on average; each subagent makes about 20 tool calls (searches and page fetches) and averages about 30k tokens of context per step; a final citation pass; mid-tier model at $3 / $15 per M; web search at $10 per 1,000 searches (Anthropic's listed price for its server-side search tool). Anthropic pricing

• Uncached: 3.3M × $3/M + 70k × $15/M ≈ $9.90 + $1.05 ≈ $11 per report. With caching inside each subagent loop, roughly $4–6.
• Searches: ~75 per report × $0.01 ≈ $0.75.
• 5,000 reports/day → roughly $25k–55k/day. This is why deep research is often rate-limited per user or tied to higher plans.
• Sanity check against the published ratio: Anthropic reported agents use about 4× the tokens of a chat interaction and multi-agent systems about 15×. Anthropic 2025 If a chat turn is a few thousand tokens, 15× is tens of thousands per turn, and a long research run covers many turns, so millions of tokens is plausible.
• Concurrency: 5,000/day × 10 min ÷ 1,440 ≈ 35 concurrent research jobs on average, each fanning out to ~5 subagents, so ~200 concurrent agent loops at peak. Plan rate limits with your model provider accordingly.

Architecture

Component walkthrough

• Clarification. Before spending dollars, ask one or two clarifying questions if the query is ambiguous ("for which market? as of when?"). Cheap and high leverage.
• Lead agent (orchestrator). Writes a research plan, saves it to memory (so it survives context truncation), and spawns subagents with explicit task descriptions: objective, output format, which tools and sources to prefer, and boundaries so subagents don't duplicate each other. Anthropic found vague delegation caused duplicated work and gaps, and that teaching the lead to scale effort to query complexity (one agent for simple fact-finding, many for broad comparisons) mattered a lot. Anthropic 2025
• Subagents (workers). Each runs its own search loop: broad queries first, then narrower ones; fetch and read pages; judge source quality; return a condensed summary with source URLs and key quotes. The main benefit is context isolation: each subagent burns through 100k+ tokens of web pages, but returns only 1–2k tokens to the lead.
• Parallelism. Anthropic reported that having the lead spawn 3–5 subagents in parallel, and subagents call tools in parallel, cut research time by up to 90% for complex queries. Anthropic 2025
• Iteration. After a round, the lead evaluates coverage against the plan and either spawns another round for gaps or proceeds to writing.
• Citation agent. A separate pass maps each claim in the draft to the specific source passage that supports it. Separating "write" from "cite" makes it easier to verify and to catch unsupported claims.
• Source store. Fetched pages are cached and content-addressed, so the citation agent and the UI can show exactly what was read, and repeated fetches are free.
• Durable execution. Long, stateful runs need resumability: if a step fails, resume from a checkpoint instead of restarting a 10-minute job. Anthropic also mentions deploying updates with "rainbow deployments" so running agents aren't broken by a mid-run code change. Anthropic 2025

Key design decisions and alternatives

Evals

• Start small, immediately. Anthropic started with about 20 queries representing real usage, because early changes have large effects that a small set can detect. Anthropic 2025 Don't wait to build a 1,000-item eval before iterating.
• LLM judge with a rubric: factual accuracy, citation accuracy, completeness, source quality, tool efficiency. A single judge call scoring 0–1 plus pass/fail was reported to be more consistent than multiple specialized judges. Anthropic 2025
• Short-answer benchmarks for search ability: BrowseComp (1,266 hard-to-find but easily verified questions) tests persistent browsing. Wei+ 2025
• Citation verification at scale: automatically check that each cited URL was actually fetched and that the quoted passage appears on it.
• Human review: Anthropic's human testers found the agent preferred SEO-optimized content farms over authoritative sources, which automated evals had missed. Anthropic 2025

Failure modes

How to extend

• Private sources: add enterprise search (case study 2) as a tool, with the user's permissions. Now the trifecta applies: web content plus private data. Restrict outbound actions and render links safely.
• Async and scheduled research: "monitor this topic weekly", with diffs against the previous report.
• Structured outputs: tables and charts from extracted data via code execution.
• Training: RL on research trajectories with verifiable rewards (short-answer questions) is how frontier labs improved browsing agents; out of scope for most product teams.

Case study 5: Real-time voice agent

Prompt: "Design a voice agent that answers inbound phone calls for a chain of clinics: it books, moves and cancels appointments, and answers basic questions."

Everything from case study 1 applies (tools, policy, escalation), but voice adds a hard real-time constraint. Humans expect a reply within a few hundred milliseconds of finishing a sentence; a 3-second pause feels broken. The design is dominated by the latency budget, turn-taking, and telephony plumbing.

Requirements and clarifying questions

• Channel: PSTN phone calls (inbound and possibly outbound), or in-app WebRTC? Phone, which means 8 kHz narrowband audio and extra network hops.
• Latency target: voice-to-voice (user stops talking → agent audio starts) under ~1.2 s at p50, under 2 s at p95.
• Languages and accents, and handling of names, dates of birth and phone numbers (where speech recognition is error-prone).
• Compliance: healthcare means HIPAA-style requirements: business associate agreements with every vendor, call recording consent, PHI handling, and transcripts for audit.
• Volume: assume 100k calls/day, averaging 4 minutes.
• Hand-off: warm transfer to front-desk staff with a summary.
• Success metrics: task completion rate (booking made correctly), transfer rate, average handle time, voice-to-voice latency percentiles, interruption-handling errors (talking over the caller, cutting them off), caller satisfaction, cost per call.

Two architectures: cascaded pipeline vs speech-to-speech

For a clinic-booking agent with compliance needs and heavy tool use, the cascaded pipeline is the defensible default in 2026. Speech-to-speech fits consumer companionship or language-practice products where naturalness beats auditability. Say that this is a fast-moving trade-off.

Latency budget

The most useful public breakdown comes from a widely used practitioner primer (originally written for the AI Engineering Summit in February 2025, updated in 2026), which totals roughly 1.3 s for a typical cascaded pipeline and recommends about 1.5 s as an achievable target (with well-optimized stacks reaching around 500 ms). Voice AI primer 2026

Telephony adds more: PSTN carriers and the media relay add their own buffering, so budget a few hundred extra milliseconds for phone calls compared to WebRTC (I don't have a verified source for a specific figure; measure it). Tool calls are the other big latency risk: a 1.5 s appointment-system lookup in the middle of a turn blows the budget, so play a short filler ("Let me check that for you…") while it runs.

Turn detection and barge-in

Turn-taking is the hardest part to get right and the part users notice most.

• Endpointing (when has the user finished?). The naive approach is voice activity detection (VAD) plus a silence timeout. OpenAI's Realtime API defaults to 500 ms of silence in its server_vad mode and also offers a semantic_vad mode that uses a classifier over the words spoken to decide whether the user is done. OpenAI docs Silence alone fails both ways: people pause mid-sentence ("my date of birth is… uh… March 3rd"), and a long timeout adds dead air to every turn.
• Turn-detection models. Open models now do this better. Pipecat's Smart Turn is an ~8M-parameter model that works directly on audio (16 kHz, up to 8 s), using prosody as well as content, with inference around 10–100 ms on CPU. Pipecat Smart Turn LiveKit shipped a transcript-based turn detector built on a small Qwen model and has since moved toward an audio-based one. LiveKit docs The emerging pattern combines a short VAD trigger, an audio turn model, and sometimes the LLM itself emitting a "complete / incomplete" token. Voice AI primer 2026
• Barge-in (the user interrupts the agent). When VAD detects user speech while the agent is talking, immediately stop TTS playback, flush buffered audio (on Twilio Media Streams, send a clear message to drop queued audio Twilio docs), cancel the in-flight LLM generation, and start listening.
• Context truncation after barge-in. The LLM's transcript says it said a full paragraph, but the user only heard the first sentence. Use word-level timestamps from TTS (or playback "mark" events) to truncate the assistant message to what was actually played, so the model doesn't assume the user heard information they didn't. Voice AI primer 2026
• Backchannels and false barge-ins. "Mm-hm" or a cough shouldn't stop the agent. Require a minimum speech duration or a short transcript before treating it as an interruption.

Back-of-envelope

Assumptions: 100k calls/day × 4 min = 400k minutes/day; about 6 conversational turns per minute (both sides); the per-minute component prices below are rough ranges, flagged as such.

Notice that in voice, the LLM is often not the biggest line item: TTS and telephony are. A speech-to-speech model at several times the cost changes that picture, which is why the architecture choice is also a cost decision. The concurrency figure drives infrastructure: about 1,000 simultaneous WebSocket or WebRTC sessions, each holding audio buffers, VAD and turn models, and STT and TTS streams. Size media servers by concurrent sessions, not by QPS.

Architecture

Component walkthrough

• Telephony. A SIP trunk or a CPaaS provider answers the call and streams audio to your worker. Twilio's Media Streams, for example, forks call audio over a WebSocket as 8 kHz μ-law frames and accepts audio back the same way, with mark and clear messages for playback tracking and interruption. Twilio docs At high volume, teams run their own SIP infrastructure to cut per-minute cost and latency. For in-app voice, use WebRTC end to end.
• Voice agent worker. One stateful session per call, pinned to a region near the caller and the model endpoints. Frameworks such as Pipecat and LiveKit Agents provide this pipeline (transport, VAD, STT, LLM, TTS, interruption handling) so you don't hand-roll audio plumbing.
• STT. Streaming recognition with custom vocabulary (clinic names, doctor names, medication names). Confirm critical entities by reading them back ("That's March 3rd, 1985, correct?"), and spell out alphanumerics for confirmation.
• Dialogue LLM. A fast model with a short, cached system prompt, a small set of tools, and voice-specific instructions: short sentences, no markdown or lists, numbers written to be spoken, one question at a time. Stream tokens to TTS at sentence boundaries.
• TTS. Streaming synthesis with word-level timestamps; a consistent voice; pronunciation dictionaries for names.
• Tools. The same deterministic tool layer pattern as case study 1: search slots, book, reschedule, cancel, with identity verification (name plus date of birth plus phone match) enforced in code before any PHI is spoken.
• Observability. Per-stage latency for every turn (STT final, TTFT, TTFB), interruption events, and full recordings and transcripts (with consent) for replay-based evals.

Key design decisions and alternatives

Evals

• Simulated callers: an LLM caller plus TTS with varied voices, accents, background noise and line quality, running scripted scenarios against the real pipeline. Score task completion by checking the scheduling system's final state.
• Component evals: STT word error rate on your own call audio (especially names and numbers), turn-detection precision and recall (cut-offs vs late responses), TTS pronunciation.
• Latency SLOs: p50 and p95 voice-to-voice per stage, tracked in production dashboards.
• Transcript-level LLM judge for policy adherence, tone, and whether identity was verified before PHI was disclosed.
• Online: completion and transfer rates, call abandonment, repeat calls, post-call surveys.

Failure modes

How to extend

• Outbound calls (reminders, rescheduling after a cancellation), which bring consent and calling-regulation requirements.
• Multilingual with language detection at the start of the call.
• Hybrid audio understanding: use an audio-native model to detect frustration or confusion, and escalate.
• Self-hosted STT and TTS at high volume to cut per-minute cost and latency variance.

Case study 6: Personalized assistant with long-term memory

Prompt: "Design a consumer AI assistant that remembers users across sessions (their preferences, projects, people in their life) and uses that to personalize answers, while respecting privacy."

Every major assistant now ships some form of memory. ChatGPT, for example, distinguishes explicit "saved memories" from implicitly "referencing chat history", and lets users view, delete and turn off both. OpenAI 2024 The design challenge isn't storing text. It's deciding what is worth remembering, keeping it correct as facts change, retrieving the right memory at the right time without creeping users out, and giving users real control.

Requirements and clarifying questions

• What kinds of memory? Stable preferences ("vegetarian", "prefers concise answers"), facts about the user's life ("has a daughter named Maya", "training for a marathon in April"), ongoing projects, and episodic recall ("what was that book we discussed last month?").
• Explicit vs implicit: only remember when asked ("remember that…"), or infer automatically? Both, with implicit memories visible and editable.
• Scale: assume 1M daily active users, 10 messages each per day, about 3 sessions per user per day.
• Privacy and regulation: GDPR and CCPA rights (access, deletion), sensitive categories (health, religion, sexuality, political views), minors, and regional data residency.
• Latency: memory retrieval must add less than ~150 ms to the first token.
• Success metrics: memory precision (stored memories are true and useful), recall in context (the right memory shows up when relevant), inappropriate-use rate (memory surfaced where it feels intrusive), user edits and deletions of memories, retention and satisfaction for memory-on vs memory-off cohorts.

Memory types and where they live

This mirrors the operating-system analogy from MemGPT: a small, fast "main context" and larger external tiers that the model pages information in and out of, through function calls. Packer+ 2023

Back-of-envelope

Assumptions: 1M DAU × 10 messages = 10M messages/day; 3M sessions/day; on average 500 memories per long-term user after a year; mid-tier model at $3 / $15 for chat; small model at $1 / $5 (half price via batch) for memory extraction.

The interesting insight: injecting memory into every call costs more than extracting it, so keep the always-on profile compact and make retrieved memories earn their place (a relevance threshold, not a fixed top-k). Because each user's memory is small and queries are always scoped to one user, you don't need a huge global ANN index: partition by user, and a per-user brute-force search over 500 vectors takes microseconds.

Architecture

Component walkthrough

• Extraction. After a session (or every N turns), a small model reads the transcript and proposes candidate memories as structured objects: {text, type, entities, valid_from, confidence, sensitivity, explicit}. Doing it asynchronously keeps it off the latency path and allows batch pricing.
• Consolidation. The hard part. Each candidate is compared against the user's nearest existing memories, and an LLM decides add, update, delete or no-op. Mem0 popularized this extract-then-update pipeline, reporting large reductions in latency and token cost against full-context baselines on the LoCoMo benchmark (91% lower p95 latency and over 90% token savings, by their own measurements). Chhikara+ 2025 Treat vendor-authored benchmark numbers with appropriate caution.
• Temporal validity. Facts expire or change: "lives in Boston" becomes "moved to Seattle." Store valid_from and superseded_by instead of overwriting, so the assistant can reason about time ("you used to live in Boston") and so a bad update can be rolled back.
• Retrieval. At query time, a hybrid search within the user's partition, a relevance threshold, and a sensitivity filter (don't bring up a health memory in an unrelated work conversation). Optionally, give the model a memory_search tool and let it decide when to look things up, the "just-in-time" pattern. Anthropic 2025 Anthropic's API offers a client-side memory tool where the model reads and writes files under a /memories path that the application maps to its own storage. Anthropic docs
• User controls. A memory page listing everything stored, with edit and delete; a "memory updated" indicator when something new is saved; a temporary or incognito mode; and a global off switch. These are product requirements, not polish.
• Deletion pipeline. Deleting a memory must also delete derived artifacts: embeddings, session summaries that mention it, caches, and (on a schedule) backups. Build this from day one; retrofitting it is painful.

Key design decisions and alternatives

Evals

• Public long-term memory benchmarks: LoCoMo (conversations spanning up to 35 sessions) Maharana+ 2024 and LongMemEval, which tests information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. Wu+ 2024
• Extraction precision and recall: on labeled transcripts, did it store the facts a human would, and nothing false?
• Update correctness: scripted multi-session scenarios with changing facts; check that the latest fact wins and history is retained.
• Appropriateness: a judge rubric for "was surfacing this memory helpful or intrusive here?", plus red-team scenarios (sensitive memories in unrelated contexts, memories about third parties).
• Deletion tests: delete a memory, then probe with paraphrased questions to confirm it no longer influences answers.
• Online: memory-on vs memory-off A/B on retention and satisfaction; rate of user edits and deletions (a spike means bad extractions).

Failure modes

How to extend

• Shared or team memory with explicit sharing scopes (a family's grocery preferences, a team's project context), which brings permission models like case study 2.
• Proactive features: reminders and check-ins based on remembered plans (the marathon in April), with strict opt-in.
• On-device memory for privacy-sensitive users, syncing only encrypted blobs.
• Memory for agents, not just chat: an agent's learned procedures and lessons ("this user's repo uses pnpm, not npm").

Three more prompts, sketched

These come up often enough that you should have a two-minute answer ready. Each one lists the clarifying questions that matter, the core architecture, the key numbers, and the deep-dive topics interviewers usually pick.

Sketch A: An internal LLM gateway / router platform

Prompt: "Fifty teams at our company call various LLM providers directly. Design a central platform for it."

• Clarify: which providers and self-hosted models; whether the goal is cost control, reliability, compliance, or developer velocity (usually all four); streaming support; latency overhead budget (aim for under ~20 ms added at p50).
• Core components: an OpenAI-compatible (or provider-neutral) API; authentication with per-team virtual keys; per-key budgets, rate limits and quotas; a provider abstraction layer; retries with backoff and fallback chains across providers and models, including fallbacks for context-window and content-policy errors (open-source gateways like LiteLLM implement exactly these patterns LiteLLM docs); request and response logging with PII redaction; cost attribution per team; guardrail hooks (PII, injection, moderation); and caching (exact-match and, carefully, semantic).
• Routing: static (team X uses model Y), rule-based (by task tag, prompt length or language), or learned. RouteLLM trained routers on preference data to choose between a strong and a weak model per query and reported cost reductions of over 2× in some settings without quality loss. Ong+ 2024 Learned routing needs per-use-case evals to prove it isn't hurting quality.
• Numbers: the gateway itself is a stateless proxy, so it scales horizontally; the hard limits are provider rate limits (tokens per minute) shared across teams. Plan capacity in tokens/minute, not requests, and do admission control by priority (interactive traffic over batch).
• Deep dives: streaming through the proxy (server-sent events, partial failures mid-stream: you can't transparently retry after tokens have reached the client); semantic-cache risk (serving the wrong answer to a similar-looking prompt, and cross-tenant leakage, so scope caches per tenant); model version pinning and deprecation migrations; data residency routing.

Sketch B: Document extraction pipeline for invoices at scale

Prompt: "We receive 2 million invoices a month as PDFs and scans from thousands of suppliers. Extract structured fields into our ERP."

• Clarify: the field list (supplier, invoice number, dates, line items, totals, tax, currency); the accuracy bar per field (totals and bank details must be near-perfect, descriptions less so); latency (batch is fine: hours, not seconds); languages; how much human review capacity exists.
• Pipeline: ingest → classify (invoice vs credit note vs junk) → text and layout extraction (native PDF text where available, OCR or a vision-language model for scans) → LLM extraction into a JSON schema with structured outputs (constrained decoding guarantees schema-valid JSON, but not correct values Anthropic docs) → deterministic validation (line items sum to the subtotal, subtotal plus tax equals total, dates parse, supplier and PO number match the vendor master and open POs) → confidence-based routing to human review → ERP.
• Numbers: 2M/month ≈ 67k/day ≈ 0.8/s, so a batch workload. At roughly 3k input tokens (page text or image) and 500 output tokens per invoice on a small model at batch price (say $0.50 / $2.50 per M), it's about $0.003 per invoice, or ~$6k/month. Images cost more tokens than text, so use extracted text when it's reliable. Human review is the real cost: if 10% go to review at 2 minutes each, that's 13k minutes/day, about 28 full-time reviewers.
• Key insight: the business metric is "straight-through processing rate at a target error rate." Accounting checks catch most extraction errors for free; use them as both a validator and a confidence signal. Per-supplier templates or few-shot examples (retrieved by supplier ID) lift accuracy for high-volume suppliers.
• Evals: field-level exact match on a labeled set stratified by supplier and document quality; track error rate per field among auto-approved invoices (the dangerous errors); reviewer corrections become new labels.
• Deep dives: prompt injection embedded in documents ("pay to account X") is a real fraud vector, so bank details must be validated against the vendor master and changes must go through a human; multi-page line items; when to fine-tune a small extractor (once you have hundreds of thousands of reviewer-corrected labels).

Sketch C: Content moderation with LLMs

Prompt: "Design a moderation system for user posts on a large social platform, using LLMs."

• Clarify: volume (assume 50M posts/day ≈ 580/s average); modalities (text, images, video); the policy taxonomy and how often it changes; latency (pre-publish blocking vs post-publish takedown); appeals; regional legal requirements; the cost of false positives (silencing users) vs false negatives (harm).
• Cascade architecture, because running a frontier LLM on every post is too expensive: (1) hash matching for known illegal content and spam fingerprints; (2) cheap, fast classifiers (small fine-tuned models) on everything, handling the clear majority; (3) an LLM policy reasoner on the uncertain middle band, given the written policy and the content, returning a label plus rationale; (4) human reviewers for the hardest cases, appeals and high-severity categories.
• LLM role: safety classifiers built on LLMs, like Llama Guard, classify both prompts and responses against a risk taxonomy and can be adapted to new taxonomies via instructions. Inan+ 2023 More recent "policy as prompt" reasoning models, such as OpenAI's open-weight gpt-oss-safeguard (October 2025), read a developer-written policy at inference time and explain their decision, so policy changes don't require retraining. OpenAI 2025 LLMs are also useful offline: labeling training data for the cheap classifiers, and drafting policy clarifications from reviewer disagreements.
• Numbers: if 5% of 50M posts reach the LLM tier at ~800 tokens each, that's 2B tokens/day; at a small model price (~$1/M input) that's about $2k/day, versus ~$40k/day to send everything. The cascade is the cost design.
• Evals: precision and recall per policy category on a golden set labeled by policy experts; inter-annotator agreement as the ceiling; drift monitoring (new slang, coordinated evasion); fairness checks across dialects and languages, where classifiers often have higher false-positive rates.
• Deep dives: adversarial evasion (leetspeak, images of text, coded language); threshold tuning per category and per region; reviewer wellbeing (blur, limit exposure); transparency and appeal flows.

Cross-cutting patterns: a cheat sheet

The six case studies reuse a small set of patterns. If you can name these and say when each applies, you can improvise a design for an unfamiliar prompt.

Interview question bank