The production LLM application stack

Anyone can call an LLM API. Building a production application — one that's reliable, safe, affordable, and improvable — means assembling a stack of layers, each independently testable, versioned, and observable. This lesson is the map; the rest of the course is each layer.

The layered architecture, as a request flows through it:

1. Model / provider layer — the LLM(s) behind an API or self-hosted, increasingly plural (multiple models + a router).
2. Prompt / context layer — system prompts, examples, retrieved context, memory, tool definitions. Where "context engineering" lives.
3. Retrieval (RAG) — grounding the model in your proprietary/fresh data.
4. Tools / agents — the model taking actions via function calls or MCP, in act-observe loops.
5. Orchestration — the control flow composing the above: workflows (fixed paths) vs. agents (model-directed).
6. Evaluation — offline eval sets and online/production eval. The differentiator.
7. Observability / LLMOps — tracing every LLM/tool/retrieval call; monitoring cost, latency, quality.
8. Guardrails / safety — input validation, output filtering, PII, injection defense, human-in-the-loop.
9. Serving — the deployment surface: sync, streaming, async, batching, versioning.

The atomic building block: the "augmented LLM." Before the stack, internalize the unit it's built from (Anthropic's framing): an LLM enhanced with retrieval, tools, and memory, behind a clean interface. Everything above — workflows, agents, whole products — is composition of augmented LLMs. Design your app as augmented LLMs behind interfaces you can swap, test, and evaluate.

The guiding principle, repeated throughout: find the simplest solution possible, and only increase complexity when needed. Start with a single well-prompted LLM call. Add retrieval when it needs your data. Add tools when it needs to act. Add agents (model-directed paths) only when you genuinely can't predetermine the steps. Each layer adds capability and cost, latency, and failure surface — so earn each one. Many "AI products" over-engineer with agents where a prompt-plus-retrieval workflow would be more reliable and cheaper.

Why think in layers? Because each layer is where a specific class of problem — and a specific engineering discipline — lives: model choice and cost in the model/routing layer, hallucination in the retrieval and guardrail layers, reliability in structured output and evals, operability in observability and serving. When something goes wrong in production, the layer tells you where to look. And each layer is independently testable and versionable, which is what makes the whole system improvable rather than a fragile monolith.

Keep this stack in your head. Every lesson slots into a layer, and the through-line of the course is the same: build the simplest thing that works, make each layer testable and observable, and let evaluation — not vibes — drive every change.