Choosing models and routing
Pick models on the capability/cost/latency tradeoff — and route between them.
Model choice in production is not "use the best model" — it's a per-task decision on the capability vs. cost vs. latency tradeoff, plus the engineering to route between models. Getting this right is one of the biggest levers on both quality and your bill.
The three-axis tradeoff. For each task in your product, weigh:
- Capability — task-specific accuracy. Test candidate models on your task, not on leaderboards.
- Cost — price per token/request. At scale, this dominates.
- Latency — time to first token and total time (p50/p95/p99). Interactive features live or die on this.
Plus: context window, structured-output/tool support (next lessons), data-residency/compliance, and integration effort.
Closed vs. open, and the hybrid default. The maturing production pattern is hybrid: frontier closed models for complex multi-step reasoning and hard agentic chains; smaller open (often self-hosted) models for high-volume commodity tasks — classification, extraction, summarization. A lightweight classifier/router sends each request to the right tier. This captures open-source economics at volume while keeping frontier capability where it matters. (Teach "frontier / mid / small tier" as durable — specific model names date in months.)
Model routing — a first-class production component. A router sits between your app and multiple models/providers and picks per-request by cost, latency, quality, or availability, while adding failover (if one provider is down or rate-limited, use another), load distribution, and a single unified API. The durable pattern to teach: route easy requests to a small/cheap model, hard requests to a strong one, and check a cache first (cost lesson). Managed routing layers exist (treat vendor names as examples and re-verify) — the capability to expect is: multi-provider, per-request policy, failover, and observability of what got routed where.
Structured output support as a selection gate. Whether a model natively supports schema-guaranteed structured output and function calling (next lesson) is now a real selection criterion for any product that needs reliable machine-readable output — which is most of them. As of 2026 all major providers support it, but capabilities and quality differ.
Practical selection process:
- Define the task and its requirements (accuracy bar, latency budget, cost ceiling, output format).
- Shortlist 2–3 models across tiers.
- Evaluate them on your own task with your eval set (Module 3) — the only reliable comparison.
- Consider the hybrid: can a cheaper model handle most of it, with escalation to a stronger one for the hard cases?
- Build behind a router/abstraction so you can swap models without rewriting — models change monthly, and you'll want to.
The strategic point: don't marry one model. Build so model choice is a configuration decision per task, route intelligently across tiers and providers, and let your evals — not hype or a leaderboard — decide what's best for your workload. That flexibility is both a cost lever and insurance against a fast-moving model landscape.
For one feature you'd build, define its accuracy/latency/cost requirements, shortlist two models across tiers, and describe the routing rule you'd use (what goes to the cheap model vs. the strong one). Note how you'd swap models without a rewrite.
Enjoying the free lessons? Get an email when we publish new courses and updates — no spam, unsubscribe anytime.
Discussion (0)
Ask a question or share what worked for you. Comments are reviewed before they appear.
No comments yet. Be the first to start the discussion!