A Practical Guide to LLM Model Routing and Cost

Here is a question I hear a lot from engineering teams building AI features: should we use the cheap model or the expensive model?

The short answer is both, strategically. The longer answer is that picking the right model for each request, which is what model routing means, is one of the highest-ROI decisions you can make on inference cost, and it does not require sacrificing quality where it matters.

How the current pricing tiers break down

As of July 2026, the LLM API market has three clear cost tiers.

Budget tier (under $1/M input): Models like GPT-5.6 Luna ($1/$6), DeepSeek V4 Flash ($0.14/$0.28), Gemini 3.5 Flash ($1.50/$9), and Grok 4.1-fast ($0.20/$0.50). These are fast, capable for straightforward tasks, and cheap enough that you stop thinking about per-request cost.

Mid tier ($1-5/M input): Claude Sonnet 5 ($2/$10), GPT-5.6 Terra ($2.50/$15), Gemini 3.1 Pro ($2/$12), Claude Sonnet 4.6 ($3/$15). These are your daily-driver models for production workloads where quality matters but you are not doing cutting-edge research.

Flagship tier ($5+/M input): GPT-5.6 Sol ($5/$30), Claude Opus 4.8 ($5/$25), Claude Fable 5 ($10/$50, from 20 July), GPT-5.5 ($5/$30). These are for complex reasoning, code generation, legal analysis, and anything where being wrong costs more than the extra tokens.

The spread matters. The difference between a budget model and a flagship model is roughly 10-50x on input and 15-100x on output. If you can route 80% of your traffic to the budget tier without quality regression, you cut your inference cost by 70-90%.

What model routing looks like in practice

Model routing means you classify each incoming request by complexity or domain, then dispatch it to the cheapest model that can handle it well.

A customer support chatbot is the textbook case. A user asking "what are your pricing plans" goes to GPT-5.6 Luna or Gemini 3.5 Flash. A user asking "why is my integration showing a 503 error with a timestamp mismatch" goes to Sonnet 5 or Opus 4.8. The simple question costs about $0.001 to answer. The complex one costs $0.02-0.05. The ratio is real.

There are three main approaches to building the router:

Rule-based routing: Match on keywords, intent classification, or request metadata. A support system routes based on detected language, message length, or presence of technical keywords. Simple to build, transparent, no extra model cost. Works well for clear-cut cases.

Classifier-based routing: A small, cheap model (Haiku, Luna, GPT-5 Mini) decides which tier model handles the request. You pay a few hundredths of a cent for the classification call, which saves you from sending every request through the flagship model. This is the most common pattern I see in production.

Fallback routing (also called deferral): Start with the cheap model. If it signals low confidence (log probabilities, refusal tokens, a specific output format), escalate to the next tier. This is more conservative than classifier routing and guarantees the cheap model gets a shot before you spend flagship tokens.

All three can run inside a simple middleware layer. Cost was designed for this. It wraps your API client, attributes every call to a route, and lets you define routing rules per endpoint without changing your application code.

A worked example

Say you run an agent for internal developer support. It handles roughly 50,000 requests per month across three task types: documentation lookups, debugging help, and code review.

Without routing, every request goes through Opus 4.8 ($5/$25). Average prompt is 4,000 input tokens and 600 output tokens. Per-request cost comes to about $0.035. Monthly total: approximately $1,750.

With routing, you set up a classifier. Documentation lookups (40% of traffic) route to GPT-5.6 Luna ($1/$6) at $0.007 per request. Debugging help (35%) goes to Sonnet 5 ($2/$10) at $0.014 per request. Code review (25%) stays on Opus 4.8 at $0.035 per request. The classification call costs roughly $0.0005 per request.

The blended cost works out to about $0.014 per request, or $700 per month. That is a 60% reduction for a one-time setup cost of writing the routing rules and a classification prompt.

These numbers are conservative. Teams that iterate on their routing rules and prompt design generally push the cheap-model share higher over time as they build confidence in the routed output quality.

The verification question

The reason most teams do not route aggressively is fear: what if the cheap model gives a wrong answer on an important request?

That fear is valid. The solution is not to avoid routing but to verify that the routed model's output is good enough before committing to the swap.

This is what Cost's replay and verify flow does. You replay real production traffic from your expensive model through the candidate model, judge the outputs automatically (using a third model as evaluator or a programmatic check), and get a pass/fail rate per route before you route any live traffic. The judge does not need to be the most expensive model either. A Sonnet or Haiku evaluating completions costs a fraction of the savings from getting the routing decision right.

The numbers I see consistently: teams that do this end up routing 60-85% of requests to cheaper models on their first pass through the eval, and the remaining 15-40% are genuinely complex enough to need the flagship. Over a month, that is the difference between a $10,000 bill and a $2,000-4,000 bill.

Open questions worth thinking about

There are a few subtleties that come up once you decide to route.

Session-level routing matters more than per-request routing. If you switch models mid-conversation, prompt caching breaks because each model expects its own cached prefix on the provider's side. Routing at session granularity (pick one model when the conversation starts) keeps caching intact and avoids paying the write premium twice.

Output cost dominates. Across every provider, output tokens are priced 3-6x higher than input. A routing strategy that saves on input but doubles output token count (because the cheap model rambles or self-corrects) can backfire. Profile both input and output savings before cutting over.

Batch changes the math. Models offered through batch APIs at 50% off synchronous pricing can shift the cost-per-request calculation. A route that looked borderline at full price may make sense at batch pricing, especially for async workloads (document processing, nightly eval runs, bulk classification).

What this means for your bill

If you are currently running everything through a flagship model, model routing is the single highest-ROI change you can make. It does not require prompt engineering, does not require changing your application code once the routing middleware is in place, and the savings compound: cheaper models also have cheaper cached input, cheaper batch pricing, and lower per-token fine-tuning costs.

The caveat is the same as every other optimisation in this space: measure first, cut over second. Run the replay, check the eval, look at the numbers. If the budget model passes on 80% of your routes, route 80% of your traffic. If it only passes on 40%, route 40%. The model is not the strategy. The measurement is.

What is the hardest routing decision you have had to make for your LLM workload?