LLM Model Routing Explained: How to Send Each Prompt to the Right Model and Cut Your API Bill

Most teams start the same way. They pick one strong model, wire it into the product, and ship. It works. Then the invoice arrives, and someone asks the uncomfortable question: do we really need the expensive model to answer "what are your opening hours?"

Usually, no. That's the whole idea behind model routing. Instead of sending every request to the same model, you decide per request which model should handle it. Easy prompts go to a small, cheap model. Hard ones go to the big one. Done well, you keep nearly the same quality and pay a fraction of the price.

This guide explains how routing works, the three patterns people actually use, a worked cost example you can check with a calculator, and the mistakes that quietly eat the savings.

What model routing actually is

A router is a small decision step that sits in front of your models. It looks at an incoming request and picks a destination. That's it. The "model" in model routing can mean different sizes from one provider, models from different providers, or even the same model with different settings, such as a reasoning mode switched on or off.

It helps to separate routing from two things it often gets confused with.

Routing is not load balancing. Load balancing spreads identical work across identical servers. Routing sends different work to different models on purpose.

Routing is not fine-tuning. You aren't changing any model. You're only choosing between the ones you already have. If you're weighing that against other approaches, our guide on covers when changing the model itself is worth it.

The reason routing pays off is simple. Prices between model tiers often differ by an order of magnitude or more, while a large share of real traffic is easy: classification, extraction, short answers, formatting, simple rewrites. You're overpaying whenever a strong model handles work a small one could have done.

Why the price gap matters more than the average price

People often compare models by looking at a single "price per million tokens" number. For routing, the number that matters is the gap between your cheap tier and your expensive tier, and the share of traffic you can safely move down.

If the gap is 5x and only 5% of your traffic can move, routing isn't worth the engineering time. If the gap is 20x and half your traffic can move, it's probably the single biggest saving available to you. Our breakdown and the are good places to see how wide those gaps are right now.

Three routing patterns that work

1. Rule-based routing

This is the boring option, and often the right first step. You write explicit rules based on things you already know about the request.

• The feature that sent it (autocomplete versus "analyze this contract").
• The user's plan (free tier gets the small model).
• The task type (extraction and tagging go small; multi-step planning goes large).
• Whether the request needs tools or structured output.

Rules are cheap, fast, predictable and easy to debug. Their weakness is that they can't read the difficulty of the prompt itself. Two requests from the same feature can be very different in how hard they are.

2. Classifier routing

Here a small model, or a lightweight classifier, reads the prompt and predicts how hard it is, or which category it belongs to. Based on that label, the router picks a model.

The classifier is itself a cost, so it has to be much cheaper than the savings it creates. In practice it reads only a short prompt and outputs a handful of tokens, so its cost is usually small compared with the main call. You train or prompt it using examples from your own logs, labeled by whether the small model got the answer right.

This pattern handles variety better than rules, but it introduces a new failure mode: the router can be wrong. A hard question misrouted to a weak model produces a bad answer, and the user never knows a cheaper path was chosen.

3. Cascade routing

A cascade flips the logic. You always try the cheap model first, check whether the answer is good enough, and escalate to the strong model only if it isn't.

The check can be simple: did the output parse as valid JSON, did the model express low confidence, did it fail a validation rule, did a cheap grader score it below a threshold? If the answer passes, you're done. If not, you pay for the stronger model on top of the failed attempt.

Cascades are attractive because they route on the actual output rather than a prediction. The trade-off is latency. Escalated requests wait for two calls, which matters in anything interactive.

A worked cost example

Numbers make this concrete. The prices below are made-up round figures for illustration. Plug in current prices from your own provider before drawing conclusions.

Assume a product with:

• 100,000 requests per month
• 1,200 input tokens and 300 output tokens per request on average
• A small model at $0.15 per million input tokens and $0.60 per million output tokens
• A large model at $3.00 per million input tokens and $15.00 per million output tokens

That's 120 million input tokens and 30 million output tokens per month in total.

Everything on the large model: Input: 120 × $3.00 = $360. Output: 30 × $15.00 = $450. Total: $810.

Everything on the small model: Input: 120 × $0.15 = $18. Output: 30 × $0.60 = $18. Total: $36. Cheap, but if quality drops on hard requests, this isn't a real option.

Classifier routing, 70% small and 30% large: Small model gets 84M input and 21M output: $12.60 + $12.60 = $25.20. Large model gets 36M input and 9M output: $108 + $135 = $243. The classifier reads about 300 tokens per request, which is 30M input tokens at the small-model price, roughly $4.50, plus a negligible output cost. Total: about $273, a saving of around 66% against the large-only setup.

Cascade, 25% of requests escalate: Every request pays for the small model: $36. A quarter of them also pay for the large model: 0.25 × $810 = $202.50. Total: about $238, a saving of around 70%, assuming your quality check is cheap or rule-based.

Two things stand out. First, the savings are driven almost entirely by what share of traffic stays on the cheap model. Second, the large model still dominates the bill even when it handles a minority of requests. That's why the escalation rate is the number to watch. You can rerun these with your own volumes in the , and the helps you measure your real average prompt size instead of guessing.

So which AI has the best routing cost?

People search for this a lot, and the honest answer is that there isn't one winner. Routing cost isn't a feature of a single model. It comes from how your own traffic is split.

What you can compare is:

• The price gap between tiers within one provider, or across providers.
• Whether the provider or a gateway offers built-in routing, and what it routes on. Read what it really does before trusting it with your quality.
• Whether caching still works after you start switching models (more on that below).
• How the models behave on your tasks, which is a question for your own evaluation, not for a public leaderboard. We explain why in .

Mistakes that quietly eat the savings

Routing on prompt length alone. It's tempting because it's easy. But short prompts can be hard ("prove this is true") and long ones can be easy ("summarize this list"). Length is a weak signal.

Skipping evaluation. If you can't measure quality, you can't tell whether the cheap path is good enough. Build a small labeled set from real traffic first. Our guide on shows how to do it without a huge budget.

Forgetting hidden output tokens. Reasoning models can spend a lot of tokens thinking before they answer, so a "small" reasoning model isn't always cheaper than a larger non-reasoning one. See before you assume the price list tells the whole story.

Breaking your cache. Prompt caching usually works per model. If a request bounces between models, you can lose discounts you were counting on. Check how fits with your routing plan, and consider keeping a conversation on one model once it starts.

Letting the router drift. Traffic changes. A threshold that was right in March can be wrong by September. Review it on a schedule.

How to build a first router in a week

You don't need a research project. A sensible order is:

1. Log everything for a week. Prompt, model, tokens, latency, and some way to judge the answer. Even a thumbs-up signal helps.
2. Pick one high-volume, low-risk feature. Tagging, extraction, FAQ answers. Not your most critical flow.
3. Write rules first. See how much traffic they can move without any model in the router.
4. Run in shadow mode. Send requests to the cheap model too, without showing the answer to users, and compare it with the strong model's output.
5. Set an escalation rule. A validation failure, a confidence signal, or a grader score under a threshold.
6. Ship to a small percentage. Watch quality and cost together, then widen.
7. Keep a fallback. If the cheap model errors or times out, the strong one should take over automatically.

When routing isn't worth it

Skip it, or postpone it, when your volume is low enough that the engineering time costs more than the savings, when every request really is hard, when latency is critical and you can't afford a second call, or when you're still figuring out the product. Routing is an optimization. It rewards a product that already works.

Quick FAQ

Is model routing the same as a mixture of experts? No. A mixture-of-experts model routes tokens between experts inside a single model. Model routing happens outside, between separate models you call. If the inside version interests you, our covers it.

Will routing hurt answer quality? It can if the router is wrong, which is why evaluation and escalation rules matter. The goal is to lose nothing you can measure.

How much can I realistically save? It depends on your price gap and how much traffic is easy. In the example above it's about two thirds, but that's an illustration, not a promise. Measure your own mix.

Do I need a custom classifier? Not at first. Rules and a cascade get you surprisingly far. Add a classifier when you have enough labeled data to justify it.

The short version

Don't pay strong-model prices for easy work. Start with rules, measure quality on a small labeled set, try a cascade for the uncertain middle, and watch the escalation rate because it drives the bill. Then rerun the numbers in the with your own traffic. The model you use matters less than how often you use the expensive one.