TL;DR
- AI routing sends each incoming request to the model, agent or tool best suited to handle it, instead of running everything through a single default option.
- It’s how systems balance cost, latency and quality at scale: a cheap, fast model for simple requests, a stronger model reserved for the ones that actually need it.
Key Takeaways
- Cost, latency and quality are the three variables routing trades off. A routing strategy is really a decision about which of these matters most for a given request.
- Routing strategies range from rule-based to learned. Simple heuristics, classifier-based routing, and semantic/embedding-based routing solve the same problem with different amounts of complexity.
- Implementation depends on being able to classify a request cheaply. The routing decision itself has to be fast and low-cost, or it defeats the purpose.
- Common mistakes come from under- or over-engineering the router. Routing every request through a heavy classifier adds latency; routing everything through crude heuristics under-serves complex requests.
AI routing is how a system decides, request by request, which model, agent or tool should respond. Without it, every request goes to the same model, whether the question is simple or hard.
We don’t sell a router, but our own systems make routing decisions, and the examples here come from them. Groundtruth, Eigenform’s open source dynamic benchmark, tests which models are good enough for which questions. The Geocluster Research Harness sends each step of an analysis to a specialist agent, and our map server sends each query only to the data sources that cover its area.
What Is AI Routing?
AI routing is the process of sending each incoming request to the model, agent or tool best suited to handle it, instead of sending everything to one default. In LLM routing, a small decision step reads the request and picks a language model by task type, difficulty, cost or latency.
The component that decides is the AI router. When every option is a language model, the idea is called model routing. It is the most common case: the options differ in size, price and speed, and most requests don’t need the largest.
Routing happens at two levels. Per request, the router picks one route for the whole request. Per step, an agent system routes each stage of a task separately: a strong model for reasoning, a small one for formatting.
How Does AI Routing Work?
A typical AI routing flow has five steps:
- A request arrives.
- The router extracts signals: type, length, user tier or estimated difficulty.
- A policy maps those signals to a route.
- The chosen model or tool answers.
- The decision and its outcome are logged for later checks.
A worked example, for an assistant that answers questions about exploration data:
| Request | Signal the router reads | Route |
|---|---|---|
| “Convert these gold values from ppb to ppm” | One known operation | Small, fast model |
| “Two reports disagree on a shear zone. What does the evidence support?” | Several sources, multi-step reasoning | Larger model |
| “Plot arsenic against gold for these samples” | Names a dataset and an action | Plotting tool or agent |
Routing can pick among agents too. In the Geocluster Research Harness, an orchestrator plans each analysis and only routes work: it has no analysis tools of its own.
The orchestrator routes each step by task type to a specialist for data operations, transforms, analytics or visualization, each with its own slice of the toolset. Every specialist runs as a separate child process on the model the user picked. A route therefore changes instructions and tools, not the model.
AI Routing Strategy: From Rules to Semantic Routing AI
An AI routing strategy is a decision about which of cost, latency and quality matters most for a given request. The four common approaches range from rule-based to learned:
| Strategy | Routes on | Cost per decision | Needs |
|---|---|---|---|
| Rule-based / heuristic | Fields read directly | Near zero | Rules someone maintains |
| Classifier-based | A predicted category or difficulty | One small model call | Labeled examples |
| Semantic / embedding-based | Similarity to example requests | One embedding call | A few examples per route, a threshold |
| Cost- and latency-aware | Price, speed, a quality floor | Depends on the policy | Quality measured per route |
Rule-based / heuristic routing
Rules route on features read directly: an endpoint, a file type, a request length, a user tier, a bounding box. They add almost no latency and anyone can read them, but they are brittle where one intent has many phrasings, and they fail quietly when a rule matches for the wrong reason.
Our map server routes queries this way. In stratigraphic-amenity, our open-source MCP server for geologic maps, each regional source carries a bounding box. A query goes only to the sources whose box intersects the area asked about. Ontario and Quebec mineral-occurrence services are the example implementation; a new region is one more source with its own box (a registry entry and an adapter for its service), not a rewrite.
Classifier-based routing
A trained classifier, or a small language model, assigns each request a category or a difficulty estimate, and the policy maps that to a route. It handles phrasing that rules miss. In exchange, it needs labeled examples of which route worked, and its own model call has to cost far less than the models it chooses between.
The RouteLLM study (2024) trained routers on human preference data to choose between a stronger and a weaker model. It reported cost reductions of over 2 times in some cases without a drop in response quality.
Semantic / embedding-based routing
Semantic routing AI turns the request into an embedding, a vector that places texts with similar meaning close together. The router then sends it to the route whose example requests are nearest, which handles paraphrase.
Its weak spot is a request unlike any example, so it needs a similarity threshold below which the router declines to choose.
Cost- and latency-aware routing
This strategy picks the cheapest option that clears a quality floor within a latency budget. A common form is a cascade: try the small model first, and escalate only when its answer fails a check.
In our published run of 150 geology questions, four models shared the top tier of the Groundtruth leaderboard: none of the gaps between neighboring models in that tier was statistically significant.
Among the three with paid generation, cost per question ran from $0.19 to $0.33; the fourth cost $0.04 because its answers were generated on a free tier. When quality ties, cost decides the route.
Latency is traded the same way. Benchmark grading is a batch job, with no one waiting on each answer. By default, the Groundtruth code sends grading calls to a discounted service tier: about half the standard price, but slower and rate-limited more often. One flag switches to the standard tier when parallel workers keep hitting rate limits.
How to Implement AI Routing
Routing is one component of a broader AI orchestration layer, which also plans the steps, passes results between them and handles retries. Implementing AI routing comes down to four decisions: criteria, mechanism, monitoring and fallback.
Defining the routing criteria: cost, latency, quality and task type
Cost and latency come from price lists and logs, task type from the request itself. Quality, by contrast, needs an explicit floor per route, such as a minimum score on a fixed test set built from requests like your own.
That measurement is AI model evaluation applied to each route. It is also why Groundtruth generates its questions from source documents: a generic public score says little about whether a cheaper model is good enough on your material.
Some calls belong outside the router altogether. Groundtruth’s grading model is fixed in code, with no flag to change it, because scores only compare when they come from one fixed judge.
Choosing a routing mechanism proportional to the decision’s complexity
Start with the simplest mechanism that works, and add machinery only where rules misroute. Rules suit a handful of clearly different request types; a classifier or embeddings suit many intents or varied phrasing.
Proportion means the router’s cost relative to the work it routes. The coverage check in our map server is four coordinate comparisons, and an out-of-area source is skipped without a network request. Our harness’s orchestrator is a language model, so each decision costs a model call, but it dispatches a specialist run of up to 600 seconds.
Monitoring routing decisions and outcomes
Log each request’s signals, route, latency, cost and outcome, and audit a sample by also sending it down the strongest route.
Also log the route you got, not only the one you asked for. Before grading, Groundtruth sends one small request and prints the service tier the provider actually used. Each grading call then goes into a cost ledger. In the harness, each specialist’s cost joins the task total, and the orchestrator sees its last 20 dispatches with objective, status and duration.
Then read differences between routes against the test’s resolution. On our 50-question sets, the smallest gap distinguishable from noise was 0.27 to 0.45 points out of 10, the threshold for benchmark statistical significance depending on the set. A smaller gap between two routes proves nothing.
Fallback behavior when the router is uncertain
Decide what happens when no route clearly fits: default to the stronger model, ask a clarifying question, or hand off to a person. Whichever you choose, make the uncertainty visible.
Our map server answers a query outside a source’s area with a warning instead of a silent empty list, stating that this is not evidence that no occurrences exist.
A fallback also covers a route that fails. In our harness, a specialist missing a Python package reports a capability gap instead of installing it, so the orchestrator routes the install to a dedicated agent before resuming the specialist. Failed runs return as tool errors, a structural signal that is harder for the orchestrator to read past than a sentence inside a result.
Common AI Routing Mistakes and How to Avoid Them
Over-engineering the router. A heavy classifier in front of a cheap model can cost nearly as much as answering directly, and it adds latency to every request. Measure the router’s cost against what it saves.
Under-engineering it with crude rules. Crude heuristics under-serve complex requests, and text-matching rules misfire when the matched words also appear for other reasons. Key rules on structure (fields, file types, coordinates) instead of free text.
Routing on cost without a quality floor. In our benchmark, a model at $0.19 per question reached the top tier, while the most expensive of the seven models we priced, at $0.37 per question, had the lowest mean score among them. Only a measured floor shows which cheap option is acceptable.
No fallback when classification is uncertain. A router forced to choose will sometimes choose wrongly and silently. Give it a threshold below which it escalates, asks or declines.
Never re-evaluating routes. In our benchmark, one new model release on free-tier pricing moved the price-performance frontier by itself. Re-run the same frozen test whenever a model, a price or the traffic changes.
When teams ask what is model drift in AI, they usually mean a model serving users, but a router’s classifier drifts the same way: trained on last quarter’s requests, it loses accuracy as the request mix shifts, with no change to its code.
FAQs
What is the difference between AI routing and AI orchestration?
AI routing decides which model, agent or tool handles a request or a step. AI orchestration is the layer around it, which plans multi-step work, passes results between steps and retries failures. A router is one component of an orchestrator, and a simple system can route without orchestrating.
Is AI routing the same as load balancing?
No. A load balancer spreads similar requests across identical copies of a service to share capacity, without interpreting what they ask. An AI router reads each request and sends it to a different model, agent or tool depending on its content. Production systems often use both, with load balancing behind each route.
What are common AI routing strategies?
The main strategies are rule-based routing on fields read directly and classifier-based routing with a trained model. Semantic routing matches a request’s embedding to example requests, and cost- or latency-aware routing picks the cheapest option clearing a measured quality floor. Teams often start with rules and add learned routing where rules misroute.
How do you implement AI routing?
Define the criteria first: cost, latency, quality and task type, with a measured quality floor per route. Pick the simplest mechanism that works, log every decision with its outcome, and design a fallback for requests the router cannot classify. Re-test routes whenever a model, price or traffic mix changes.
What are common mistakes when implementing AI routing?
The usual mistakes are building a router heavier than its savings justify and relying on crude text rules that misroute complex requests. Others are routing on cost with no quality floor, having no fallback for uncertain requests, and never re-evaluating routes as models, prices and traffic change.
