Embedding a router into your agentic system sounds like an easy win. Send simple requests to cheaper models, reserve expensive models for harder tasks, or route by specialty—code to Claude, multimodal to Gemini, and so on. Use a classifier or heuristic rules to make the decision, costs drop, performance stays the same. Done.
But it’s not that simple. Most routing systems assume model selection is a classification problem. In our experience building routing for agentic systems, what appears on the surface to be a model selection problem quickly becomes a system optimization problem. There are three dimensions that make this unexpectedly difficult.
1. Cost is More Than Just Model Pricing
We expected GPT-4.1 to be cheaper than Claude Sonnet 4.6. It wasn’t.
Across 417 tasks in the AppWorld Test Challenge, using the same CodeAct agent, the total cost for Sonnet was $79 ($0.19 per task), while GPT-4.1 cost $155 ($0.37 per task)—nearly double. Looking at the paper prices alone, this makes no sense. GPT-4.1 has lower input and output token pricing, and Sonnet required about three times as many reasoning steps to complete the same tasks. Based on list price alone, GPT-4.1 should have easily won.
Why? Caching—something most routing discussions completely ignore.
Agent workloads tend to reuse a lot of context across multiple steps. When cache hit rates are high, effective input costs drop significantly. Sonnet’s lower cache read pricing allowed it to benefit more in this mode, enough to offset its higher base pricing and longer reasoning trajectories.
The takeaway: actual cost depends on the interaction between the model, workload, and serving infrastructure. Routers that only look at pricing tables are optimizing for the wrong metric.
2. Complexity is More Than Just Task Difficulty
A common routing strategy is to estimate how hard a task is, then send the harder tasks to stronger models. This approach is intuitive, but it has two problems.
First, task difficulty is often invisible at the time of routing. A request like “summarize this contract” looks simple, but before completion, it might trigger retrieval, compliance checks, tool calls, and multiple rounds of refinement. Meanwhile, a highly technical prompt might be efficiently handled by a smaller but more specialized model. You usually don’t know how hard a task actually is until execution begins.
Second, even if you could perfectly estimate difficulty, it is only one of many signals. In production, routers need to simultaneously balance cost, latency, model expertise, and reliability. Enterprise deployments add even more constraints: compliance requirements, data residency rules, privacy restrictions, and approved model lists. A task that ideally should go to a certain model might have to go to another due to governance requirements—and the router must handle this situation gracefully.
Routers aren’t solving a single problem. They are constantly trading off cost, quality, latency, compliance, and reliability all at once.
3. Latency is More Than Just Model Speed
It is easy to think of latency solely in terms of model size—large models are slower, small models are faster. But the actual user-perceived experience is much more complex.
Routing itself introduces overhead. Infrastructure factors—what hardware the model runs on, whether the cache is warm, how busy the endpoint is—often dominate end-to-end response time. Even a theoretically faster model can end up being slower if serving conditions are unfavorable.
In addition, there is the issue of routing granularity. Routing only once per task has the lowest overhead. But if you route at every step—which gives you more flexibility to adjust during execution—it means every added decision point introduces latency and operational complexity.
Routers that ignore the serving system are optimizing for the wrong reality.
So, How Did We Handle It?
These lessons shaped how we build routers. The key shift: we no longer view routing as a classification problem, but as an optimization problem. Instead of asking “which model is best for this task?”, our algorithm optimizes across cost, quality, and latency simultaneously—and is lightweight enough to avoid becoming a bottleneck.
The chart below shows our results on the AppWorld Test Challenge using the CodeAct agent. Each blue square represents a different configuration of our router, outlining a cost-accuracy frontier. The key is not a single point, but that the router provides a set of operating points for you to choose from depending on whether you prioritize cost, latency, or accuracy. Configuration 1 (optimized for latency) achieved 84% accuracy, costing $93 and taking 83 seconds—compared to running Opus alone, this reduced cost by 21% and latency by 9%, with only a 4% drop in accuracy. Configuration 2 pushes costs even lower.
[
](/news-images/2026-07/be1081986