I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
> Their internal confidence can be measured and returned
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.
I recently wrote about first principles of model routing that I've learnt building a model router.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.
> Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
Even the simplest router that differentiates models between plan & execute will ensure that this is consistently followed. Folks have too high FOMO to choose themselves.
Their router classified prompts into difficulty buckets. This obviously won't work. Consider a senior developer routing work based only on the task description. Clearly you need to dig a bit deeper into the task.
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
You cant fail over to a different model though. Even failing over to a different provider isn't always as simple as set it and forget it. For example bedrock doesn't support the tool search tool feature from anthropic.
Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.
I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
> Their internal confidence can be measured and returned
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
At least one model can today. https://github.com/cactus-compute/cactus-hybrid
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
You do still need to know though, different prompts have different affects on them.
I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.
Found the drywaller
>Our router was classifying each request into one of four different tiers of complexity: simple, standard, complex and reasoning.
Seems like a naive classification model lacking context?
I recently wrote about first principles of model routing that I've learnt building a model router.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
https://try.works/first-principles-of-model-routing
I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.
Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
what’s wrong with the sentence? reads fine to me
'keeps' is the wrong tense and could be 'will keep'
i think it should be "by adding stickiness to the initially chosen provider of that model"
> Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
I'm still working on mine: https://github.com/rush86999/atom/blob/main/docs/architectur...
Routing belongs on the client side
Even the simplest router that differentiates models between plan & execute will ensure that this is consistently followed. Folks have too high FOMO to choose themselves.
Their router classified prompts into difficulty buckets. This obviously won't work. Consider a senior developer routing work based only on the task description. Clearly you need to dig a bit deeper into the task.
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
I think fail-over for a production critical service is an equally important responsibility of the router.
You cant fail over to a different model though. Even failing over to a different provider isn't always as simple as set it and forget it. For example bedrock doesn't support the tool search tool feature from anthropic.
They beta support for it now I believe
Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.