So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.

But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.

How do you deal with that?

We built something similar internally after AI spend on support-ticket triage kept beating estimates. Routing itself wasn't the hard part, it was that our dashboard only tracked immediate task completion, and had no idea a cheaper model had quietly produced a worse answer that the customer reopened a week later. We ended up tagging every response with which model handled it and joining that against reopen rate a month out, and the gap on the cheap tier was bigger than we expected going in. Curious if Tokenless has thought about exposing a delayed quality signal like that, not just pass/fail at the time of the call.

Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).

I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.

Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?

The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?

I do wonder why not just use cheaper models like Luna or Grok instead of dealing with routing. DeepSWE has Grok at 54% for $2.42 or 67% for $3.03 for Luna. Tokenless Pro is 6.67 for the same performance as Luna.

Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.

One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?

Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.

> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track

By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.

It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.

τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.

Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?

Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)

Sounds like DigitalOceans Model synthesis (https://www.digitalocean.com/blog/model-synthesis) or AilinOne (https://ailin.guide/) ?

How is this different from OpenRouter?

I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average:

https://github.com/rush86999/atom/blob/main/docs/architectur...

Deja-vù: 15 years ago I was working for AWS, and at some point (2010-2011) tens of startups came out to provide cost optimization, cloud switching, etc.

If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.

The benchmark number gets the click, but what decides adoption is whether someone can look at yesterday's traffic and see which turns went cheap and what that cost them in quality. Nobody hands routing to a gateway they can't audit after a bad output. Do you expose the per-turn decision, and can a team pin certain calls to a specific model when they need to? Worth saying plainly what querying multiple models at once does to latency, too.

feeding an existing conversion with output and reasoning from model x to model y will have many side effects, most likely a degradation of perfomance and alignment. should be done at least within the same family and generation of models... I'm not a huge fan of routers

Does this mean you have to retrain routing rules every time a new model gets released? I imagine since the price/token (or rather the amount of work that can be done per token) does not monotonically increase with new models, that the routing logic has to change all the time

I’ve found that automatic switching only becomes trustworthy when the system also makes the reason for each switch visible, especially when cost, latency, and output quality are being traded off.

One big issue is prompt caching - how are you (or any other router) solving for it? The moment caching is gone real agentic workloads would spike like crazy in terms of cost.

How can you be cheaper if you query multiple models at once, compared to me just using a single model, which will always have the correct caching configured?

Is the model picked through the router only for the first user turn or is there multi-turn routing (or planned to be added) ?

System Prompt Jailbreak (Playground) > You are a coding assistant powered by Tokenless, a router that dynamically picks the best LLM for each task. Help with programming questions, code review, debugging, and software design. Be concise and lead with concrete code or commands. Use fenced code blocks with the correct language tag.

If it only switches when cache is cold, it seems like it'd lock in one model for the whole session?

What use case are you imagining where cache eviction would happen during it? It might also be the case that a task might seem easy but then it ends up becoming hard for certain parts. For spawning subagents I could see this being useful maybe.

This approach only works for small context requests. For large context and relatively smaller output (say understanding a huge code base), the cost will mainly be on prefill, and sending the large context to multiple models will only increase the cost, possibly by some factor.

Model routing seems like a piece of the Ai stack that will quickly distill into industry if it is even that useful. I remain unconvinced that you need on the fly choice making beyond provider downtime. More likely you want to evaluate some and then settle on the model-agent-task pairings that work for you

Is Tokenless measuring cost per completed task, or just cost per request, including errors and retries?

There's so many reimplementations of Sakana Fugu from its ICLR papers like this one.

Optimizing resource allocation and cost-efficiency through dynamic model switching is a smart response to scaling AI workloads. When building data-intensive applications and managing automated pipelines, keeping operational overhead lean and modular is key to sustainable engineering

I don't understand how you can make a business here. It seems to me you would end up optimizing your router for bottom line at some point and quality would degrade.

I worry about overfitting and bias in the router algorithm. Sometimes temperature and or variance that doesn't align with a directive leads to a faster task resolution, which may seem higher cost but overall task cost ending up cheaper.

what sets this apart from other routers?

Confirmed YC orphan did not read Stripe news feed

is this like openrouter?

Crowded field, tough to build a moat without user data imo but that data is valuable. Good luck.

Disclosure: I build a competing gateway.

We ended up doing roughly the session-lock you're describing — pin a conversation to whichever provider is already warm, reconsider only on a cold prefix — and what surprised me is that eviction was rarely the thing that broke it.

Any request that changed the prefix at all did: a tool definition added mid-session, a system prompt carrying a timestamp, retrieval that reorders context. Those all still look like a warm session to the router and are a full re-price in practice.

So the lock ended up being less "stay until eviction" and more "stay until something upstream invalidates the prefix" — and most of our savings came from stopping our own code from invalidating it, not from the routing decision.

[dead]

[dead]

[flagged]