tkdb
8 hours ago
I wonder if model router adoption is higher for LLMs performing in-app functions, instead of the author's case of writing software?
One of the ways I can make a custom app/harness for a given task more of a competitive advantage is by striving for model agnosticism - the thing works and so has the valuable part outside of a model.
We've already seen older models being unavailable, and the newer (supposedly better) model doesn't do the work as well for some reason. Or a service is just down for minutes or hours.
verdverm
6 hours ago
Yea, I tend to agree with this. The models are going to get good enough at the flash sizes that it won't matter much. I'm more in the camp that you want multiple model families around, and that we will use gateways for upstream provider resilience (eg. when one has downtime, fallback to another provider/model)