Very true. In fact, there's a lot more under the surface that will come to light once more efficient tiny models come out.

P.S. Building something that proves that you don't need those many params even.

https://github.com/guilt/tinytot

But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.

As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.

They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.

In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.

The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.

Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.

Coming from this perspective, it is pretty easy to see what they are.

Yeah and these latest and greatest models universally suck hard for any use case outside of the few 'blessed' ones. Like I'm sure their ability to write prose and generally sound like a human being has regressed quite a bit, but even if not. Opus 5 is barely above GPT4 when it comes to stuff like home improvement advice.

Fine timing takes time and data, though. If smart enough models get cheap enough, then most people have lots of use cases that are cost insensitive enough that it's not worth the effort.

Of course, "smart enough" is a low enough threshold for most uses that this is still a problem for the frontier labs.

But at the same time, a truly smart enough closed model could also potentially command almost whatever they'd care to charge for it.

Whether they can actually get to that level remains to be seen, but I can definitely see a situation where most people are perfectly happy with cheap middle of the tree models while large corporations pay magnitudes more than current API pricing for access to models never even marketed as a mass market product and keep the labs afloat.

It's of course be a lot easier for them to find the path towards that of they didn't need to compete with open models in the meantime.

seems that, like software engineering and other areas before, they also have to rediscover that one single monolithic solution that handles everything is too inflexible and not maintainable, it's just a bad approach. people don't need the models they use to generate their codebase to also be able to translate Shakespeare into gen-z slang

Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.

what are those cheap fine-tuning services these days?

> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages

I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.

They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.

the major labs want to advance science. current business use cases are a happy accident

Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.

Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.

Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.

And the hidden, trial and error runs tweaking the hyperparams. guaranteed, it cost >$5000 in training runs alone.

What do you mean with maintaining the model? I am puzzled.

Anything but Sol is not sufficient for complex (or even just large) enough code.

The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart.

Anything else? Sure use a cheaper model.

This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.

But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.

Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).

Most of the recreation humans do in front of screens is tending to (para)social relationships.

I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.

> I wrote the whole thing down here

I like the idea, but it looks really hard to read. Try reading it top to bottom without skipping.

Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?

The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.

This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.

It doesn't mean frontier models aren't good at harder tasks.

Huge SOTA models are like a floodlight. They elucidate a huge area at once.

A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.

Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!

I agree very much. I'm getting the same take-away even more now that I'm using autoresearch as my main strategy.

And I wonder: We've got all these amazing (programming) languages to define solutions; where are the languages to clearly define the problems?

Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.

Better prompting doesn't improve response time or price!

They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.