Yes.

The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.

The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.

For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.

But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.

Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable.

DeepSeek v4 flash has been dirt cheap and so fast that my development loop is;

- small prompt

- review

- small prompt

- review

I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding).

In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine).

I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself.

That said, most of my team vibe codes and reviewing their work is like pulling teeth.

Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then.

However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.

Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.

What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.

I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.

But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.

Same. Mistral 7b has been more than I ever needed for text for years now.

Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.

Mistral 7b can do anything, and it’s basically instant even on an M3

While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.

It will be interesting to track the improvements of these 7B model over time.

There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.

Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.

There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.

ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai.

A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation

I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.

> We have, right now, access to things that 10-20 years ago would be considered magic

These things would be considered magic even 4 years ago!

BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

[flagged]

There is a thin line. Lots of AI powered things are just workflows you could implement with claude code and some skills. Or even just a prompt on the web chat. But I think these big technological swings sometimes take a long time to shake out. Society is still adapting to the internet.

I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do more purpose driven things not "this npc is just a chatbot with a personality in its system prompt." I mean we literally have systems in games called "AI" and we dont think these much more powerful AIs have a place?

And almost all consumer software has AI now.

> But what if you want to add AI to your product? Well, now you have some real inference costs on every request!

Eventually, these companies just lower their costs by using more efficient models. There were consumer companies built on GPT-3.

Yeah, I think if we reframed it as "why is there no consumer AWS?" It would make more sense.

AI is a utility that can abstract code to such a high level it is indiscernible from natural language.

> What would consumer AI company even be?

Something that helps homeowners build anything on their property, for example https://myhomecat.com

I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.

> July 2009

> Thanks to Sam Altman, … for reading drafts of this.

Oh how times have changed!

I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.

Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed

Sol is 100 EUR/m and I prefer it over Fable, and you get practically unlimited usage unless you go really overboard with it. So I don't get the cost argument. Would be better to own hardware, but with the shortages that's not very attractive atm.

I received access to AI in May of this year and used smaller models a lot in May, June, and July. Haiku 4.5 is terrible for coding. This month Luna was added and it's incredibly capable. I have to reach for a better model 1/50 times instead of 1/2 or 1/3 with Haiku. Luna is better and cheaper than the Haiku and Sonnet models I've been spending most of my time with.

It's safe to say I didn't come to this conclusion because I was "addicted to chasing frontier models"

Exactly. Even 32b parameter models you can run locally on consumer hardware are "good enough" at this point for some workflows!

no they are not good enough for "most" tasks

> for most tasks

The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.

People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.

Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.

The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.

And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.

Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.

yeah, I think they will get smaller so they can be run everywhere, and really just be an interface to various non AI systems.

[flagged]

Is a "truly intelligent home" something I should want? None of the current generation of "smart" addons are true value adds - their entire purpose is data collection. The pretext is always absurdly thin. I just thought "I bet there's a wifi enabled microwave", googled it, and indeed, Samsung have released such a thing - you can control it with your voice! Wow! Never mind that you can't un/load food with your voice, or that you're never more than 10 feet from it in the kitchen anyway. Oh but there's an app. Of course there's an app. It "suggests recipes". Right.

I think hoping for a locally hosted "smart" home is backwards. They have no other function than to invade your space. "Smart" objects are agents, and they don't work for you.

Same. I was in the doom AI crowd ~2 years ago, then I realized how many problems could be mitigated or solved by having these local models, and I became part of the cautiously excited crowd

Can we stick one of these in something that looks and sounds like HAL 9000 ?

I don’t love the “forced upon them” framing; if that’s really how people are thinking about it then maybe they should pause and reflect for a moment yhat it isn’t their money being spent. Amd the default isn’t always having the latest and greatest, it’s not paying for anything at all.

Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH my instinct is that that experiment is likely to cost more than the potential cost savings.

What I will say is that my own sense from experimenting around in a non-rigorous way is that the answer depends on how you use the tool. For actual vibecoding you should always go for the SOTA model because it will need less oversight. It’s also less likely to get stuck in a vicious loop that fruitlessly wastes tokens. But for a more hands-on approach where you move in small, carefully planned increments that you review and test in human-comprehensible chunks, smaller models may be preferable. SOTA ones don’t do that much better when working that way, and the slower inference adds a detrimental amount of friction to the work cycle.

It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate.

If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.

I’m building an internal tool for our company, basically an agent to help with on-call and alerts via Slack. I have evals running across a few scenarios, and my favorite models so far are Sol medium and Luna xhigh.

Sol medium has been a nice balance between intelligence and response time. Luna xhigh can achieve similar scores on the evals, but it takes noticeably longer. My impression is that the higher reasoning effort helps compensate for the lower base intelligence.

Cost is definitely a big factor, but latency and intelligence matter too. If I had the budget, I’d take Sol medium over Luna xhigh.

From using both on real scenarios, Sol is noticeably better at navigating around issues, exploring alternatives, and being creative when the obvious approach doesn’t work. That matters quite a bit when you’re investigating live alerts, where the path to the root cause isn’t always straightforward.

Luna as a doer, with a smarter model planning, can be a good compromise. Using sol for everything can be expensive without much gain, as a lot of steps don't need that sort of intelligence.

I choose to use Luna for most tasks because it is cost efficient, even though I get a pretty generous budget from my company.

Sometimes I will use Fable or Sol for large features/projects, or research/exploration.

I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.

Is sol better?

Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.

Going from sol to luna is a downgrade.

It is not a question, it is a fact.

> Is sol actually worth the extra cost?

Is a question only you can answer, because it has no generic answer.

Right now, for me, being able to use sol is worth the cost, but using it all the time is not.

I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.

Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.

Team will quickly shift. People hate losing access to ai.

They should try "downgrading" to people who don't need a random number generator to do their job for them.

It's silly to discuss it. Just do the evals.

Yes. If they don't like the cost then they should fire the "leader" who introduced the AI there to begin with.

Smaller models + more effort has strong diminishing returns, especially if your goal is to save money.

Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.

[flagged]

Not that I know of, but https://www.canirun.ai/ might be of use

Are you familiar with Ollama [1]? It is a particularly easy to use tool to download and run local models. They sort models recent popularity and specify size for the various quantization levels.

I would try using ~1/2 your available ram and iterate from there.

If you have 32GB of RAM, I would give Qwen 3.8 a try. All you would have to do is run "ollama pull qwen3.8:27b" then "ollama run qwen3.8:27b". If you have 16GB of RAM, I would try Gemma 4.

[1] ollama.com

Several exist actually. Try whichllm.app or fitmyllm.com.

I just asked my current LLM for that advice. Funny that they dont block it, I guess they are not very threatened.

For my anemic 6GB built-on 14GB Qwen seems to be the best bet, not great reviews but from my limited testing its pretty impressive.

[dead]

I think in general these smaller applications of AI are seriously overlooked even though they will actually add lots of economic value for companies and lots of quality-of-life features for users.

I’m doing this now. Homebrew DnD engine and mobile app chat interface. Rich 3d sim and haptics for the dice rolls. But otherwise no fancy graphics rendering. The juice is in the imaginative generate group story telling. Local real time STT on self hosted video chat, to convert the banter and side talk into next player action dice rolls. Qwen on an m3 ultra 96gb handles NPC, dm, bot party members, arc, consistency, history, memory, adventure beats. There’s even a mechanic for bards to compose an end-of-adventure ballad that compresses the shenanigans into verse.

Bootstrapped it with fable, opus or sol 5.6 when I had extra usage to burn over the last month. Playtesting it solo as both DM and PC was fun too. Character consistency and originality is remarkably compelling.

Upshot is that for the gaming hours that my friends have available, the social interpersonal gaming experience competes well for their AAA gaming time. There’s even some good async passive campaign continuation patterns that run in our signal thread to keep things going between sync sessions.

And players have requested features directly in the session transcript that have been direct implemented.

[flagged]

I read the full post. It is more about cheap models with good enough performance rather than small models. "Local" was not mentioned at all

I was wondering the same, trying to find if OpenAI has started to release the ampumt of model parametres. Basically they seem to mean ”cheaper”.

FrontierCode is prob the closest. [1] It's closed source (so no direct benchmaxxing), and it was calibrated by 20+ open source maintainers. It shows Opus 5 (medium), beating out the other reasoning levels by a large margin. i.e. Opus 5 w/ higher reasoning levels actually reduces performance. [2]

However, you'll have to gauge for yourself how closely their tasks resemble your tasks.

1/ https://cognition.com/blog/frontier-code

2/ https://cognition.com/frontiercode

Many benchmarks now measure the total cost or energy usage per completed task.

The "IQ 180" label seems to be referring to the (human) CTO of their startup.

Opus is good at deciding what to do beyond literal interpretation of the prompt (like checking assumptions before coding). Smaller models need to be told not only what to do, but how to behave.

oh-my-pi's Advisor feature can fill a lot of this gap (a smaller model told to reflect on its own output catches when it's doing a rush job and skips instructions).

I’d imagine it’s more focused on individual methods and how they connect

I wouldn't call $0.2/$1.2 "ridiculously cheap"

You should be glad to know digital ocean already offers this

I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally.

That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.

I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?

Segment had a $3B+ exit to Twilio back in 2020.

Is there evidence that small models hallucinate less than big ones?

Which vlm are you using? Ive found a reddit thread listing like 20 of them, but its hard to find any concrete info on what is worth even tryin.

I only use open models now, I really think the era of open models is upon us, big or small, but I also agree small models reached the point where you don't have hand hold them with qwen 3.8 27B

Can you elaborate on this?

And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.

My experience has been that responsiveness is value. For tasks where you need steering, responsiveness allows for better steering. For tasks which you want unattended, better models are just better.

There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.

Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.

Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.

First, smaller models are fun for hackers: you can run them locally, or run them faster.

Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.

Some reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment

It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.

It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.

There's a difference between want and need. I want a 650hp V8 supercar. I need a 150hp I4 toyota corolla. Why choose a less capable car? Because I don't want to spend 10x as much money to get groceries.

Can you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?

Cost is the point

[flagged]

This is how your comment displays on my screen, perhaps a typo:

>”@dang I really need an IP &/or account ban”