Sol is way cheaper than Fable by the token.

We’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company.

The amount of workloads we can shift with an advisor model pattern continues to grow.

It’s seriously amazing.

Wait isn't Fable like 2x more expensive if we compare under 272k tokens

You could say the token usage is "load-bearing".

The animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel

> Answering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out.

I believe the assumption of the benchmark creators is that it's meant to measure the sequential speed that a single instance of the LLM and piece of hardware can get through all the tasks from start to finish, like running a race. As a rudimentary comparison, sort of like doing prime number calculations as a benchmark of the CPUs in one bare metal server. Of course you'd get a speedup in total number of primes searched if you ran the same software of GIMPS on 8 servers with the same hardware in parallel rather than 1 server.

Of course if you took all the individual questions in humanity's last exam and fed them in parallel into separate queries to Claude that land on separate hardware instances of the claude model you'd get a speed up. Because each question is independent and not related to knowledge/calculations that are performed in any other question it is indeed very open to speed up by breaking it into separately dispatched parallel tasks.

>Answering 2,500 independent questions is an embarrassingly parallel workload

So is calculating the total time required to answer all of the questions.

i assume they meant serially....

Presumably when they're talking about benchmarking, they're using a fixed amount of resources, like everyone else does when benchmarking?

Amadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.

Luna is incredible considering the cost. It’s very good for routine work.

Luna has impressive results on DeepSWE.

I mostly split work between Luna and Sol. If something seems simple enough I always try it with Luna first.

I feel odd to use these models, because it feels like a faster model doesn't feel that much faster if it spends reading files, making edits and running checks.

It feels too situational.

Yep, https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law

ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.

Never heard of it before, that's fucking insane.

Apparently they baked the Llama 3.1 8B model weights [0] into silicon (the actual hardware is called Taalas HC1).

I guess for the trillion parameter models this would not scale due to cost? Imagine buying GPT 6 in the form of a PCI-E card, pulling these speeds, with up to 120 cct agent sessions. It'd be beyond wild.

[0] the weights are also using some cut down small format, but HC2 will have regular FP4 supposedly, and support for 20B params on one die

Unfortunately AMD bought them, so I don't think we will get to see another release from them.

If anything, writing "7x" is actually just a poor emulation of writing "7×".

Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.

I think this is what they need to actually have a margin :)

Nothing at all. They're not designed to, so they don't. Change that, and they would.

The question is the wrong one. The right question: why aren't frontier models designed to work that way? The answer: it's slow and expensive.

The other answer: that's basically what you're selecting with "Medium", "High" and so on, how many tokens they'll blow on muttering to themselves before they get back to you with an answer. There's more to it, but not that much more.

No quality compromise/degradation is something I have had this industry, including especially OpenAI, claim multiple times in the past and I have more than once been able to verify that it was in fact not the case. Examples being gpt-3.5-turbo vs text-davinci-003, GPT-4-Turbo and all the other post training checkpoints they had under one name (which was a major bug bear for me back then witnessing degradations with no naming change, industry got better in transparent checkpoint naming since), Opus 4.6 Fast Mode (which just was faster by skipping much of the required work), etc.

Same for massive performance differences in the way providers like Cerebras, Groq, etc. have deployed models including K2.6 on Cereberas specifically. Massive deltas in tool call and overall quality despite there being far more clarity in open weight vs proprietary model deployment.

The AA suite graph with that animation is the only time in either post that absolute parity is being asserted and I'd be amazed if that was the case, but am doubtful why their phrasing is so cagey.

Why not assert full parity in writing? It "performs the same (within run-to-run variance) across all evals that Sol has been tested with" is very different to "no quality compromise/degradation", the later allowing for a lot more wiggle room and interpretation in what evals you use to assess that, what quality truly means, etc., the former meaning identical in all situations.

Could also be a language barrier here in fairness, maybe this phrasing is more iron clad than I give them credit, but especially with OpenAI, I have seen enough checkpoints asserted as unchanged in "quality" to where I am skeptical. Ironically, I never saw that with Anthropic (which has gotten far more heat for degradation accusations) while a model was deployed with one exception in mid-late April this year. Pure speculation, but believe it wasn't noticed much before "agentic coding" became more popular, because chat output is far more subjective without a rating framework vs code passing which can be an objective metric with more potential for frustration.

This just means, we once saw it run at 750 tokens per second, in this insane test config :)

It’s suspicious that they repeat “without any quality compromise” instead of just outright saying it’s the exact same model. It’s like a motel that insists they have clean beds.

The stake in the side of cerebras has always been that the economics are pretty poor.

Who knows if they will subsidizes it to mitigate sticker shock, but it's a safe assumption that it will be scarily expensive. However if you are in a "cost is no obstacle, speed is god" position, it will likely be pure magic.

> attention span

IME waiting for an agent to work through a problem is a detriment to attention span; your mind drifts to other things while you wait. Maybe you can steer several agents in a round robin instead, but then there's a cognitive tax from context switching. Faster models mean fewer gaps in focus.

There are speed thresholds that open up new use cases.

Imagine speeding up current agents 10x, you switch from directing agents to pair-vibing on the fly.

Speed them up 10x more, and you get a SOTA model capable of analyzing and rethinking your entire file in between your key strokes. That would make for one hell of an autocomplete.

Pivot over application, going from coding to anything else, and this can easily give computers features previously impossible to make. In video games, fully general characters reacting realistically to arbitrary dynamic situations. In "serious" apps, interactive work with a system that understands your goals and adapts to you on the fly. Hell, even an OS that can tell you "hey, the data you're obviously looking for is in the tab over there, now highlighted".

And that's just tip of the iceberg. I'd personally love to explore the possibilities.

I just finished waiting almost four hours for Fable to write 700 lines of code, based on my three paragraph prompt. Some speed on these harder tasks would definitely be welcome!

It also spent almost 800k tokens on these lines…

> I'm not even yet optimizing cost by batching things like review to slow local models over night

well, I am.

look at what the market thinks of CPU manufacturers and general computation now that agentic workflows have taken up, all went to the moon after being picked over in favor of GPUs and RAM for years

most computers have been idling, waiting for human input, for decades, and if there was a computationally intensive process it was offloaded to GPUs a long time ago, over the last decade, so CPUs and general processors have remained idle, relegated to just defined conditional statements to switch between tasks with no reasoning capability to occupy compute

now, there are reasoning capabilities to tell a CPU what to do (as a byproduct of the varied processes). Cerebras is not a CPU, it is a special purpose chip for inference, but is hosting LLMs that tell CPUs of all its clients what to do faster than a human can. Outside of Cerebras, LLMs are not doing much to optimize compute of the system they're affecting, as they're reading or compiling code when being used for coding, very few processes are intensive and the CPU is just waiting as if a human was using it because the LLM can't digest and output information fast enough. The CPU ecosystem is very mature for general and varied tasks, but is underutilized.

To the what: any kind of compositing or configurations that humans do, agents can do. AutoCAD, video editing, sequencing in music, all forms of media, all forms of configuration done digitally. right now they rely on snapshots to see and react, and this increases the 'framerate' per say, and rapid and relentless iteration they can do.

You can get stuff done live on a call.

Humans have human attention spans and there's a lot of study going back decades for UX design because of this. < 100 Ms is instantaneous, 100-300 ms is noticable but still responsive. At 1 second, flow.gets interrupted, 2-5 seconds, you're clearly waiting, 5-10 attention wanders and 10+ seconds, you've lost them. The 0.1 / 1 / 10 second rule comes from Jakob Nielsen's HCI work. Perceived latency matters almost as much as actual latency, which is why chat interfaces drip out/stream words instead of just dumping out the answer at the end. At 750/tok/s, for Sol grade inference, it can spend 3 seconds on thinking tokens before outputting something to the user for a better answer while still feeling usable.

Right! Not sure that will happen since Talaas was just bought by AMD, so more likely a US model. Still, we can dream.

Indeed, you could literally brute-force it that way :)

Right now the bottle neck is not the CPU, so people aren‘t spending big $ on them. But with this ultra fast mode, CPU becomes a bigger part of the bottleneck and thus Intel and AMD can charge more $$$.

I agree with the diminishing returns on spend, but worth noting that when on an Enterprise seat and paying API rates, I'd say that you can easily spend above 200/mo before seeing the curve begin to flatten

Obviously there are a ton of ways to spend money / tokens and people have different levels of experience that will put this ceiling at very different levels for different people.

If you're talking Opus pricing, it's more like $200/day.

at the corporate level you pay full price.

And a moderately heavy user dev can easily spend a few $K a month, so yeah. Not impossible, but a high cost, and the diminishing returns definitely kick in

no, because closed sourced model pricing has no relationship to its size. Thats what im saying. the inference margins are crazy, but people think the fonrtiner models must be 10T params or something because theyre expensive

I'm guessing a new $1000/month tier

It will eventually. Right now everyone is stuck on the equivalent of dialup.

No. This conveys a fundamental misunderstanding of how anything pertaining to programming works. This will never happen, ever. For example, take...

  printf("Hello, world");

  48 65 6C 6C 6F 2C 20 77 6F 72 6C 64
  48 83 EC 28
  48 8D 0D F5 0F 00 00
  E8 F0 00 00 00
  33 C0
  48 83 C4 28
  C3

A great future for us all to look forward to. Human readability gone, any changes must go through a providers API so cost+restrictions are a constant and we'll need far more tokens for even simple operations. Call me old fashioned, still like to understand the code I merge, especially since I've seen Fable and Sol make sometimes utterly egregious mistakes in simple, well scoped requests that erode any trust I can muster...

That still requires a specification language to constrain the generated code even if the implementation is produced directly.

Why? Because you are defining the implementation based on its observable behaviour rather than as a rule set to be followed.

Speed is independent of quality. LLMs are currently completely incapable of producing a good compiler. I'm not convinced that's going to change (and if it does change, we're all unemployed).