And being worst than previous model...

"...The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one..."

I have been claiming that I don't think Chinese AI companies are benchmaxxing harder than American AI companies, which has gotten mixed reception: sometimes people agree, sometimes they disagree.

It seems I was wrong. American AI companies might actually be benchmaxxing harder.

"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration."

I guess there is no way this can happen without benchmark being part of the training data??

I was just thinking they need to mark each model per benchmark as "model released before the benchmark was released" and "model released after the benchmark was released".

https://xcancel.com/quietnning/status/2080786711861407883

I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

It could be that the set of your day-to-day workload which could feasibly be accelerated by AI just happens to be saturated around Opus4.5, but you can still see lots of “reasoning” which makes you think the model is more performant in the first days of use. That’d mean you couldn’t perceive any meaningful difference in more powerful models’ results, even though you can see a difference in the raw output due to the length of reasoning traces leading up to the result.

So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.

It's called frog boiling.

We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age.

If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.

5 seems incredibly smart to me in my conversations today about some pretty niche ideas in.longitudinal modeling. It.felt.like a big step up.from 4.8, to me

Some 20 years ago, the telecommunications sector in Germany was liberalized. Many telephone card providers entered what had previously been a barely competitive market. They advertised their products with aggressive claims like: “Buy our €10 top-up card and get 660 minutes to destination X.”

For the first few weeks, they would actually provide those 660 minutes to establish trust in their cards. But after a while, they would quietly start reducing the number of minutes on subsequent top-ups—say, from 660 minutes down to only 300. They wouldn’t do this for every card, so it was difficult to prove. Instead, they relied on averages across their customer base to make the economics work.

Lately, I’ve found myself wondering whether something similar may be happening with frontier AI models. Companies launch with an exceptionally strong model and generous compute limits to build adoption. Once the model is established as a market leader, the incentives change, and users may start perceiving the service as becoming more constrained or less capable over time.

I don’t have evidence that this is what’s happening with Anthropic—or with any other AI company. It’s simply a pattern that the current situation reminds me of.

Going to call it user error if you find Opus 4.5 better than 5, sorry.

Same, like I prefer 5.3 codex over the “stronger” models.

Well, what kinds of things do you see Opus 4.5 completely fail at? Maybe those are not the ones that newer models have improved on.

I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.

Enshittification.

I've worked with these systems for four years now and they have not meaningfully improved in that time frame.

We still have:

- statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system)

- Math completely fails in longer contexts

- "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion

- smearing of properties between logically distinct objects (a red ball and a green cube can quickly become a red cube and a green ball)

Many models are much cheaper through their subscriptions' included usage. That could be what's happening here.

Claude gives you something like $5000 of tokens on a $200 plan.

Great comparison! We only have to take into account that applying lipstick well bears no consequences, but applying it poorly (i. e. new model tanking the benchmark) could amount to potentially losses of billions of dollars for the pig-breeders (AI labs).

There are private datasets, and 3rd party providers of these models. Fable doesn’t have a datapoint here because of its particular data retention policy. Even if you don’t trust AWS, do you think Opus on AWS is also sending the data to Anthropic? Do you have any evidence?

It's like we've come full circle:

First people practiced L33t3cod3 problems for interviews

Then people built AIs to build software

And now the AIs are studying L33t3cod3 problems

It is RLVR, Not a puzzle, Not leetcode

How believable is this benchmark? EG maybe opus was training on this? (You can try to identify the IP of wherever previous ARC questions came from)

Why? 30% is passing the first two problems only, which are really very simple.

,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''

Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.

I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.

My own private benchmark shows the same thing.

There are other benchmarks for that

I also noticed that Opus 5 doesn't show a corresponding gap on the ARC-AGI-2 leaderboard. There is a significant increase in performance between Opus 5 and 4.8 on ARC-AGI-2 though.

> it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems

We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).

> Why is Fable not on here?

Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.

I don't know why exactly, but Fable has felt the most human LLM to arrive.

Because they're cheating.