Excellent idea, most terrible execution. Comparison are completely subjective, problem space is too simplistic for today's AI, the resultstable simply ignores that a face isn't a cube (therefore, gpt 5 shouldn't have 100% success) and the retry is uneven. Also, given the random nature of AI, sampling once each model isn't very scientific.
This feels like a kid trying to do science. The will is there, but lacks experience.
It’s still science if you say “this appears to be a specimen of X” even if you don’t do a genetic test. Things don’t automatically graduate to science either by repetition or by formal verification. What makes a rubics cube is obvious enough that you can pass or fail.
So strange to write a whole post with Claude giving the best results and Grok consistently the worst, but awarding Grok the winner because at least it did the worst fastest?
GPT was the worst on the Rubik's cube
Grok did not render anything, they had to prompt it again.
I tried to one-shot the first test (the Rubik's Cube test) with LucidQuery's Swift model, to test it, as there are not much benchmarks about it and that they brag a lot about it, and I was pleasantly surprised to see it achieving a result similar to Grok 4.5 but in one shot (there is the same issue that if you scramble twice the solve button does not work anymore, but it got it in one shot).
Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.
I am 99% sure the post was written by AI
The honest takeaway: this is 100% written by an LLM.
That was the honest giveaway ... :-D
I did a quick skim and the usage of phrases like "snappy stylist" and "speed-and-value monster" were what instantly stuck out to me as AI. I decided I probably didn't need to actually read the article after that.
But just think of the amazing advancements over the past 12-18 months in fully automated slop-posting! We've reached a new high water mark.
i will give you the remaining 1% because i felt the same way.
Chat got ones were slow on Firefox mobile
Love the idea, I think more complex games would show the gap in ability better.
Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.
Isn’t the number of turns most important? Some agents take repeated input, while others can mostly one-shot what I’m looking for.
Why not wait one more day for GPT-5.6?
Also throw in GLM 5.2 for good measure
I worry that GPT 5.6 will be heavily restricted and have the same feature to fallback to another model like Claude fable 5 does all too often. That fallback shenanigans mess up actual benchmarks and I don't like it.
That will be in the Part 2 article.
If we wait for the next models, we will never test anything because there will always be another model. Like the Ai Scotsman:
> "Nay, laddie, that’s no’ the real AI Scotsman! He’s grander still! More powerful! Just wait for the next model!"
I'd like to see the comparisons with DeepSeek, Qwen, Mimo, Kimi and GLM
I just did the tests, Mimo and GLM delivered working cubes but GLM was the only visually perfect with smooth movements and great effects.
GLM is the clear winner:
https://chat.z.ai/space/t19sx5kvw631-art
Interesting that all four models converge on such similar designs, for such short prompts.
They were trained pretty much on the same data.
Barring the retry thing, n=1 on all models? Am I misreading, or is this a joke?
Variance in quality on these things is so, so high.
Tried at work , this release def a moment I will remember. My work is not the same . The model is the first model that offer exactly as I want :
For hard tasks , that needs precision I will wait and pay expensive tokens
For everything else , query data , logs, rolling out releases , I’m using grok and it’s much better vs other tools and much cheaper too .
Who the hell tries something that's been out a few hours and says "My work is not the same"?
Too nice to Grok, if there are really cost savings it should say how much each of the three demos cost so we can judge if it's worth the lower quality (probably not). The time to complete each would also be interesting.