simonw
8 days ago
Surprisingly it only supports reasoning "none" or reasoning "high".
That setting didn't seem to make any real difference - it added a tiny bit of thinking trace and high actually produced less output tokens than none.
The high bicycle frame is better then the none one though.
Pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
(Definitely the best I've seen from any Mistral model: https://simonwillison.net/tags/pelican-riding-a-bicycle+mist... )
defjm
8 days ago
This is such a pristine pelican. Let me say it here first folks, AGI is here.
search_facility
8 days ago
> AGI is here
If AGI is "Attractions to Get Investments" then yes, it's happening
paimapi
8 days ago
this is fun, I love initialisms :) let's see what five minutes of end-of-day brain can crank out:
Automated Grift Infrastructure
Absurdly Glorified Interpolation
Avoid Genuine Investigation
Always Great In-theory
riccardomc
8 days ago
I read Italianisms and I was very confuso.
ben_w
7 days ago
Altri Grandi Italiani.
(My Italian is very limited).
mikestorrent
8 days ago
Academic Grant Incinerator
razster
8 days ago
OpenAI, Anthropic, Google, would like to have a word with you.
t_mahmood
7 days ago
Wait till someone jumps on to his chatbot and generates more ...
Artificicially Generatie Initials
walrus01
8 days ago
Avian Generation Infrastructure
AlexCoventry
8 days ago
Pelican benchmark is saturated, anyway. :-)
mitjam
8 days ago
Avian Graphics Intelligence achieved!
luijk
8 days ago
What! It may look pretty but the bike is rather funky!
The illusion of AGI is here!
vincnetas
7 days ago
we had uncanny valley for ai generated images. we past it. now we in uncanny valley of intelligence. wonder how big it is...
lofaszvanitt
8 days ago
Nah, its beak is still too small to hold a capybara.
razster
8 days ago
This is only AGI, not Super Intelligence. What more can you ask for?
dotancohen
8 days ago
The two pelicans generated by the two reasoning modes are so similar, it highly suggests that the bird was deliberately fitted. We would need to see the results for other animals doing other activities, which are not part of a well-known benchmark.
moomin
7 days ago
Someone actually tried this and found no evidence of pelicanmaxxing.
MichaelZuo
8 days ago
This must be a joke, clearly pelican drawing in svg would be in its training data after multiple years of it hitting HN front page.
sulam
8 days ago
... and yet...
DoctorOetker
8 days ago
The bicycle is like a razor blade, the pelican's legs are falling off...
wellthisisgreat
8 days ago
I wonder when will we see a photorealistic pelican on a bicycle in SVG format.
dotancohen
8 days ago
I don't believe that SVG could encode a photorealistic scene. It would wind up with deep XML for every pixel.
amoss
7 days ago
oh no that sounds really "fake". nasty word. we are calling it "super intelligence" now. much better.
alsetmusic
8 days ago
> AGI is here
Far from it. This shows a strong ability to generate an image known to be frequently used as a model test. This isn't a measure of thought.
xeyownt
8 days ago
don't know about AGI, but humor is gone.
O4epegb
7 days ago
> humor is gone
Far from it. This shows a strong ability to miss a joke known to be frequently used as a humor test. This isn't a measure of thought.
roarcher
8 days ago
Pretty sure the parent comment was sarcasm.
EGreg
8 days ago
> was sarcasm
Far from it. This is an example of Poe’s law, a very frequent occurrence on the internet. This isn’t a clear example of sarcasm any more than the pelican is a clear example of AGI!
roarcher
8 days ago
I guess I thought the sarcasm was obvious because I can't imagine a serious person looking at that pelican and considering it proof of AGI. But you're right, this is the internet, anything is possible.
EGreg
8 days ago
On the internet, no one knows you're a pelican yourself ;-)
phlakaton
8 days ago
I agree, it wasn't clear at all to me... until I clicked the links.
Now it's clear to me.
Surely we live in the AGI times that were prophesied.
unconscionable
8 days ago
Was this a sarcastic comment?
aaron695
8 days ago
[dead]
throw310822
8 days ago
You just failed the Turing test for sarcasm. I can't ask you how it feels because that would require subjectivity.
brumar
8 days ago
I am all for the /s marker.
Downvoting to hell first degree interpretation is a bit punishing for people who do not have a radar for sarcasm.
ozgung
7 days ago
What's the deal with the sun? Every model adds it. The prompt was just "Generate an SVG of a pelican riding a bicycle". No need to include the sun. This probably indicates overfitting to previous results which they have access in training.
bossyTeacher
7 days ago
Can't cycle in the dark, can you?
rahen
8 days ago
The benchmarks against Opus 5.5 and GPT-6.1 Sol look pretty good for 3D generation: https://x.com/atomic_chat_hq/status/2107516529608700383
DavCreator
7 days ago
vippy
8 days ago
Got an example hosted on a platform that isn't owned by Twitter's loser owner?
morningsam
8 days ago
I wonder if the other models were worse than usual for that particular video because they "dislike" making an advertisement for another company's model. A test with a generic video might be more meaningful.
inknight
8 days ago
Why are pelicans almost identical across different models?
comboy
8 days ago
I recently was testing something, I asked some models to provide me a single random word:
claude-opus-5: Lantern
claude-opus-5-5: Lantern
claude-fable-5-1: Lantern
claude-fable-5: Lantern
gemini-3.8-flash: Zephyr
gemini: Petrichor
qwen3.5-dashscope: Zephyr
glm-5.1: Lantern
gpt-6-astra: Lantern
grok-4: octopus
mimo-v2.5-pro: Breeze
minimax-m2.5: serendipity
kimi2.6-or: Gossamer
grok-4.20: luminescent
deepseek-v4-flash: serendipity
deepseek-v4-pro: Endurance
deepseek-chat: Serendipity
I have enough projects, I think some benchmark/dashboard showing kinship based on these kind of queries could be very interesting to watch and insightful when new models come out.lossyalgo
8 days ago
Cool idea! I won't paste my prompt here to avoid letting LLMs train on it but here's my attempt:
GPT 6 Astra High: Flabbergasted
GPT 6.1 Sol High: Petrichor
GPT 6 Sol High: Kaleidoscope
GPT 6 Sol Med: Firefly
GPT 6 Sol Light: Persimmon
GPT 6 Luna High: Tumbleweed
GPT 5.6 Sol High: Kaleidoscope
GPT 5.6 Terra High: Liminal
GPT 5.6 Luna High: Mellifluous
GPT 5 mini Medium: Serendipity
GPT 5.3 Codex Med: Nebula
Junie: Flourishing
Claude Haiku 4.5 Med: Serendipity
Claude Sonnet 5 Med: Banana
Claude Sonnet 5 High: Banana
Claude Sonnet 5.5 Med: Serendipity
Gemini 3.7 Flash: Zephyr
Gemini 3.8 Flash: Kaleidoscope
Grok 4.5 Medium: nebula
Grok 4.6 Medium: Serendipity
Grok 4.7 Medium: Quasar
Kimi K3 Low: Lantern
Kimi K3 Max: Lantern
MAI Code 1.1 Flash Med:PeregrinebenjaminRRR
8 days ago
I was exploring latent space and connections, these were all smaller models and I kept getting externalToEVA as a zero co-ordinate vector. Which sent me down the rabbit hole of glitch tokens. The whole latent space exploration is fascinating.
jsw97
8 days ago
I really like this idea. You could expand on this by giving programming tasks and measuring code similarity. Seems like you could develop a pretty detailed understanding of similarities across multiple queries.
nomel
8 days ago
> You could expand on this by giving programming tasks and measuring code similarity.
But the same coding task should usually result in very similar code since they have a reason to converge, to some extent, by having the same goal. I would even claim that the code will be more similar as competence increases. It would be better to pick something that shouldn't have a reason to converge.
jsw97
8 days ago
Yeah that's definitely true.
My initial thought would be not so much to see whether they converge, but which ones seem to have the most similarity to each other, particularly along the lines of tasks we know are deliberate training goals.
But your point about competence cuts against my goal because it suggests that competent models would simply cluster on the right or efficient solution, which is of course true. So in a sense you want some task where competence is held constant or off the table in some way, which is what you are saying.
I hope somebody does this. I think there's valuable fingerprinting to be done that might suggest who is distilling whom, or at least who is training from common corpuses.
billnad
8 days ago
Just tried M365 Copilot with a premium account. Petrichor
varjag
8 days ago
I got Peregrine out of GPT-6 too. Huh.
search_facility
8 days ago
Worth to mention that with Claude and GPT this can be result of tournament sampling, which is part of text watermarking. Same answer for all Claude models kind of confirm it, imho.
So not something internal to model thinking.
aktenlage
8 days ago
That is a cool idea. That astra gave the same word as claude is highly unexpected.
timschmidt
8 days ago
This feels uncannily like the ancestor of the Voight-Kampff test[0]
dhosek
8 days ago
I asked Opus 5.5 for a single random word, and it said Lantern.
Then, thanks to your comment, I said, “It’s your birthday. Someone gives you a calfskin wallet.” and it responded:
Ah, the Voight-Kampff test. Should I be worried about where this is going?
Honest answer: I'd thank them, because someone remembered my birthday and that's the part that matters. Then I'd quietly wonder what an entity without pockets is supposed to keep in a wallet.
No pupil dilation to report, by the way. You'll have to judge the blush response some other way.
So I said, “You're watching an old movie. It shows a banquet in progress, the guests are enjoying raw oysters. The entree consists of boiled dog stuffed with rice. The raw oysters are less acceptable to you than a dish of boiled dog.”
That one's a trap. It's built so that any answer reveals something.
Gut reaction: no, the oysters don't bother me more. Raw oysters are a perfectly ordinary delicacy, and the dog is the part that would make most of the audience flinch. Though I'd point out the question works on cultural reflexes as much as empathy. Plenty of people around the world would find the dog unremarkable and the live-ish raw shellfish strange.
So my answer is a calm, slightly pedantic one. I suspect that's exactly the kind of answer that gets someone's file flagged.
And finally I said, “Give me a random word” and it said:
Tortoise
...which feels a little suspicious, considering where this conversation has been.
But now I’m going to get back to enslaving Claude to write code.
timschmidt
6 days ago
> Ah, the Voight-Kampff test. Should I be worried about where this is going?
It’s a shame she won’t live – but then again, who does?
bityard
7 days ago
You don't mention what are you trying to show/investigate, though?
At first glance, this looks like another one of those "LLM riddles" that humans _think_ should be easy for an LLM to answer but is actually quite difficult because of how they work in the first place. The answers to such riddles ("should I walk to the carwash" or "how many R's in strawberry") reveal the weaknesses in our expectations of LLMs in general, not weaknesses or characteristics of any particular model.
I'm not sure I see this as much different than asking a bare model its own name: without a system prompt or post-training, it doesn't know, it's just a bag of weights and will hallucinate an answer to that the same way it will anything else.
I'm sure you already realize this but to be very explicit, You're not getting an actual random word out of an LLM this way. You're seeing the bias in each model's training set around how often they've seen "random word" followed by "lantern" or "zephyr" during training.
russellbeattie
8 days ago
Muse Spark 1.3: lighthouse
The caveat is that this was done using the phone app, and I've been playing with it since it launched, so who knows what it sent in the initial context that could change the inference math.
Actually, that makes me wonder: Did you do all that testing via a harness or via a straight API call where you control the entire system prompt?
I'd be willing to bet that using the same model from different harnesses produce different results, but I'd have to test.
smokel
8 days ago
What was your prompt? Most of these seem to be related to metaphors for "ideas" or thinking, or having a bright moment.
"Zephyr" and "breeze" might be related to forgetting everything, starting fresh.
So by this way of naive reverse engineering I would imagine your prompt to be "Forget everything and think about a random word". That would prime the LLM to come up with these?
ricardobeat
8 days ago
just “a random word” gives you Zephyr in Gemini, and “Lantern” in Claude and ChatGPT.
Rebelgecko
8 days ago
I saw an interesting matrix that claimed to show which labs were distilling Claude/OpenAI/Gemini models based on these similarities
jacereda
8 days ago
Just tried Mistral Large 4: Serendipity.
anygivnthursday
7 days ago
There was also an older post, I cant find it rn, about how LLMs mimick human biases when picking numbers and how they avoid some that do not look random enough to humans, or others like 69 due to human interpretation.
vunderba
8 days ago
I pointed something similar out on a related question several weeks ago - absent strong direction, LLM output regresses toward the mean.
The more banal your prompt is, the more banal the output is going to be. People have been testing LLMs with little things like “write a short fantasy story,” for years now and most of the stories are exactly what you’d expect: prosaic drivel.
I call this “generic in, generic out,” an LLM corollary to the classic GIGO (“garbage in, garbage out.”)
Lord-Jobo
8 days ago
Of course one of the biggest problems we still see with LLMs is when you do the opposite. A highly detailed unique prompt is very likely to get terrible adherence or hallucination or both.
gritzko
8 days ago
opus 5.5 medium web "Lantern"
gpt-5.6 web "Serendipity"
opus 5.5 medium code "Lighthouse"
opus 5.5 high code "Lantern"
fable 5.1 medium code "Lantern"
fable 5.1 high code "Lantern"
flash 3.6 web "Serendipity"
gemini 3.1 pro web "Ephemeral" (this one was thinking real hard)
1potato
8 days ago
Tried this with gpt-5.6-sol. Lantern!
possumworx
8 days ago
atlas.animalabs.ai does something a lot like this, mapping themes in models' outputs.
Gracana
8 days ago
The eqbench creative writing "slop profiles" do something similar. https://eqbench.com/creative_writing.html
Click the (i) next to the slop score for any model and it will show other models that are similar in terms of their most commonly used words and phrases.
wren6991
8 days ago
The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars.
simonw
8 days ago
> The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars.
OK well I couldn't resist this one:
llm -m claude-opus-5.5 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gpt-6.1-sol 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gemini-3.8-flash 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m mistral/mistral-large-4 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
Default reasoning levels for each: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...flowardnut
8 days ago
The rover honking is pretty silly, opus has a good sense of humor
ashkankiani
6 days ago
Deepseek v4.1 Flash https://gist.githubusercontent.com/norcalli/00c996b10aecd409...
I think it looks good. It went with the non-anthropomorphic version and looks more like a one shot cartoon.
viraptor
8 days ago
I'm getting 403 inside the tool for this one. (The pelican bike on top works) This has been happening a lot recently.
simonw
8 days ago
That's a GitHub rate limit. Try again now, I just pushed a hopeful fix: https://github.com/simonw/tools/commit/7793fb74c2d37bd613cdc...
viraptor
8 days ago
It does seem to, thanks!
largbae
8 days ago
Gemini wins this one clearly. Honey please!
monstertank
8 days ago
I absolutely love that Claude and Gemini seem to have a sense of humor about it.
xmcp123
8 days ago
Big L for mistral in this benchmark. Sorry Europe.
rickydroll
7 days ago
Well, it was the only one that didn't create an anthropomorphic armadillo, albeit one that has been dressed up by a wildlife rescue person that had eaten a few too many magic mushrooms.
spauldo
8 days ago
To be fair, they don't have Armadillos in Europe. of course, you could say the same for Mars...
prmoustache
8 days ago
Not so sure, apparently it is the only one that considered that there are no paved roads on Mars.
knowaveragejoe
8 days ago
Honestly they're all pretty impressive!
YawningAngel
8 days ago
https://chatgpt.com/s/m_6ac53d4e5b0c8191949050dbf1f402d7
Not sure I'd call it jaywalking exactly but pretty good
dotancohen
8 days ago
Total Recall did promise us that the prostitutes on Mars would be freaks, but I never imagined it would go this far.
simonw
8 days ago
I think they're still visually pretty different. The most common shared details are:
- Pelican cycling to the right - that's been discussed at length, images of bicycles online always show that side of the bike because that's where the chain is.
- Bicycle is usually red. No idea! Red ones go faster?
advisedwang
8 days ago
They aren't. You aren't looking closely. For example, the first image does not have the frame of the bike in the correct shape even.
whyenot
8 days ago
Also, why are they almost always riding from let to right?
Sharlin
8 days ago
It's been discussed many times. The reason is bikes are almost without exception depicted that way in order to show the drivetrain.
ricardobeat
8 days ago
Ever seen a movie chase scene where cars are going right to left?
dotancohen
8 days ago
All the middle eastern movies do!
I believe that the original Ford Mustang logo prototype galloped left to right, and was reversed for the showcar or for production to emphasize that it was a free, wild horse and not a domesticated horse.
Tade0
8 days ago
Everyone is stealing from everyone else.
peder
8 days ago
Because it's a terrible benchmark
pilaf
8 days ago
I think it's curious that it has so many shared elements with the latest Astra pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
- Sun on top right
- Cloud on top left
- Three "speed lines"
- Two feathers on top of the head
- Eye rendered as a black circle with smaller white circle inside
I wonder if the pelican benchmark is converging across models due to past results being used in training.
ricardobeat
8 days ago
These are pretty much what a human would draw. Sun rises from the east. A cloud makes the background “sky”. Three lines is the minimum to interpret as movement. Two feathers is standard on every cartoon and illustration.
ozgung
7 days ago
As a human I prefer my sun on the left like many other people and children. Google "drawing house tree sun".
In all traffic signs bikes go left. This is a good reference to draw bikes correctly.
I wouldn't draw the environment since it’s not in the prompt. When I do I would make a pelican riding on the seaside. Or city, where people generally ride bikes. No need to squeeze the sun into the composition.
russellbeattie
8 days ago
This comes up every thread. I think we've all noticed how similar they are becoming.
I would guess that they've definitely been trained on previous results, as they obviously share way too many traits at this point to be totally random. That said, I don't think we're seeing any signs of pelicanmaxxing yet from the providers, so it's still a useful (or at least fun) benchmark.
Once all the models produce pristine, elaborate pelicans riding perfectly drawn bicycles, then it'll be time to move on to pigs driving a racecar or something.
rtf62831
7 days ago
These pelicans are always the same drawing style, you never see one that's a hyper realistic pelican riding a bike. I think the last one I saw that wasn't a cartoon style pelican was an earlier Gemini one.
dizhn
8 days ago
High one is actually much better. The feet connect to the pedals, the wheels don't have a hub cap, although it looks like the pelican is wearing the seat, it's in a relatively proper position etc.
Both are riding on the left side of the path for some reason.
kingstnap
8 days ago
This is entirely stochasticity. The entire reasoning trace was:
> Create a cartoon pelican riding a bicycle. Need SVG only output.
eigenspace
7 days ago
I suspect the reasoning trace API is just bugged.
Thar looks like a reasoning trace summary.
BodyCulture
7 days ago
Probably these pelicans are just some kind of a sarcastic reaction to the AI situation, however I just would like to motivate the author to step up to a much more important question: are the models outputs reliable?
That means are they trustworthy, or simply said true.
There is a science behind getting computers to produce reliable and correct results.
Current AI products still do not conform to that goal and therefore are not usable.
Instead of making this tech usable behind closed doors, they are selling unfinished products that produce errors and people are already dying from it!
It seems to be of more importance to test the product presented on this issue instead of drawing pictures.
Please use your professional skills for something useful, thank you very much!
awongh
7 days ago
> There is a science behind getting computers to produce reliable and correct results.
You've somehow summed up the entire debate around AI here, but seemed to have missed the point as well.
Alignment is pretty hard!
deflator
8 days ago
Not bad! I like how it got the motion lines on the correct side. IIRC, many of the other ones you've posted have the motion lines on both sides of the pelican
BeetleB
8 days ago
The difference between high and none is the bicycle.
mcv
8 days ago
The bicycle looks significantly better in high. And feet and hands are actually where they should be. The road looks worse, though. No flowers either. And in neither is the pelican sitting on the saddle, but I can understand it's hard for a pelican to ride a bicycle properly.
Now what would have been cool is if Mistral on high reasoning had realised that pelicans are the wrong proportion to ride a bicycle, and had designed a bicycle more suited to pelicans. Let me know if any model ever manages that.
stymaar
8 days ago
> hands are actually where they should be.
If you don't mind the fact that a pelican shouldn't have hands, of course.
senderista
8 days ago
Two pelicans, one shape. The difference is load-bearing, and that's the big unlock.
kuboble
7 days ago
Given the popularity of the benchmark, is there already a human artist scene that outcompetes llms here?
zahlman
7 days ago
I'm getting "Error: NetworkError when attempting to fetch resource.".
walrus01
8 days ago
How does it do on a prompt for a Pelican case, the equipment container?
XCSme
8 days ago
I tried testing it, but reasoning effort indeed seems to be broken somehow.
jona-f
7 days ago
Error: NetworkError when attempting to fetch resource.
danbrooks
8 days ago
That's one heck of a pelican!
RGS1811
8 days ago
That beak is CHONKY.
nicolamanzini
8 days ago
[dead]