renehsz
15 hours ago
Are the pelicans still a useful benchmark at all?
I would guess that
1) they're probably in the training data, and
2) models have gotten too good at this task to compare them meaningfully (I tried doing a blind test and couldn't reliably tell 6.1 Astra, Sol, and Terra apart at reasoning levels that approximately match in cost)
basch
5 hours ago
counterpoint https://static.simonwillison.net/static/2026/openai-pelicans...
look at the bicycle not the pelican. only astra low made a bicycle that resembles a functional bicycle in the low/none category. luna needs xhigh to make a bike. the legs are still wrong, on the wrong side of the bike, in all but like four of the outputs.
andrewstuart
12 hours ago
>> Are the pelicans still a useful benchmark at all?
If you are using these models for software development, then I ask you this question…… when you are interviewing job candidates to join your team, do you ask them to write a program to draw pelicans?
Thus you have your answer.