This showdown benchmark was and still is great, but an enormous grain of salt should be added to any model that was released after the showdown benchmark itself.

Maybe everyone has a different dose of skepticism. Personally I'm not even looking at results for models that were released after the benchmark, for all this tells us, they might as well be one-trick ponies that only do well in the benchmark.

It might be too much work, but one possible "correct" approach for this kind of benchmark would to periodically release new benchmarks with new tests (that are broadly in the same categories) and only include models that predate each benchmark.

Z-image was released recently and that's what /r/StableDiffusion all talks about these days. Consider adding that too. It is very good quality for its size (Requires only 6 or 8 gigs of ram).

GPT Image 1.5 is the first model that gets close to replicating the intricate detail mosaic of bullets in the "Lord of War" movie poster for me. Following the prompt instructions more closely also seems better compared to Nano Banana Pro.

I edited the original "Lord of War" poster with a reference image of Jensen and replaced bullets with GPU dies, silicon wafers and electronic components.

https://x.com/singhkays/status/2001080165435113791

It failed my benchmark of a photo of a person touching their elbows together.

So when you say "X attempts" what does that mean? You just start a new chat with the same exact prompt and hope for a different result?

This leaderboard feels incredibly accurate given my own experience.

Nano Banana has still the best VAE we have seen especially if you are doing high res production work. The flux2 comes close but gpt image is still miles away.

Stupid Cisco Umbrella is blocking you

I really love everything you're doing!

Personal request: could you also advocate for "image previz rendering", which I feel is an extremely compelling use case for these companies to develop. Basically any 2d/3d compositor that allows you to visually block out a scene, then rely on the model to precisely position the set, set pieces, and character poses.

If we got this task onto benchmarks, the companies would absolutely start training their models to perform well at it.

Here are some examples:

gpt-image-1 absolutely excels at this, though you don't have much control over the style and aesthetic:

https://imgur.com/gallery/previz-to-image-gpt-image-1-x8t1ij...

Nano Banana (Pro) fails at this task:

https://imgur.com/a/previz-to-image-nano-banana-pro-Q2B8psd

Flux Kontext, Qwen, etc. have mixed results.

I'm going to re-run these under gpt-image-1.5 and report back.

Edit:

gpt-image-1.5 :

https://imgur.com/a/previz-to-image-gpt-image-1-5-3fq042U

And just as I finish this, Imgur deletes my original gpt-image-1 post.

Old link (broken): https://imgur.com/a/previz-to-image-gpt-image-1-Jq5M2Mh

Hopefully imgur doesn't break these. I'll have to start blogging and keep these somewhere I control.

> the only model that legitimately passed the Giraffe prompt.

10 years ago I would have considered that sentence satire. Now it allegedly means something.

Somehow it feels like we’re moving backwards.

I'll be running gpt-image-1.5 through my GenAI Showdown later today, but in the meantime if you want to see some legitimately impressive NB Pro outputs, check out:

https://mordenstar.com/blog/edits-with-nanobanana

In particular, NB Pro successfully assembled a jigsaw puzzle it had never seen before, generated semi-accurate 3D topographical extrapolations, and even swapped a window out for a mirror.

I just tested GPT1.5. I would say the image quality is on par with NBP in my tests (which is surprising as the images in their trailer video are bad), but the prompt adherence is way worse, and its "world model" if you want to call it that is worse. For instance, I asked it for two people in a row boat and it had two people, but the boat was more like a coracle and they would barely fit inside it.

Also: SUPER ANNOYING. It seems every time you give it a modification prompt it erases the whole conversation leading up to the new pic? Like.. all the old edits vanish??

I added "shaky amateur badly composed crappy smartphone photo of ____" to the start of my prompts to make them look more natural.

Counterpoint from someone on the Musk site: https://x.com/flowersslop/status/2001007971292332520

I really enjoyed your experiments. Thank you for sharing your experiences. They've improved my prompting and have tempered my expectations.

I've been a filmmaker for 10+ years. I really want more visual tools that let you precisely lay out consistent scenes without prompting. This is important for crafting the keyframes in an image-to-video style workflow, and is especially important for long form narrative content.

One thing that gpt-image-1 does exceptionally well that Nano Banana (Pro) can't is previz-to-render. This is actually an incredibly useful capability.

The Nano Banana models take the low-fidelity previz elements/stand-ins and unfortunately keep the elements in place without attempting to "upscale" them. The model tries to preserve every mistake and detail verbatim.

Gpt-image-1, on the other hand, understands the layout and blocking of the scene, the pose of human characters, and will literally repair and upscale everything.

Here's a few examples:

- 3D + Posing + Blocking: https://youtu.be/QYVgNNJP6Vc

- Again, but with more set re-use: https://youtu.be/QMyueowqfhg

- Gaussian splats: https://youtu.be/iD999naQq9A

- Gaussians again: https://youtu.be/IxmjzRm1xHI

We need models that can do what gpt-image-1 does above, but that have higher quality, better stylistic control, faster speed, and that can take style references (eg. glossy Midjourney images).

Nano Banana team: please grow these capabilities.

Adobe is testing and building some really cool capabilities:

- Relighting scenes: https://youtu.be/YqAAFX1XXY8?si=DG6ODYZXInb0Ckvc&t=211

- Image -> 3D editing: https://youtu.be/BLxFn_BFB5c?si=GJg12gU5gFU9ZpVc&t=185 (payoff is at 3:54)

- Image -> Gaussian -> Gaussian editing: https://youtu.be/z3lHAahgpRk?si=XwSouqEJUFhC44TP&t=285

- 3D -> image with semantic tags: https://youtu.be/z275i_6jDPc?si=2HaatjXOEk3lHeW-&t=443

I'm trying to build the exact same things that they are, except as open source / source available local desktop tools that we can own. Gives me an outlet to write Rust, too.

> how do I make (at least) new content protected?

Air gap. If you don’t want content to be used without your permission, it never leaves your computer. This is the only protection that works.

If you want others to see your content, however, you have to accept some degree of trade off with it being misappropriated. Blatant cases can be addressed the same as they always were, but a model overfitting to your original work poses an interesting question for which I’m not aware of any legal precedents having been set yet.

We are probably entering the post-copyright era. The law will follow sooner or later.

Using references is a standard industry practice for digital art and VFX. The main difference is that you are unable to accidentally copy a reference too close, while with AI it’s possible.

A middle ground would be Chat GPT at least providing attribution.

Back in reality, you can get in line to sue. Since they have more money than you, you can't really win though.

So it goes.

> Question: with copyright and authorship dead wrt AI, how do I make (at least) new content protected?

Question: Now that the steamboats have been invented, how do I keep my clipper business afloat ?

Answer: Good riddance to the broken idea of IP, Schumpeter's Gale is around the corner, time for a new business model.

my question to your anecdotal: who cares? not being fecicious, but who cares if someone reproduced your stuff and millions of people see your stuff? is the money that you want? is it the fame? because fame you will get, maybe not money... but couldn't there be another way?

> however im not sure if these are true uv maps

I can tell you with 100% certainty they are not. For example, Crash doesn't have a backside for his torso. You could definitely make a model that uses these as textures, but you'd really have to force it and a lot of it would be stretched or look weird. If you want to go this approach, it would make a lot more sense to make a model, unwrap it, and use the wireframe UV map as input.

Here's the original Crash model: https://models.spriters-resource.com/pc_computer/crashbandic... , its actual texture is nothing like the generated one, because the real one was designed for efficiency.

> however im not sure if these are true uv maps that is accurate as i dont have the 3d models itself

also in the tweet

> GPT Image 1.5 is **ing crazy

and

> holy shit lol

what's impressive if you don't know if it's right or not (as the other comment pointed out, it is not right)

My pet theory is that OpenAI screwed up the image normalization calculation and was stuck with the mistake since that's something that can't be worked around.

At the least, it's not present in these new images.

Meta's codec avatars all have a green cast because they spent millions on the rig to capture whole bodies and even more on rolling it out to get loads of real data.

They forgot to calibrate the cameras, so everything had a green tint.

Meanwhile all the other teams had a billion macbeth charts lying around just in case.

Not really, but there's a number of theories. The simplest one is that they "style tuned" the AI on human preference data, and this introduced a subtle bias for yellow.

And I say "subtle" - but because that model would always "regenerate" an image when editing, it would introduce more and more of this yellow tint with each tweak or edit. Which has a way of making a "subtle" bias anything but.

My pet theory is that this is the "Mexico filter" from movies leaking through the training data.

I never heard anything concrete offered. At least it's relatively easy to work around with a tone mapping / LUTs.

There's definitely an analysis on the net somewhere, can't remember the details though.

maybe their version of synth-id? it at least helps me spot gpt images vs gemini's

I'm guessing that it was intentional all along, as no other models exhibit this behavior. It was so it could be instantly recognized as ChatGPT

Colloquially called the urine filter

Not always, it started at a very specific point. Studio Ghibli craze + reinforcement learning on the likes.

It's strange to me too, but they must have done the market research for what people do with image gen.

My own main use cases are entirely textual: Programming, Wiki, and Mathematics.

I almost never use image generation for anything. However its objectively extremely popular.

This has strong parallels for me to when snapchat filters became super popular. I know lots of people loved editing and filtering pictures but I always left everything as auto mode, in fact I'd turn off a lot of the default beauty filters. It just never appealed to me.

It would creep me out if the model produced origami animals for that prompt.

I can actually imagine actors selling the rights to make fake images with them.

In late stage capitalism you pay for fake photos with someone. You have chat gpt write about how you dated for a summer, and have it end with them leaving for grad school to explain why you aren't together.

Eventually we'll all just pay to live in the matrix. When your credit card is declined you'll be logged out, to awaken in a shared studio apartment. To eat your rations.

The arena concept doesn’t work for image models due to watermarks.

I haven't really kept up with what Midjourney has been doing the past year or two. While I liked the stylistic aspects of Midjourney, being able to use image examples to maintain stylistic consistency and character consistency is SO useful for creating any meaningful output. Have they done anything in that respect?

That is, it's nice to make a pretty stand-alone image, but without tools to maintain consistency and place them in context you can't make a project that is more than just one image, or one video, or a scattered and disconnected sequence of pieces.

This is surprising. Is there a gallery of images that illustrates this?

That's the opinionated vs user choice dynamic. When the opinions are good, they have a leg up

That's because it's a two-way street, a multi-modal model that is highly proficient at real-life image generation is also highly proficient at interpreting real-life image input, which is something sorely needed for robotics.

This is a cultural flaw that predates image generation. Even PG has made statements on HN in the past equating “rendering skill” with the quality of art works. It’s a stand-in for the much more difficult task of understanding the work and value of culture making within the context of the society producing it.

Yeah I just tried it and got a 500 server error with no details as to why:

  POST "https://api.openai.com/v1/responses": 500 Internal Server Error {
    "message": "An error occurred while processing your request. You can retry your request, or contact us through our help center at help.openai.com if the error persists. Please include the request ID req_******************* in your message.",
    "type": "server_error",
    "param": null,
    "code": "server_error"
  }

  POST "https://api.openai.com/v1/responses": 400 Bad Request {
    "message": "Invalid value: 'blah'. Supported values are: 'gpt-image-1' and 'gpt-image-1-mini'.",
    "type": "invalid_request_error",
    "param": "tools[0].model",
    "code": "invalid_value"
  }

My Enterprise account got an email 1.5 hours ago that it is available in API but my other accounts haven't gotten any email yet

It's a staggered rollout but I am not seeing it on the backend either.

Makes you wonder what's really meant when we talk about progress.

all I can hope for is that a new industry or reliable ecosystem of vetters of real human talent will emerge. Are you really as good a writer as you claim to be? Show us the badge. That or AI firms have to be forced to 'watermark' all their creative outputs, and anyone misleading the public/audience should be punishable by law.

Not if you strip the EXIF data. Also, it will strip the star watermark and SynthID from Gemini if you paste a Nano Banana pic in and tell it to mirror it.

I know OpenAI watermarks their stuff. But I wish they wouldn't. It's a "false" trust.

Now it means whoever has access to uncensored/non-watermarking models can pass off their faked images as real and claim, "Look! There's no watermark, of course, it's not fake!"

Whereas, if none of the image models did watermarking, then people (should) inherently know nothing can be trusted by default.

There are ways to tell if an image is real, if it's been signed cryptographically by the camera for example, but increasingly it probably won't be possible to tell if something is fake. Even if there's some kind of hidden watermark embedded in the pixels, you can process it with img2img in another tool and get rid of the watermark. Exif data, etc is irrelevant, you can get rid of it easily or fake it.

https://help.openai.com/en/articles/8912793-c2pa-in-chatgpt-...

It doesn't mention the new model, but it's likely the same or similar.

I think society is going to need the opposite - cameras that can embed cryptographic information in the pixels of a video indicating the image is real.

I ran exiftool on an image I just generated:

$ exiftool chatgpt_image.png

...

Actions Software Agent Name : GPT-4o

Actions Digital Source Type : http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgori...

Name : jumbf manifest

Alg : sha256

Hash : (Binary data 32 bytes, use -b option to extract)

Pad : (Binary data 8 bytes, use -b option to extract)

Claim Generator Info Name : ChatGPT

...

It's pretty common for infographics to be wrong. The people making them aren't the same people who know the facts.

I'd especially say like 100% of amateur political infographics/memes are wrong. ("climate change is caused by 100 companies" for instance)

We don’t, but most Marketing departments salivate for them.

the next generation of humans growing up will not even care whether media is real or not any more. The saturation of AI content and FUD around real content is going to blur the lines to the extent that there's no point even caring about it. And it's an intractable problem.

hopefully this leads to greater importance of seeing things with your own wetware.

I didn't have a good experience with NB. I am half Indian. Immediately changes my face to a prototypical Indian man every time I use it.

This tool is keeping my look the same.

Nano Banana became useless for image edits once the safety training started rejecting anything as “I can’t edit some public figures.”

My own profile picture? Can’t edit some public figures. A famous Norman Rockwell painting from 80 years ago? Can’t edit some public figures.

Safety’d into oblivion.

You're just not describing what you want properly. Looks fine to me. Clearly you have something else in mind, so I think you're just not describing well. My tip would be to use actuall illustration language. Do you want a wide angle shot? What should depth of field be? Oil painting print? Ink illustration? What kind of printing style? Do you want a photo of the book or a pre-print proof? What kind of color scheme?

A professional artist wouldn't know what you want.

You didn't even specify an art style. 1970s sci-fi novel cover isn't a style. You'll find vastly different art styles from the 70s. If you're disappointed, it's because you're doing a shitty job describing what's in your head. If your prompt isn't at least a paragraph, you're going to just get random generic results.

counterpoint: we should make energy abundant enough that it really doesn't matter if sama wants to generate gpt-shirt ads or not.

we have the capability, we just stopped making power more abundant.

It's a joke about one of his old fits.

https://x.com/coldhealing/status/1747270233306644560

If memory serves me, Nano Banana allows generating/editing photos of children. But anything that could be misinterpreted, gets blocked, even absolutely benign and innocent things (especially if you are asking to modify a photo that you upload there). So they allow, but they turn on the guardrails to a point that might not be useful in many situations.

I was able to generate photos of my imagined children via Nano Banana

I haven't seen that, meanwhile gpt-image-1.5 still has zero-tolerance policing copyright (even via the API) so it's pretty much useless in production once exposed to consumers.

I'm honestly surprised they're still on this post-Sora 2: let the consumer of the API determine their risk appetite. If a copyright holder comes knocking, "the API did it" isn't going to be a defense either way.

on top of the prompt. It has a weird layout; I had to scroll up to see it.

The endgame is to make money during the hype and then cash out before it crashes.

Haven’t their prices stayed at $20/m for a while now?

Nano banana pro is almost as good as seedream 4.5!

I was reading a trend report on art and it seems like collage, squiggly hand drawn text, and lots of intentional imperfections are becoming popular. I'm not sure how hard it is for AI to recreate those, but it is nice to see people trying to do more of what AI struggles with.

Good luck getting it developed unfortunately. I have to ship it off now, there isn’t a single local spot in my city that will develop anymore

We'll merge that thread hither to give some other submitters a chance.

Among those, Photoshop.