https://gist.github.com/simonw/b6149a49d327164d67d62c3d12992...

> Aside from reading identically forwards and backwards down to the letter

No it doesn't.

Here's quite an impressive follow-up. I have a tool which knows how to render Markdown documents with embedded SVG content - I use it for the pelican test.

Since this transcript has HTML in it, I decided to upgrade that tool to also render HTML.

I set Gemini 3.8 Flash the task, using my own VERY shonky coding agent tool (llm-coding-agent) - and it did a solid job.

So now you can see the "cool thing in html" rendered within the Markdown document using code that Gemini 3.8 Flash also wrote: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Transcript where it built that is here: https://gist.github.com/simonw/3e36b98292dfdc1b3baff158faa74...

Definitely cool.

I noticed it felt a little janky on my PC despite being "60 FPS"...then I noticed the "60 FPS" is hard-coded into the HTML.

Focus on speed and being OK with temporarily being #3/4 in intelligence might be the counterintuitive approach which makes Google win long term (whether accidentally or strategically). Can't wait to try Gemini Pro later this year!

I saw your username, clicked the link without reading, and was very confused to see a cosmic vortex and not a pelican.

> this thing is really good at HTML JavaScript is pretty exciting.

I would hope the people who make one of the most used JS engines in the world are capable of making a model good at JavaScript ;)

OK, but what about a pelican in a bycicle.

LLM: produces a toolbox of an id and a clock

User: use them both

Made me giggle.

it's such a weird split how most AI companies are trying to be the best, but Google really has a different mission statement. they already have users. lots of users. they need to be working on building models they can deploy and use with the most number of people, as they already have the users.

i don't know if Gemini models per se are fully is in line with that purpose, but the results we see keep seeming to be in-line with that split-of-focus.

Mission accomplished. That's both cool and fast.

I've been benchmarking[1] models for trip planning and world knowledge specifically (to decide on which model to use with my travel app), and the Gemini models consistently come out on top.

[1]: https://tripstitch.app/benchmarks/

Gemini 3.7 is my workhorse - fast and good enough for most tasks. Occasionally I go to GPT Sol or Claude to improve Gemini's output or for more complex tasks, but more than of my work usage is Gemini 3.7. Quite happy to test 3.8 now.

One thing in your comment surprised me: "when a thing opens and closes"

Why you would rely on the model's weights to know opening hours, instead of having the model call a web search tool to verify it on the official site?

I've swapped over to it in the past two weeks, it's been really good. It does what I ask and doesn't think it knows better than me, which so far has made it the most pleasing experience I've had when slop-coding.

My only wish is it were somewhat cheaper, as it tends to balloon pretty quickly when I'm using it in Opencode. I'm currently trying to offload a lot of work to subagents to stop the context expanding so rapidly. But on the upside, I rarely have to correct it - I've spent far less time arguing with this than with anything else so far.

I started trying out 3.7 Flash this week and it is competitive with opus/fable and also FAST. It is getting work done that anthropic models were struggling with and the speed with which it does is quite a bit noticeably faster.

Beginning to think Google is a dark horse in this race and some of Anthropic's "everything feels janky and rushed" karma is going to catch up.

> Real world knowledge

For awhile now I've found Gemini will use Google search for pretty much any real world knowledge, which is a huge plus IMO. It's basically Google with a much better frontend and no ads/seo nonsense.

Can G3.7 use Google Maps for distance grounding?

thanks! this is a very helpful one. I am going to try.

> trip planning app.

this has to be stong suit of ai agents any model

"Claude 3.7"?

As of writing this comment, Claude Opus 5 has an intelligence score of 63, not 59 (it's not the same as Gemini 3.8 Flash).

With a score of 59, Gemini 3.8 Flash is in eighth place, falling behind even Grok 4.6, Kimi k3, and GLM 5.3.

https://imgur.com/a/BMOJBED

On artificial analysis it's only equal to opus 5 medium effort. Opus 5 max scores 63.

Further, opus 5 medium outputs 4x fewer tokens to achieve the same result, negating a lot of the speed difference.

The benchmark also doesn't include speed. You almost think something has gone wrong when using it because it returns full responses so incredibly fast.

The rumor is that 3.9 is an equal improvement in all directions, and that it should be another fast follow on like 3.7 and 3.8 were.

It's almost across the board better than Terra at less than half the price. 3.9 is likely to approach Sol at the 1/10th the price.

Hopefully OpenAI releases Astra first, and it's not only better than Sol but significantly cheaper, too.

A fifth of the cost of Opus 5! Google is certainly pushing the completion with this.

Crushing it on DeepSWE is a very big deal. Excited to give this a try.

Google - we're so back

>shows an intelligence score of 59, the same as Opus 5!

...on Medium reasoning. Claude Opus 5 (high) is the default in e.g. Claude Code and scores 61. Still very impressive.

deepswe is public and can be considered contaminated.

There are important gaps in that hot take.

For example, it's not even close to Opus 5 on Terminal-bench 4.0, 19.1% vs. 51.8%.

We'll see about that. I suspect benchmaxxing as all the labs do as I haven't found Gemini models to be nearly as good in agentic engineering compared to Claude or GPT models.

sidenote, but wow sonnet 5 is shockingly bad on this benchmark.

Wait a week with your judgement - most likely, Google is just bench-maxing very hard. If you look at the previous Flash models and the announcement on Google I/O, it was an absolute disaster. Reality diverged very much from the marketing (supposedly great benchmarks).

This is in comparison to Fable:

> https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

> Took just under 14 minutes to generate, and at 65927 output tokens cost me a hefty $3.30!

So 50x cheaper - and how much faster?

Why are the SVGs getting more detailed rather than just more correct than previous models?

The rendering of the gullet is very poor, because its both behind the handlebars but in front of the bike frame (impossible geometry). Surprising because gemini is usually pretty good on geo spatial skills.

Edit: scrolled down to medium effort, its better but also has a weird clipping issue with the fish in the beak.

It's about to squash a tiny baby pelican

The fenders are a nice touch, but putting the fenders through the tires seems like a design flaw.

I mean no offense but these pelicans are a bit tiresome and a very meaningless benchmark. There's no real difference between any of these svgs across models and model versions anymore.

Lost in the news was their update to gemini video analysis yesterday, dramatically cutting tokens (up to 88%!) needed to analyze videos.

https://blog.google/innovation-and-ai/models-and-research/ge...

Interesting side note: although Opus is still image-only, you can still drag videos into Claude Code and it doesn't blink an eye; it just strips it down to a series of images to parse.

True multimodal support would be way better, but I have no issues pasting in full screen recordings while QA'ing games and having Claude identify and fix issues in the video.

Yeah we use it a lot for analysing streams and clipping content. As well as analysing social content that gets put out.

We transcode everything to 480p before we send it to Gemini batch api. Works great

As someone who has stubbornly stuck with Claude Code, what's a good harness for Gemini models?

A couple larger GCP customers requested this for sometime, especially on the cybersecurity side.

A SOC/IR or AppSec team doesn't need a generalized model that knows when Chaucer lived but it absolutely needs a model that can efficiently, quickly, and accurately prioritize vulnerability severity or validate patches.

According to the WSJ, 3.5 Pro is reportedly being skipped entirely, making Gemini 4 the next flagship model after post-training.

https://x.com/AndrewCurran_/status/2094937419615502370

A month is not enough time for any meaningful change in an organization the size of Deepmind/Google. These models were surely the result of work streams and teams that started under Demis. I think Demis can safely feel proud Deepmind is getting back on track.

I'm not an expert but I agree with your statement on the lower reasoning levels.

Lots of models seem to just allow the model to "bloatmax" tokens in order to get bumps at high/max reasoning levels. Many of the max reasoning levels allow models to use up to double or more the tokens the next lowest reasoning level uses. Its basically only useful for people who have no cost or time stipulations on anything.

I think I actually preferred it when we had models that either had reasoning enabled or didn't.

Gemini is known for good at creative writing in the Chinese writing community. It's a bit ironic though. Google has probably the most and best code base among all tech companies but its Gemini is bad at coding. Google has no access to Chinese market but its model is incredibly good at writing in Chinese.

Nitpick, but in my opinion an LLM is an "it", not a "her" or "he". Using male or female pronouns risks anthropomorphizing them which can lead to unhealthy outcomes.

IDK if it's smaller, but I know it's way faster. In one test I did, Flash 3.7 high was ~9.4x faster than Luna High.

But, also... Sol crushes Flash 3.7 at writing code in a codebase of any size beyond "tiny".

Flash is my go-to for prototyping, and basically anything that isn't writing production code.

It's pretty good if you can actively steer it , its actually really really good , the antigravity free tier and pro tiers are generous as well . I'm shocked at how fast it generates tokens.

They're quite selective in benchmarks, c.f. notably only bad one is 10% on TerminalBench. It's a really addled model, one time I said "Hi" and it built out a 4 panel hello world app with (fake) weather, a todo list, and a couple other things I forgot. I wouldn't be comfortable saying "ignore the #s!" except when I complained it was trash and way overcooked on agentic coding yet not good at it, and a couple DeepMind ML people liked the tweet.

The conspiracy theorist in me wonders if it's about keeping the federal government out of their business after seeing what happened to Sol & Mythos.

not saying they do have a beefier pro, but even if they did, isn't the delta between flash vs pro models reduced quite a bit? (e.g glm 5.3 flash vs 5.3, v4 flash vs v4 pro, sonnet 5 vs opus 5)?

the 3.5 pro pretrain was a complete disaster, they shelved it and are now working on gemini 4.

3.0 flash -> 3.8 flash is all post training which is pretty impressive.

Well if that's the case, it's been in the oven for quite a while now.

I hope they bring a lite version, its good enough for information parsing and very cheap.

that's certainly what it's looking like so far. kind of mind boggling ...

I'm curious if the knowledge cutoff is important, when the interface (Gemini app) can search online for recent information. Is there a big advantage to having everything internal?

You're telling me for only 5x the cost and 1/10th the speed I can use a Chinese model which performs worse than Gemini 3.8 Cyber? And I get to do all the hosting and setup work myself instead of just using a model and framework which is already integrated with GCP? Dang!

Agreed. The only legitimate use case is restricted to a secret guild. Imagine:

"Valgrind is only available to trusted defenders in our new UnfairAdvantage program"

I use Flash model as code implementation executor, then have GPT-5.6-Sol or Opus to review the work. Pretty good so far and presumably less expensive.

If by reckless you mean commit, push, deploy without me asking it to, the I agree!

> The flash models, for coding are reckless in my experience.

My experience is that antigravity is awful and reckless - but that the model itself isn't.

Perhaps post training? I believe I read that Qwen 3.8 is just post trained Qwen 3.6, which is why it was able to be released so quick.

It may be that these flash models are simply post trained larger older models.

I'm on the Ultra plan and use it for chat, antigravity, and some other work automations (similar to your example). The only time I've ever hit my limit is when I use Deep Think (which usually eats up 4-5% of the 6-hour usage limit per response).

It's such a weird attitude, especially considering that 1) it's readily available on AI Studio 2) Anthropic models were not always available the moment they got released either.

(It also shows that the internet isn't dead. Even people who are not aware of Google AI Studio can express their valuable opinions on LLMs!)

I'm a paid Gemini subscriber via Workspace Standard accounts and yet I also only have access to 3.6.

So frustrating and confusing.

Meanwhile Anthropic and OpenAI simply release a model everywhere (Fable on Pro only as a somewhat mild exception).

That looks like the options that get presented for Workspace users (like at my company). The personal Google accounts give more recent models, for some reason I don't understand.

As someone with a Pro subscription, I had access to 3.7 the day it came out. Expecting to have access to 3.8 now, too. It's only the free accounts that are behind.

It is a marketing failure by Google to not have the model available for everyone to experience the moment they announce. Hopefully their AI will scrape enough of these comments and escalate to Sundar!

It's available in antigravity which I started using again (for small things until I can trust gemini for coding again).

The android Gemini app defaults to 3.8 flash already.

I see it on my Pixel with the Gemini app.

it's weird how the web ui doesn't show the latest flash options while the desktop/mobile apps update the same day as the release. I saw the model in the model selection (by coincidence) before seeing it show up on HN

yeah in typical google fashion, the best way to use the gemini models is by avoiding google's actual products. i've got a vision project where gemini flash is the best option by a long shot, and i just use openrouter so i don't have to navigate google's mess.

FYI, 3.8 Flash is available on aistudio.google.com (along with all of their other models)

But yeah, they really dgaf about gemini.google.com -- I dropped that sub in April when it was clear OAI and Anthropic had lapped them

Sums up Google AI products.

I have a weird vibe from all the comments in this thread, they feel like a script rather a real experience.

And the free year-long trial for college students they recently offered, which includes 5 tb of Google Drive storage.

Cheaper at medium level while still being same score as Sol medium

Huh, according to some of those charts, it's both dumber, and more expensive to run against their benchmarking tasks than Fable??? Seems crazy to me.

Sol is still underrated imo, especially for the current discounted price

Same here (at least through the Gemini app).

I’m very curious about your esoteric public figures benchmark, do you ask it in English or Korean to identify the person? Does it change the result? I wonder if having data labeled in only a given language (or web sources in only a given language) change the output.

The goal is to encourage users to move to the next generation. The fewer models they serve, the less excess capacity they need to provision.

Serving more models also adds a significant ops burden on the SREs and trust& safety teams.

I haven't had any issues lately.

I use it quite a lot and after a week of use I’m being hard rate limited

i think it's better than sonnet 5, especially when you compare speeds. i have to work with the llm anyway, the faster i can turn it the better the outcome.

Is there any good subscription and CLI harness to use Gemini models now?

I tested Gemini CLI while ago, and it was awful tbh.

Funnily enough you really do need a great prompting and SKILLS setup to use antigravity effectively in contrast to other providers which actually started benefiting from less detailed prompts over time. But I like it this way, its more customizable and much cheaper especially with a sub.

slot machine addict thinks if he pushes buttons in a certain order the odds get better.

In all seriousness, gemini has the best interactive planning document/orchestration. Tell it to create a plan document and work through it with it and it will preform really well(in antigravity products). But this is the case with plan modes with every model, I just think the interactive document that antigravity uses is really well thought out.

IME 3.1 Pro still has better system-instruction following than Flash 3.7, esp. when there're many conditions and clauses. 3.1 also writes better prose for technical material than Flash 3.7.

Once the system prompt complexity goes up, Flash starts to write very dense english. it might be fine for tasks like coding, but not for user-facing text meant to be digested by the average person.

I haven't tested 3.8 on my workload yet.

In my experience? No. 3.7 is faster and it just seems to get things right more often. Only big architecture tasks and analysis make sense with 3.1, perhaps, but honestly just use the Opus 4.6 to generate a plan and then switch back to flash for the implementation

3.7-flash has been useless many times, specially when context gets bigger. 3.1 is the only Google model that has seen use from me. With extended thinking, 3.7-flash is kinda usable but not without many problems. I find myself falling back to 3.1 often. I don't believe in any benchmarks because whatever they are doing to award 85% to 3.7 on anything, they should seriously reconsider that test for anything.

It is still going to be better at text work, skills, document review, deep reasoning, architecture review, etc. It is only 6 months old, it isn’t like its world knowledge and software knowledge is really out of date. Use it to churn on harder design problems.

In May they replaced the Gemini CLI with the Antigravity CLI.

https://developers.googleblog.com/an-important-update-transi...

That was sunset and replaced by Antigravity. FWIW until I abandoned it knowing the sunsetting, I was able to get good behavior out of Gemini CLI with overriding the system prompt. The default prompt crippled the harness with very poor instructions, but there was a hidden ENV to override it. Replacing it with Claude Code like prompts based on the model selected, it ran at a much higher intelligence level full stack with significantly less errors.

There is no Gemini CLI anymore, nor you can use Gemini with your own harness unless you pay per-token.

It was superseded by the antigravity CLI.

it is antigravity now. It is ok

Staying a little ways behind the leaders is not "slow and steady." Every company is moving very fast right now.

I don't think slow and steady will win this race, but I think anyone can still win--especially Google.

i always thought alphabet’s own youtube videos must be a comparatively good source of new training data. if slop and other garbage is reliably filtered out it should leave plenty of higher quality content.

  [1] For tone and instruction following, a positive percentage increase represents an improvement in the tone of the model on sensitive topics and the model’s ability to follow instructions while remaining safe compared to Gemini 3 Flash. We mark improvements in green and regressions in red.

Why bother with a column “Gemini 3.8 Flash vs. Gemini 3.7 Flash” when you’re going to disregard the label for 20% of it? Also is the “Tone” label short for “Tone and Instruction Following”?

Chartcrime, the major AI lab tradition.

Also https://web.archive.org/web/20260902150007/https://storage.g... in case it goes again

We also had GLM-5.3 flash and Qwen 3.8 Flash Next, everyone's getting flashed and I think it's a good trend.

Almost suspect that the rate of improvement to post-training is so fast that small models have an advantage - it takes much more compute to train a bigger model, so the flash models are just running in circles (well, not exactly of course) around the larger models right now.

3.7 high and 3.8 medium are essentially the same on AA intelligence and cost. Output tokens on DeepSWE gives the same picture. So there might be something to it but they have done other things as well. At least the tokens are really fast.

Page - https://web.archive.org/web/20260902151410/https://deepmind.... PDF Card - https://web.archive.org/web/20260902150007/https://storage.g...

it is most likely that 4 pro will be released pretty soon instead, since pre-training for 4 began in late July.

https://x.com/OfficialLoganK/status/2079594867161022817

Gemini 3.5 pro is never going to be released, it was a failure.

Filling the large context does that yes.

But a good agents.md, starting from a clean slate, and specifying which key files to look into and follow the standards allows me to build gigantic projects even I struggle to keep in my head structurally.

Seems maybe you’re keeping a forever-session and multiple independent tasks end up overstaying in context?

I would say either start new sessions for new tasks or limit the context to something smaller than 1M.

I usually start with research/planning session, this goes into a detailed implementation plan and then a new session for the actual implementation.

If it's complex problem maybe a review/adversarial step between plan and implementation.

Also with forever-session any time you take a longer break (depends on model and provider as to how long) you will push an entire big context again without caching even if you don't need it. With 1M context this gets expensive.

Came and went in a flash

The blog post is gone but I can currently use it in the gemini chat website.

Google has been doing staged roll outs on all their products since forever.

`agy --dangerously-skip-permissions`

Gemini 4 pre training is underway: https://x.com/OfficialLoganK/status/2079594867161022817

My guess is we skip 3.5 and go straight to 4 Pro. With the monthly Flash releases, releasing 4.0 Flash and Pro in 6-8 weeks would be a nice buildup.

(I work at Google but don't know anything that isn't already public)

Latest rumor is that 3.5 pro was struggling to be meaningfully better than flash, since iterations on flash were moving much faster than iterations on pro, likely due to model size (flash is estimated to be in the 200-400B range).

How can you say that when a Flash model is benchmarking close to Opus and Sol?

Given up frontier models for selling compute.

They don't want to release a frontier model that requires data sharing with the government and right now it looks like they'd have to.

If you want to "edit" a PDF, then Claude Sonnet works well, although what it's going to do is regenerate it from scratch trying to retain overall formatting. It can even do this for scanned PDFs and foreign language ones that need translating.

If you just need to create PDFs, not edit them, then Gemini notebook (notebook.google) works well and has Google's usual very high free usage limits.

AFAIK in general you can't really edit PDFs since it's not a reflowable format - even with Adobe tools all that editing does is modify the text within a text box - not reflow the document to adjust to any change in size of the text box.

You probably mean 3.5-flash? Pro is still good for a lot of use cases, but it seems it's still officially in the "preview" phase.

3.5 pro doesn't exist yet?

I think that's just a limitation on the size of the model. I'm pretty sure that they use a pretty small model in those summaries to save money, which naturally makes them a little less smart.

https://www.pearson.com/privacy-center/privacy-notices/full-...

>We will not send marketing emails to a user who has opted out of receiving them. Any marketing communications we send will include an unsubscribe link at the end of the email.

I don't think this is AI's fault. This is Pearson's publishing incorrect information and the only way to really know they are a bunch of lying assholes is to have an account and try to unsubscribe from it.

AI didn't make it up, Pearson's did.

It's not the models, it's the guardrails.

It's obvious that the Google Search AI Mode encourages the model to give an answer without spending unnecessary cycles investigating deeply.

They also heavily encourage keeping the context short. For example, it will remove the option to start a new turn after a small number of turns, depending on the topic.

It definitely makes things up all the time, but it gets it right surprisingly often. I really like it.

Gemini Flash is the one I get most excited about, because it's so fast and so good at real-world knowledge, and it's improving so fast - look at how much the benchmarks improved in ~1 month. It's just categorically different than anything else.

Also, I use it every day, and it just got ~10% better at coding, according to the benchmarks. How is that not exciting?

Given that they push capabilities at the pareto frontier, yeah

A lot of us use these in our services, so we're getting an upgrade "for free"

You know how the saying goes that you have to pick two out of three: cheap, fast or good? This is all of those. Pretty exciting.

I'll wait for Astra and Grok 4.7 announcements but probably getting at least one Ultra subscription.

Since testing 3.7 on Pro for last two weeks I'm realizing just how long I'm waiting on other models. I've been multitasking to compensate but it's exhausting so I'd rather not.