whats confusing they have different pricing for output. Here [1] it's $5/million output (starting 1st october) and on vertex AI [2] it's $2.5/1 million (starting 7 october) - but characters - so it's overall gonna be more expensive if you wanna compare to equivalent 1 million tokens. It's actually even more confusing to know what kind of characters they mean? 1 byte? UTF-8?

[1] https://ai.google.dev/pricing

[2] https://cloud.google.com/vertex-ai/generative-ai/pricing

I wonder if they're pulling the wall-mart model. Ruthlessly cut costs and sell at-or-below costs until your competitors go out of business, then ratchet up the prices once you have market dominance.

GPT-4o is 2.5/10$. Unless you look at an old checkpoint. GPT-4o was the cheapest frontier model before.

> For comparison, GPT-4o is currently $5/million input and $15/million output and Claude 3.5 Sonnet is $3/million input and $15/million output.

Google is the only one of the three that has its own data centers and custom inference hardware (TPU).

It doesn't matter if it's cheap, it's unusable.

They CHANGED the pricing from $2.50 to $5.00 stealthily unannounced. Look at the web site again; it says $5 per million now, and this comment on this website might be the ONLY evidence in the world that I wasn't gaslighting myself or hallucinating!

This is the most important update.

Pricing and speed doesn't matter when your call fails because of "safety".

There's still basic filters even if you take all the ones that you can turn off from the UI all off. It's still not capable of summarizing some YA novels I tried to feed it because of those filters.

The "safety" filters used to make Gemini models nearly unusable.

For example, this prompt was apparently unsafe: "Summarize the conclusions of reputable econometric models that estimate the portion of import tariffs that are absorbed by the exporting nation or company, and what portion of import tariffs are passed through to the importing company or consumers in the importing nation. Distinguish between industrial commodities like steel and concrete from consumer products like apparel and electronics. Based on the evidence, estimate the portion of tariffs passed through to the importing company or nation for each type of product."

I can confirm that this prompt is no longer being filtered which is a huge win given these new lower token prices!

The recitation error is a big deal.

I was ready to champion gemini use across my organization, and the recitation issue curbed any enthusiasm I had. It's opaque and Google has yet to suggest a mitigation.

Your comment is not hyperbole. It's a genuine expression of how angry many customers are.

It seems the recitation problem has been fixed with the latest models. In my tests, the answer generation no longer stops prematurely. Before I resume a project that was on hold due to this issue, I'm gathering feedback from other users about their experiences with this and the new models. Are you still having this problem?

They are going for large corporate customers. They are a brand name with deep pockets and a pretty risk-adverse model.

So even if Gemini sucks, they'll still win over execs being pushed to make a decision.

I do like the trend.

Imagine if Anthropic or someone eventually release a Claude 3.5 but at like a whopping 10x its current speed.

Would be incredibly more useful and game changing than a slow o1 model that may or not be x percent smarter.

I have used Github Copilot extensively within VS Code for several months. The autocomplete - fast and often surprisingly accurate - is very useful. My only complaint is when writing comments, I find the completions distracting to my thought process.

I tried Gemini Code Assist and it was so bad by comparison that I turned it off within literally minutes. Too slow and inaccurate.

I also tried Codestral via the Continue extension and found it also to be slower and less useful than Copilot.

So I still haven't found anything better for completion than Copilot. I find long completions, e.g. writing complete functions, less useful in general, and get the most benefit from short, fast, accurate completions that save me typing, without trying to go too far in terms of predicting what I'm going to write next. Fast is the key - I'm a 185 wpm on Monkeytype, so the completion had better be super low latency otherwise I'll already have typed what I want by the time the suggestion appears. Copilot wins on the speed front by far.

I've also tried pretty much everything out there for writing algorithms and doing larger code refactorings, and answering questions, and find myself using Continue with Claude Sonnet, or just Sonnet or o1-preview via their native web interfaces, most of the time.

The Aider leaderboards seem like a good practical test of coding usefulness: https://aider.chat/docs/leaderboards/. I haven't tried Cursor personally but I am finding Aider with Sonnet more useful that Github Copilot and its nice to be able to pick any model API. Eventually even a local model may be viable. This new Gemini model does not rank very high unfortunately.

I know you aren't necessarily talking about in-editor code assist but something about in-editor AI cloud code assist makes me super uncomfortable.

It makes sense I need to be careful not to commit secrets to public repositories but now I have to avoid not only saving credentials into a file but even to paste them by accident into my editor?

I tried it briefly and didn't like it. On the other hand, I found Gemini pro better than sonnet or 4o at some more complex coding tasks (using continue.dev)

I use it and find it very helpful. Never tried cursor or copilot though

God, it sucks

They put a 42 minute video on twitter? That's brave.

> What makes the new model so unique?"

>> Yeah It's a good question. I think it's maybe less so of what makes it unique and more so the general trajectory of the trend that we're on.*

Disappointing.

i think it's unusable if you are trying to use via GCP. Using via ai studio is a decent experience

[flagged]

Pretty sure Google's got more than 700 million active users.

In fact, Google is most likely _the_ target for that clause in the license.

They do: https://cloud.google.com/vertex-ai/generative-ai/docs/partne...

AI companies should not pick up naming models after astrological signs, after a while it will be hard to tell apart model reviews from horoscope.

Llama seems to be tricked up by simple puzzles that Gemini does not struggle with, in my experience.

I've found Gemini Pro to be the most reliable for function calling.

> Geminis are unusable

how so?

https://www.businessinsider.com/big-tech-org-charts-2011-6

Google is now looking more like the Microsoft chart.

People have said that for many years. Very few companies are choosing Google's TPUs. Everyone wants H100s.

I've had it fail when a video does not have subtitles - I'm guessing that's what it uses. I have had good success having it answer the clickbait video titles like "is this the new best thing?"

It's not watching the video as far as I can tell.

I imagine you'd be paying more in ML costs than YouTube makes off your views.

Has it changed much since March when this was written? Gemini won, 6-4. And it matches my experiences of just using Gemini instead of ChatGPT because it gives me more useful responses, when I feel like I want an AI answer.

https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt

If you are on the pay as you go model your data is exempted from training.

> When you're using Paid Services, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products, and will process your prompts and responses in accordance with the Data Processing Addendum for Products Where Google is a Data Processor. This data may be stored transiently or cached in any country in which Google or its agents maintain facilities.

https://ai.google.dev/gemini-api/terms

Makes sense, as soon as your data leaves your computer, it's safe to assume it's no longer private, no matter what promises a service gives you.

You want guaranteed private data that won't be used for anything? Keep it on your own computer.

> Docs are flatly wrong

This +999; I couldn't believe how inconsistent and wrong the docs were. Not only that, but once I got something successfully integrated, it worked for a few weeks then the API was changed, so I was back to square one. I gave it a half-hearted try to fix it but ultimately said 'never again'! Their offering would have to be overwhelmingly better than Anthropic and OpenAI for me to consider using Gemini again.

The engineers at google are bad, they keep hiring via pure leetcode. Cant ship working products

Same experience here.

I had hopes of Google able to compete with Claude and OpenAI. But I don’t think that’s the case. Unless they come out with a product that’s 10x better in the next year or so I think they lost the AI race.

The math score exceeds o1-preview (though not mini or o1 full) fwiw.

Moat could be things like direct integration into Gmail (ask it to find your last 5 receipts from Amazon), Drive (chat with PDF), Slides (create images / flow charts), etc.

Not sure if their models are the moat. But they definitely have an opportunity from the productization perspective.

But so does Microsoft.

> Does google have moat?

Potentially (depends if the EU cares)...

E.g. integration with Google search (instead of ChatGPT's Bing search), providing map data, android integration, etc...