In the early Gemini 2 days (don't remember which version exactly) I had Gemini running as a voice assistant in my kitchen, and asked it to flip a coin and tell me if it was heads or tails. It responded with "heads". I was curious if it was actually doing something to simulate randomness, so I asked a few more times and saw a pattern: "tails", "heads", "tails", "heads"...

It continued alternating between the two until I got bored (around a dozen turns).

Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.

Hah! I did the exact same tests as you! I found that if you give it the choice to say "not sure", it picks that 100% of the time. But if you pin it in a corner, then yes it does these weird things. Also yes, the continuous options were much more accurate than the choices. Not sure why that is.

If you instead offer probabilities as answers, it picks the right one with high credence.

But "problems about actual probabilities whose answers can't be known at all" are exactly the problems where calibration is important. Since calibration is one of the big claims about Jev I'd expect it to perform well in these problems.

Echoes a bit of a philosophical distinction with a long history: "Knightian Uncertainty" versus "Probability".

Maybe I’m confused here, but it’s perfectly reasonable to just guess the same dice roll every time right?

So you want the probability that the answer is correct, but Jev is providing the probability that its answer is optimal?

Did you expect it to be good at it?

Humans also don't give a perfect 1/n probability when asked for a random number.

> The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%

I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.

However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.

1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.

If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.

While I don't believe they are doing the following: you can calibrate by inspecting the reasoning traces. That is the relevant distribution. If you ask someone to explain how/why they are classifying something one way v another, you can get a reasonably good understanding of their confidence level.

It's on OpenRouter if you want to try it.

I suspect they meant it as "they claim it is already calibrated, but it can't be, because there is no universally correct calibration", and not "it is not possible to calibrate Jev", but I read it as the latter at first too.

They seem to have a really good social media astroturf marketing campaign.

It’s not just price and speed, the API is very well designed for classification. Going beyond the current structured outputs.

Jev’s context should have signals/features to operate on. Same way LLMs can use CV & code to analyze an image.

I suspect that it's as handy as a coin flip.

And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.

Is Jev good at detecting AI-generated text?

But why? Jev-style models seem useful for "I have no clue what my incoming distribution looks like but I need to give some sort of answer". If I know what my incoming distribution looks like I'll just upload a CSV of that into ChatGPT and ask it to fit a basic ML model on my data.

If you want calibrated probabilities you'll be forced to fine-tune it

"Not especially good at classifying" does sound like a serious drawback for a classifier

do you have a benchmark name and results we can compare to other models?