thinking_effort: max Reasoning trace: This is the classic pelican-on-bicycle SVG test.

Opus 5.5 had similar response on max: This is a classic test request

https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

I think your test is already embedded into the models. You should search for new frontier tests to subject the models to. Maybe they should now try to unify the standard model and general relativity in physics. I'm pretty sure this is nowhere to be found in any training data nor shared in any chat between a scientist and a LLM ;)

Of course, the sun again. Everyone knows that a pelican can't ride a bicycle without a sun in the frame and can only go right.

I always find the time/token differences between the xhigh and the max effort levels for Claude models absolutely insane.

Even more so, because in a lot of their benchmarks they use the max models. I honestly think I'd rather these labs use their xhigh models as the default for benchmarking instead since I don't think the average person is even using max.

Long legs to illustrate me attempting to stretch my budget by using Haiku the last week of the month.

I'm still getting network errors. Seems to be CORS-related.

I find it helpful when you post your link that compares the model to other models in the same class or family, or shows progression over time.

The pelicans all start to look the same after a while.

But seeing the comparison to other models by class, family, or historical progression gives an excellent frame of reference.

I thought Anthropic models didn’t generate images.

Haiku 5.5 is noticeably smarter than GPT-6 Luna, so I can see their pricing strategy here.

For a while Anthropic has lacked a cost effective “cheap” LLM for summarisation, compacting, RAG helpers, etc.

These ‘ephemeral’ workloads are often under 100k tokens, or can be structured to be under 100k.

In some coding benchmarks, Haiku 5.5 beats Sonnet 5! (Especially implementation; do a well defined Jira ticket; etc), it’s really impressive how much intelligence per dollar has grown in just a few short months.

There's also a tokenizer efficiency difference: modern Claude's 100K tokens are about ~60-65K modern GPT tokens, so in reality the Luna cutoff is much further away than the Haiku one.

You can test with Anthropic's count_tokens endpoint or with https://crates.io/crates/tokwc

It's actually existing flat per-token pricing that is weird.

Neither encode nor decode are linear in compute, so providers need to price for average expected length.

This is just getting closer to the true cost of generating tokens.

If you can't get any coding done with 100K context that is either a broken model, a broken harness or a skill issue. I would mostly use Haiku in task or explorer subagents. I'm not saying I stay under that on every task, but I do have quite a few sessions that cap out well below that, so that price difference would be very meaningful.

I use Luna for this day in and out and its excellent - if Haiku is that much better I will be changing things up.

Notable that one suggested use case for Haiku is "classification requests", i.e. Jev competitor, and the pricing matches GPT-6 Luna which is behind OpenAI's "Decisions API" Jev competitor.

For this application 100K token input is plenty.

Of course Anthropic and OpenAI, both at $0.10/M, are still 2.5x the cost of Jev's $0.04/M.

Isn't it less than a year since Claude models went from 100k token limit to 1M limit? Don't get me wrong - my main agent normally gets to 25% or so before I clear it these days, but as a subagent, doing research or summarisation, I don't think 100k is "absurdly low".

Luna does as well, but just at a higher limit.

From OpenAI's website: Prompts with more than 272K input tokens are priced at 2x input and cache rates and 1.5x output for the full request.

They are targeting businesses/API use for fast decision making and agent integration. Plus they now need to be competitive with Jev-type models in that space.

If you look at how different reasoning levels can easily exceed task cost of sonnet 5.5 you will see that you will basically never fall into that under 100,000 token threshold. I mean maybe you can choose low and do a basic summary task, but then you could choose something much cheaper instead. I don't know what Anthropic is thinking with its dumber models.

You could also use it as a subagent prompted eg by Sonnet/Opus orchestrator agent and for many agentic workflows significant part of the dispatched tasks might be under 100k budget.

I with they'd give Haiku like 400k tokens roughly, I think between 400k or even 600k tokens is a sweet spot, but Haiku is basically designed to be for small edits is my understanding, but it sucks because any time I ask Opus to "try" letting Haiku do the work, it just falls apart and Opus comes back and tells me it switched to Sonnet (even before Sonnet finally jumped up to 5.x).

I will try the new Haiku, but it would be worthwhile if Haiku could take sane instructions and do all file editing for Opus / Sonnet / Fable then it would be worth using.

>> 100k tokens is an absurdly low cutoff and it is only applicable to Haiku and not Sonnet or Opus. It's a low enough cutoff that it will be quickly exceeded if you are doing anything with Agents

Your vibes don't appear to be supported by facts. From the announcement:

>> Claude Haiku 5.5 is priced 90% lower than Claude Haiku 4.5 for requests up to 100,000 tokens, and 50% lower for requests over 100,000 tokens. On Haiku 4.5, 90% of requests fell into the former category.

I mostly use Haiku for really, really basic stuff, never for actual engaging work. I've used it for first-pass analysis to triage bugs, for example - all it does is related N bugs together to see if any potentially relate. Then I have Sonnet investigate further.

This is pretty good tbh

encode and decode tok/s which is ($/s) when it comes to pricing drops heavily above 100k tokens.

There are plenty of workflows like translations where you'd easily be under the cap.

Chatbot could be < 100k tokens.

Who in their right mind would use haiku while Mimo or GLM cost 10% of what they are charging with much smarter models?

It's their creative way of 'matching' Luna's prices.

It could be to incentivize people to not be lazy users of tokens.

This is them sneaking in taking the Claude Agent SDK (claude -p) off of subscription plans through the back door along with a model release. They previously wanted to do this in June, but backpedaled after huge backlash:

https://support.claude.com/en/articles/15036540-use-the-clau...

This is literally for you to get tangled in their api and when they stop giving you the allowance they hope you will just continue to pay

This is massive. So on top of the regular usage, we now have USD 200,- to freely use via the API however we please, even resell? That is a statement, even knowing that inference does not cost them nearly as much as they charge, this is very developer-friendly. Does some minor de-risking for testing concepts. Terms seem to be reasonable [0].

Of course, they don't do this out of pure kindness, but I really struggle to see a negative for subscribers already using a Claude Max subscription, especially given changing to another model is essentially frictionless via OpenRouter.

Compared with "Sign in via OpenAI" which they just announced, this is far less lock-in for anyone hosting services but less interesting for users of said services. With Anthropics approach, you can just use the allowance on your users however you see fit along with any other models and once it's used up, you can still just decide not to use their models for the remainder. With users bringing their tokens meanwhile, there is less flexibility in terms of switching for you, though might be cheaper for users.

Both interesting, each approaching this from a very different direction, each having their own trade-offs. On the OpenAI front, will be interesting whether developers can set specific temp, reasoning budgets, etc. for such "provided tokens" or whether OpenAI exposes that only via the actual API.

[0] https://www.anthropic.com/legal/credit-terms

OpenAI will probably add this to their plans within a week

If we look at their performance per dollar charts,

In OSWorld 2.1 Haiku is better.

On GDPval-AA v2.1 Haiku is equal or worse than Luna.

On Humanity’s Last Exam they don’t seem even be comparing Haiku with Luna.

For these baby distillations of flagships, I expect their users to be very price sensitive.

Yep. I was looking at the prices of lower tier models a few weeks ago for zero/few shot tasks (pre Jev) and Haiku rates just didn't make sense at all. I ended up using 5.6-luna.

Good to know that is going back to being an actual option from perf/price perspective.

They do not, however, have the same price per task or task execution ability at any thinking level. Token cost alone is not a sufficient metric.

Pac-Man Bench:

Considering the price, no model comes close to being as good as this. However, it did take an extremely long time.

TIME 19m COST $0.16 https://jonclegg.github.io/pacman-bakeoff/#claude-haiku-5-5

All results: https://jonclegg.github.io/pacman-bakeoff/

> likely best used for small subagent tasks / tightly scoped work.

Hasn't this always been the case with Haiku?

To be fair, you're making it compete with the best public LLM right now that's 2 size/price tiers above it.

Neither of these look "good" to me. There is so much visual noise on the page, like someone turned the "AI Slop" dial to 11. In fact I prefer the simpler design Haiku made.

Note that this is Anthropic Trojan-Horsing the previously announced June change in with a model release, where the Claude Agent SDK can no longer be used with Claude subscriptions and is now billed with API credits only.

https://support.claude.com/en/articles/15036540-use-the-clau...

yeah totally agree. esp how efficient it can be to have a subscription quota-paid orch spin up a bunch of API agents, this is kind of like free money to encourage what was already an easy way to save money (via batch pricing)

OpenAI has been a disaster lately.

> Being able to actually use my Claude plan for other harnesses

Wait what? This has gotten their blessing?

How do you validate the functions are correct? I did something similar, letting it (mostly deepseek 4.1) translate from assembly to C but it commonly made mistakes, some really hard to discover and fix.

can you share more details? was this very involved or asking codex/claude/open code with a 1 shot like approach?

i love Age of Mythology, but why did you feel the need to decompile it? Its got a great world editor if you were trying to "mod" it.

Exact same thing here.

Both evals and Human pairwise tests for our use case are giving Haiku 4.5 first place in pretty much all tests.

No we'll try understand if we need to change our prompts to match performance ...

edit: maybe this will help: https://platform.claude.com/docs/en/build-with-claude/prompt...

Curious as to why. Haiku 4.5 has been far away from pareto frontier for a long time. Maybe you need to update your prompt for the newer model in your workflow.

Something seems wrong here. Haiku 4.5 is a pretty bad model for just about everything.

Did you have prompts that were especially tuned for Haiku 4.5 or something?

A really big difference can be seen in this basic CSS animation generation of a solar system of Luna vs Haiku:

https://aibenchy.com/compare/anthropic-claude-haiku-5-5-xhig...

Yes -> every 18 months they've gotten 90% more efficient for the same level of quality for about 5 years. There's little sign that trend is slowing. If anything, there's reason to believe that System 1 models (plus potentially 1-2-3 workflows) may increase that over the next 3-5 years.

You'll know when the trend stops -> when the intelligence differential between smaller models like 7B starts to grow instead of shrink from 32B models -> that means 7B is getting about as smart as it can get. Then, 32B will follow next, then 70B, etc etc.

We haven't yet seen that at any size AFAIK.

According to Epoch AI:

> The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. [0]

0. https://epoch.ai/publications/the-plunging-price-of-thought

reminds me of this blog post: https://campedersen.com/singularity

I've heard tell about 100% of certain types of work being ended in batches of six months. For years. Truthfully, I'm skeptical, but accuracy wasn't prioritized.

double the information density every 2 days?

serious bit: if you think about how these smaller models work, at the end of the day it seems that they are now capable of forgetting useless information because they're able to derive it in reasoning allowing models to become smaller at the cost of requiring more reasoning tokens to solve a task.

Hopefully enough runway for an existing model to train the next to be better than itself with absolutely no human intervention.

Maybe they think it's of interest.

Is that page showing Opus TPS stats in fast mode? IIRC fast mode is 2.5x speed, so that would be 293 TPS, no?

I'm excited for API use. I run some agents, mostly on Luna 6 right now. It's just tool use, web browsing, etc, so something dirt cheap, but also not super dumb, is much appreciated. Having a Luna competitor is nice.

there is no way in hell that im gonna use Haiku too, tho weekly limits become a problem for me in a last couple of month tbh

It's a brave new world... of idleness!

https://xkcd.com/303/

Is it realistic or cynical for me to assume this is to wean developers off the heavily subsidized subscriptions? Presumably it's using similar compute.

Translating GPT’s word salad to English at the end of an agent interaction.

I have a zsh functions that calls claude code with haiku to suggest commit messages, is faster and the instructions are two lines.

I also have an "ask" script that I use daily to ask simple stuff, it can access websearch and webfetch, it's more than enough to parse logs, ask for commands, quick research on the internet, small stuff. https://github.com/mariocesar/dotfiles/blob/main/common/.loc...

I use haiku for things that needs to be quick, have really clear instructions.

I've been using GPT-6 Luna in some capacity for nearly all my agent workflows. It's just a really good model, and the pricing is cheap. If Haiku 5.5 is better, and the same price (under 100k context... which is a big caveat) i'd probably swap it.

Opus often picks it when it's doing a "find me something" subagent. But largely it's been held back by being fully a year old at this point, and priced at a much higher price than models that are far more capable.

not haiku, but luna - last week i used it for things like "read this historical dump of 15k support tickets and break them into categories that make sense, then propose help docs that i could write to handle the most frequent queries in each category"

used <10% of my 5hr limit on a $100 codex plan.

I was waiting for this.

Planning on doing flash analyses of PRs that impact evals in some way, and then post comments on GitHub whenever there’s flaws in them

( https://evalship.com )

It's great at parsing documents inexpensively. For the few skills/plugins I've made, I usually instruct Claude to use Haiku for low-reasoning grunt work.

My software application uses Haiku in production more or less as a Jev. I do not use it for coding or development.

I'm building a game that incorporates LLMs as a game mechanic.

I've been using Luna, but I'll probably switch to Haiku.

haiku < sonnet < opus (poetry)

lune < astra < sol (astronomy)

That was effectively required to match GPT-6.1 Sol (costs and caching prices are now equal). Sonnet 5.5 made zero sense to use over Opus 5.5 under the old cache prices.

IMO, this is better. Luna is super cheap, but it's not that capable. At higher levels of reasoning, it's not that fast.

This is more expensive, but it also looks like it's better enough that it's far more useful.

I also won't be surprised if you look at cost per completed task + wall clock time that it comes out ahead for the majority of what you'd want to actually use it for.

Luna will still be a great option for doing non-engineering tasks super cheaply.

For prompts under 100k tokens, it's priced the same as Luna - $0.10 in, $0.50 out.

For prompts over 100k tokens it's 5 times more expensive - $0.50 in, $2.50 out.

they are afraid of regular people having access to open weights, or at least that's what they tell us and the government

To correct myself. The price was not lower. It was up about 20% for the tokens. But, the big price hike was that it used a lot more tokens for the same tasks.

[flagged]

Yeah it's too little too late, cat's out of the bag as people know that GLM 5.3 exists and is great at defensive and offensive cybersec.

(sadly Mistral Large 4 isn't up to par - but Mistral serves GLM at 130 tps!)

I use it for title generation basically. Will have to see where this one fits in

Opus 5.5 does :^)