bartread
5 days ago
The thing is, it's not actually that hard to spend at that kind of rate.
I wouldn't say this was the norm, but I have burned through £1k of tokens in a couple of days before. And, on that same gig, I realised a test I was considering running would burn £2.5k worth of tokens over the course of a few hours so spent a couple of hours of my own time re-engineering it to get that down to £150.
The point is that I don't necessarily know that spending a shitload of money on tokens really tells us anything useful. If he's getting a return on that investment, fine, whatever.
Obviously most of us couldn't afford to spend that much ourselves, no matter how much value it might deliver, and many of our employers wouldn't be too happy if we spent it on their dime either. Equally, if you can, and it's delivering real value, then I don't see why you wouldn't?
nodesocket
5 days ago
At what point do you buy a Mac studio or AMD Gorgon Halo and try running some of these tasks in local models? Surely don’t need frontier models for all this work.
bartread
5 days ago
You definitely don't need frontier models for all this work.
And, in fact, I got it from £2500 to £150 by careful model selection (I ran a trial over a small dataset with different models until I found the least expensive/capable model that would reliably do what I needed), using a local model for one step, and paring down the inputs as well so the token count plummeted.
wolvoleo
5 days ago
Yeah I see the same thing at work. Most of the people burning crazy numbers of tokens aren't doing any amazing work. They're just using fable to write their emails and do their web searches.
They feel like they're super smart but it's just as smart as hiring a $500k/yr PhD as a PA to schedule your meetings and answer the phone. Sure it does a good job but you're just wasting money and abilities.
They feel super edgy and bleeding edge but all they really do differently is switching the AI chat from auto to the most expensive option all the time. When I see what they're using it for they're only grasping the basics of its abilities.
They're also people sending 2 page AI generated emails that make no sense for literally everything where a "Yeah sure let's do that" would have sufficed.
Unfortunately most of these are business leaders who just want to look edgy and "all in on AI" so we can't really limit their tokens. Yet they're also the ones saying we spend too much and want to limit some of our dev teams actually putting it to good use.
bigfatkitten
4 days ago
I tokenmaxxed in my last role.
Management succumbed to AI psychosis, and so we were directed to show that we were adopting and using AI, but what we were using it for was not considered important. So we went out of our way to burn as many tokens as possible.
If you tell Claude to be inefficient and burn as many tokens as it can, it’ll happily oblige. It will also offer useful advice as to how to use even more tokens.
jorvi
5 days ago
You don't.
Once you factor in cost of electricity and depreciation of hardware, it is more expensive to run locally.
There are many arguments for running local but cost isn't one of them.
nodesocket
5 days ago
> but cost isn't one of them.
Let's do quick back of the napkin math.
- Mac Studio M5 Ultra, 80-core GPU, 256GB memory, 2TB storage -- $11,299.00.
- Power (385w continually 100% of month) is $40/mo (my regional) or $480 annually. Let's say power is 3x more expensive in California, which it probably is, that's still only $1,440 annually in power.
The Studio M5 Ultra has a worklife of at least 3 years.klrefg
3 days ago
You have to compare with how much Deepseek costs since all the models you will be running will be inferior anyway, certainly not Opus or similar.
jcelerier
4 days ago
it's going to be quite slower than remote providers, won't be able to run full models, only quantised, and for more cost than the highest Claude subscription tier where you can do roughly 60-100M token per month at 200$, e.g. 7200 for three years. It makes sense if you prioritize local AI but not really otherwise sadly.
bitfilped
4 days ago
You have to be using an incredibly small amount of usage for your math to math here. Go price out a few MW or better site then ask a neo cloud how much that same amount of compute will cost at an API.
plusfour
5 days ago
> would burn £2.5k worth of tokens over the course of a few hours
details please, if it's interesting
deadnomore
5 days ago
Evaluation loops do this easily. You have to run hundreds of reasoning jobs in parallel and very quickly you can torch thousands of dollars.
bartread
5 days ago
Yeah, it was basically this, and running a whole lot of agents concurrently to churn through the dataset faster.
sixtyj
5 days ago
Yup, run multiple agents on expensive model in loop via API.
You can do it if your employer has endless pockets.
Example: $30/1M input tokens (I don’t count output or cache, for simplification), you have several daily agents for everything => 3M tokens daily per agent * 10 agents ~> $900/day => $328k/year (API spending)
hparadiz
5 days ago
Any 3D work in blender can burn that easily. Add video editing to that mix in any capacity.
jmathai
3 days ago
If I were spending $200k on tokens then I sure would expect some mind boggling results. I don't know much about what this person is doing with them and maybe they've launched several companies or created several new inventions .... but that's what I'd expect personally at that spend.