This is, sadly, obvious and inevitable in retrospect.

The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself.

This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.

I’m not usually one to ask this because learning to do a thing can be fun, but why exactly have you spent 25 thousand dollars on getting an LLM someone else made to answer maths exam questions?

I got an RTX 6000 pro too. I like running locally, I've learned a lot more than if I had used an API and there's less worry about overspending tokens. I accidentally spent $100 on claude api in like 2 days because I didn't know what I was doing.

The problem is that while one these gpus is a huge improvement over a laptop or a single 3090, you very quickly wish you had more. I would buy a second one, but I did the math and realized that with the current crop of models, 2 Blackwells doesn't buy me any new capability that I didn't have with one. So I would need a 3rd one. And when I buy a 3rd one I will feel like I want to running a higher quant, so then I will want a 4th.

>> find a reliable place to sell it without getting ripped off by scammers.

This is a real problem and why I've just about given up on ebay or fb marketplace, esp for computers. If you are in Canada though sellit9.com is a great solution to having to deal with sketchy buyers.

If you're in a decent sized city, you should be able to find a local buyer on Craigslist or FB Marketplace... Beyond that, for higher value, smaller items like your M3 Ultra, I would talk to your local police department and/or library to see if you can do the exchange there. Larger libraries usually have a police officer on site or nearby, and the PD office near you may also provide a "safe" exchange location... I'd bring a monitor/keyboard/mouse so you can demonstrate the system working properly.

YMMV but between your nearest PD office and Library, you should be able to use one or the other for your exchange of goods/money. The biggest thing I've sold is a mid-range video card during late covid (I managed to get a better one via newegg shuffle) so I sold the old one (RX 5700XT -> RTX 2080) to make up the difference a bit. I just did the exchange at the Starbucks near me for that.

I have three m3 512gb units and want a fourth to run an exo set up. Like you, I am worried about scammers. Let’s discuss if you still want to sell.

https://calendly.com/ryanwmartin/open-office-hours

Which of these has been the most productive for you? Sounds like you've enjoyed the RTX6000 the most?

All of these have appreciated in value. How much are you looking for the Ultra?

    > if I can find a reliable place to sell it without getting ripped off by scammers.

Well if it makes you feel better those frontier LLMs are all technically taking a big loss, and they may all be in your shoes after a few years.

I’ve had the best luck selling in Craigslist. Every other platform has been sub par.

Better sell it fast before the M5 ones come out.

> I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment.

It's going to be a non-trivial haircut. This stuff depreciates pretty fast.

How are you using the 6000 with a Mac ?

You'll probably make a profit by selling them today. I bought a M1 Max Studio with 64 GB last year off FB Marketplace for $1000 and today I'm seeing numerous 32 GB M1 Maxes for $1200-1500.

If you are in the bay area, i'm happy to buy that M3 Ultra from you, i've been unsuccessfully looking for one and can't find any.

Running LLMs on Macs is still terribly slow. They simply lack the optimizations other platforms have.

An RTX 6000 pro Blackwell is a pretty good card

You definitely want to get rid of your M3 Ultra before the M5 Ultra get officially announced.

No harm in listing it for $20k, and if it sells, that's an easy $5-10k for you.

Given that the tokens are being subsidised by a couple orders of magnitude, would it still be as cost effective long term?

I'll buy your macbook if you're trying to get rid of it!

I'm not really asking this from the perspective of whether I should buy hardware. I'm trying to understand the economics.

The AI space is moving so fast that it is hard to know which conclusions are stable. After all the discussion around local models, is the practical conclusion still that API/frontier providers have a huge structural advantage because of datacenter hardware, high utilization, batching, optimized inference stacks, and perhaps strategic pricing?

In a comparison like this, a $25k local setup versus buying tokens, what multiple are we really talking about? 10x? 100x? Or is it too workload-dependent to reduce to a single number?

Has someone written a good breakdown that separates true infrastructure efficiency from temporary underpricing/subsidy? The part I'm trying to understand is less ideological (local vs. cloud) and more basic economics.

How do you use the RTX 6000 with the Macs? Exo? I would think that would be pretty snappy if configured properly.

I'd buy that Mac studio m3

I looked into the M3 Ultra 512GB Mac Studio before it was discontinued and the as best as I could determine it just wasn't worth it... yet. The GFLOPS and memory bandwidth just arne't there even though it can hold a much larger model in memory.

But the trend here is interesting. I think by 2030 you'll be able to buy fairly cheap hardware that is currently $10k+. I don't know what this does to the trillions invested in AI data centers because the next NVidia architecture after Blackwell will essentially half the value of purchased cards overnight.

I'm not convinced Apple has yet pivoted the Mac Studio line towards this market and the expected M5 Ultras in Q3 2026 will likely be an incremental improvement rather than big leap forward but I'd like to be proven wrong.

> our infosec department doesn't buy the "zero retention" promise

They are wise to be skeptical! It is neither a promise nor zero data retention.

Look at Anthropic's Zero Data Retention policy -- and remember, this is the policy that applies to the exclusively eligible enterprise partners who can even qualify for a ZDR agreement with Anthropic:

> When ZDR is enabled, prompts and model responses generated during Claude Code sessions are processed in real time and not stored by Anthropic after the response is returned, *except where needed to comply with law or combat misuse*.

> Even with ZDR enabled, Anthropic may retain data where required by law or to address Usage Policy violations. If a session is flagged for a policy violation, *Anthropic may retain the associated inputs and outputs for up to 2 years*....

This means that Anthropic is actively inspecting all of your data with machine learning classifiers. When the usage is flagged for whatever reason as violating any aspect of Anthropic's Usage Policy, then they get to keep your data for 2 years, with no apparent limitation on what they can then use it for.

Crucially, you have ZERO guarantees about the sensitivity or specificity of these classifiers. For all anyone knows, Anthropic is silently flagging 75% of queries and retaining the data.

https://code.claude.com/docs/en/zero-data-retention

> don't seem to outperform Qwen3.6 that much in agentic coding/tasks

idk i imagine you'll hit less edges with a larger model just because.. more data

if you think of them as a kind of NN compression, it's ~obvious that the larger model can have more stuff encoded in it and hopefully accessible

i don't use LLMs much right now but using midrange models seems like an unnecessary compromise in most cases, especially since the big open models sound to be rivaling opus and not just sonnet :p

I have a 5090 machine sitting idle that I'm considering turning into a machine for my own small team (3 devs).

Are you willing to share any lessons learned, etc. that I could make use of? We are evaluating paying for a SOTA sub or trying this, and the talk about Qwen3.6-27B makes me want to try deploying this machine.

How can a single 5090 serve 80 people? Something doesn't add up here.

> Does anyone here have experience running large models in a multi-GPU setup with several RTX 6000s in a high-concurrency regime and with large context lengths? (something like Deepseek 4 Flash, Minimax 2.7 etc.)

Join the RTX6kPRO tribe!

- https://discord.gg/pYCvaQTf

- https://github.com/local-inference-lab/rtx6kpro

Wouldn't that be a fairly ideal setup for layer parallelism? That doesn't need the high-performance communication of tensor parallelism, and the high-concurrency regime would make it easy to keep the pipeline full with microbatches. You'd also be able to scale out your KV cache storage since that naturally splits layer-wise.

> 260k context

with a single 5090?

Thank you for the insight. This makes me feel confident, the L40S we are about to acquire with 48GB VRAM for engineering application should be useful for agentic coding as well.

Qwen 3.6 27B is fine but it's not in the same ballpark as GLM-5.1 or Kimi K2.6.

If you truly want to scale up, you should get the 8xH200 with NVLink.

Does anyone here have experience running large models in a multi-GPU setup with several RTX 6000s in a high-concurrency regime and with large context lengths? (something like Deepseek 4 Flash, Minimax 2.7 etc.)

For what it's worth, I've been seeing ~100 tps with 4-bit MiniMax 2.7 on two RTX 6000 boards, just running under llama-server without any optimization effort at all. I have no serious long-context experience with that setup, but at 30K context it's still above 90 tps.

If you are happy with Qwen 3.6 27B, I would personally switch the 5090 out for 2x RTX 6000s and keep running 27B. That will give you ~2x your current throughput with a lot more headroom for multiple users. More important, it would buy time to see how things develop over the next few months before you spend a whole lot of money.

Also something about subscriptions vs pay-for-usage. I feel the need to use all my weekly tokens or I'm wasting and I bet they would never get this kind of usage out of me if AI ended up being same price per token.

At the end of the article, the author has this to say:

> UPDATE: Launch was a success! 400K+ views, and multiple companies reached to use my IP. Read more here[0]

[0]https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...

Their more recent post seems to suggest it was worthwhile. https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...

Abstract/TLDR: LLMs are notoriously formulaic at writing, overusing certain tokens or phrases. I show that models trained with SFT fail to match the distribution of the training data by using Maximum Mean Discrepancy (MMD), Judge Model Quality (JMQ), and L2 Token Distribution.

Doubt it, feels basically like just an ad to get attention "Oh look, that's where the magic happens" vs running their code on existing infrastructure and thus just showing the results, like everybody else. This "feels" more "tangible".

That's the case with Self-hosting anything. It is the privacy that matters.

One value of learning on my Macbook is that mps is not as well supported as cuda which forces me to go down roads I would not have traveled.

Except this math is 10x too high (unless accelerated depreciation is all of it) - a million tokens at 28 tokens/sec and 75W and 20c/kwh should cost $0.15 not $1.50. (And less with MTP.)

It's comparing laptops to dedicated GPUs in a server environment. The best comparison would be the Mac Studio but the current release is almost 2 years old at this point. We'll see what a likely M5 Ultra Mac Studio looks like, probably in Q3 this year.

But yes, for pure inference, the M5 Max Macbook Pros probably aren't there yet. They have other utility though of course. And you can get 64GB and 128GB MBPs at a discount. Micro Center currently will let you buy a 64GB M5 Max MBP for under $4k currently, for example.

Why didn't you take into account batching, input tokens, different costs of electricity, and the fact that a laptop can still hold a decent % of its resale value, and is useful for many other tasks than running an LLM?

At then end they briefly mentioned how they started a service to post-train LLMs on producing more human, less formulaic-obviously-AI text.

One of these things is not like the others. If you don't spend the money on one of them, you can get a visit from government officials that might decide to take that "item" from you. You'd also be a worthless human to spend that money on the other 3 while not on the one.

Well a lot of people have that in their garage, even "worst" it's on wheels so even easier to steal.

I'm not saying it's worth it just that it's not such a crazy amount in comparison.

>> To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade.

The high cost and power consumption are both signs of the death of Moore's law, so you are probably correct that this system will be near state of the art for some time.

I was looking at Ultras for sale, and had same worry, so didn't end up getting one. I have some peace of mind comfort about applecare and technical repair, but i couldn't find insurance that would cover theft (or rather, i did, but it was too expensive)

Yes! It scared me too. I tried to insure it under my renter's insurance policy, but they not surprisingly refused. I had to get business insurance to cover it

Not even a single mention of gaming.

No wonder gamers hate AI bros.

I guess it was supposed to be a humorous aside, but it wasn't actually helpful because the relevant issue is when you pull more total amps from a single circuit than it's fused for (usually 15 or 20 amps in U.S. residences). The failure mode is usually tripping the circuit breaker.

That issue can often be addressed fairly easily by splitting the power draw between two adjacent circuits. You can have an electrician do it permanently or temporarily DIY it with an appropriately rated extension cord. The real issue was OP was in an apartment at the time so an electrician would have been difficult. I assume they decided to just have a system integrator build it because they didn't want to figure out how to segment and route the power rails in a dual power supply system, but it's not exactly rocket science. Problems are often more due to choosing power supplies that aren't up to their claimed spec, not pre-testing them under load or using incorrect or under-spec cables.

Agreed, if anything an electrician was needed, not a professional PC builder.

The picture shows two power supplies. Powering what is effectively one appliance from different circuits is a definite no-no, and I can't think of any circumstance where it wouldn't be in a home.

If his mains supply was sufficient to run the server and the house in the first place then the simplest solution would be to simply upgrade one of the MCBs/RCBOs on one of the circuits to the required capacity. I am not sure a landlord would even notice something like that, and if the house is wired correctly in the first place, it's unlikely to be dangerous. So going from say, 6A to 12A, on a 20A mains supply is generally fine if the gauge of wiring is correct.

Probably means hiring someone who has more knowledge about PSUs and especially about having two simultaneous PSUs. There are questions like: when you press the power button how do the two PSUs turn up and in what sequence? How do you deal with the PWR_OK signal? What if there are voltage differences between the two PSUs? What about power backfeeding?

I read this as; the "professional PC builder" would carry some sort of insurance. So it isn't really "safer", but if something goes wrong, the investment is (potentially) still safe.

Just an assumption, though!

My first advice is to have a test set with clear improvement, and a clear "wow" demo use case. There are lots of "breakthroughs" that seem good but aren't (e.g. some new architecture that doesn't mask past tokens correctly and leaks information), so people will assume it is wrong. To prevent this, you need to be extremely rigorous in your launch materials. If you can make it into a product that people can try out themselves, that goes a long way. You don't need to open source any code (I haven't yet) if people can try it out some other way like a demo website. Good luck! Ping me if you want to chat more

They posted their research results, it's linked at the end of the article.

https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...

In the article the author says they are doing reinforcement learning with LLMs.

I did this with used parts and cheaper consumer cards (3090s) and did much of the same calculations. I found it was way cheaper for me as well.

The main advantage, however, is that the friction of "this is going to cost me in tokens to even try" goes away. I was so much more willing to take chances and try new things on my own hardware than I would have been if I were paying API costs. I feel like this point isn't made clearly enough by those of us who run these absurd self-hosted inference systems.

Thanks for the write up, was a fun read. I spent an order of magnitude less, but I could relate to your story from beginning to end.

Epyc (Milan), 512gb ram, 4x 3090

You kind of bury the lede in that Article, it's a good article, well done getting interest in your work.

Will you now be selling these GPUs for a profit?

I'm sure thr plan was to build a holodeck all the time..

Agreed. And the gained time either goes toward 1) more experiments, or 2) leisure, which makes you sharper in the lab and happier overall. Not sure the "I saved $17,000 so far" framing is the most useful way to look at it, but it's a cool project and I love that people are doing this kind of thing.

Right, you can rent from a v100 from llama cloud for $0.79/hr. An h100 is $3.99 /hr.

$48000 is equal to 12000 hours of renting an h100, which is about as long as you’d spend at your job for 6 years!

They have a subsequent post (from Monday) about what they've been working on: https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...

(I would assume they haven't made a lot of $ off of this, if nothing else because they've only just put out that post and demo. They do seem to have produced a model that doesn't sound very LLM-y to my ear, though it also seems rather weak for its size.)

I am not the author, but he has been training/tuning? a model that produces text that mimics the source material in a more natural way. So getting the LLMs to produce less bland and boring LLMisms, according to the following up blog post.

citing from the article:

"I spent a long time trying high risk/high reward experiments and failing. But now I have something good. I’ve solved a major problem with LLMs. And I’m launching next Monday so we will soon see if it’s actually a breakthrough or just LLM psychosis "

Maybe ai companies today have some bounty program?

What do you want to do with the workstation? I have a similar setup:

- 512 GB

- Epyc 9684x

- 2x RTX 6000 Pro

- 1400 W PSU x 2 but in redundant mode

Mine is in a colo where it stays nice and cool. In my case, I went with less RAM and more GPUs (bought 4). Secondarily, the Max-Q blower version of an RTX 6000 Pro Blackwell is easier to keep cool and also only needs 300 W at the cost of very little performance. The non-max-q also only really use 300 W during inference, but the good thing about a lower power use is you can put more GPUs in very safely.

I assume you want the Threadripper Pro to maximize single-core performance? So you're spending a lot of time on CPU? Interesting stuff.

I gained a lot putting the machine somewhere else. TTFT on a thing like this is between 100-800 ms depending on batching and model size and so on, and your nearest datacenter is likely <10 ms. It sits on nice dual redundant power in a place where it's blown icy cool.

Good luck with your setup. If you get around to it, and end up writing about your setup on a blog, do share. Email in profile.

I think their retrospective at the end of the article is grounded and logical:

"If I were to do this again, I wouldn’t do a custom build like this. I would buy a standard datacenter server and rent space in a colocation center"

I'm sure there are use cases when renting makes sense, but it can get crazy expensive really fast if you're not careful.

You can turn off training with codex an gemini. Not sure about Anthropic.

Doesn't it benefit me if the models I use improve?

If nothing else, rosmine's DFT [1], which is what they were working on with this setup, seems like a worthwhile investigation.

While I'm skeptical that there is much of a moat, at least for the large players, it should at least hopefully set rosmine up with for the next job :)

It does seem to fix the current biggest issues with using LLMs for writing at various publishers. If you're The Economist, you have a very specific house style and you have a decent corpus of articles written in that style. At least on my reading of it, rosmine can use DFT to get a model to closely match its outputs, in terms of the language quirks that are generated, to that of the corpus it is fine tuned on. ie it will very much match the house style, particularly as it is used in writing, vs giving a system prompt to an LLM that has some Economist articles in its vast training set, and telling it to write in that style- it will do an ok job, but still exhibit LLM language quirks despite itself. Even if you feed it the specific "style guide" that they give their authors, I dare say the reality of their writing is the best place to learn, and it sounds like DFT can ground the writing of a model in a specific corpus like that.

[1]: https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...

From the author’s POV it seems like they were going to do this research regardless, so this is asking what the most cost-effective way to do that research.

Or, for a person who did have a great way to monetize the same workload they’d probably find a lot of value in reading this post.

I agree :) Also, I heard Teknium trained the original Hermes model on 2x 4090. You can do a lot with a little compute

Openrouter is great for experimenting with models. I did exactly what you're saying to test smaller models that will run on commodity hardware and determine if it might be worth it to drop $10k on hardware. For me the answer was no, but it's close. I'm very excited for the next few innovation cycles to arrive.

I've hit command+f and then looked for this.

Cloud servers have cheaper electricity, the scale of industrial-level cooling, no issues for you (as a user) with hardware failure (ie you just use a different server; it's not your problem) and can amortize their cost by running 24x7. I've seen H100 computer hours for as little as $2.

As the author notes, there are also electrical/wiring issues that cap how much compute gear you can run in a space not designed for it. I suspect a standard 20A 110V circuit can probably handle 2x RTX 6000 Pros. 15A probably can but that requires more research. Anything more than that and you're using multiple circuits, which has issues, or you need an upgraded circuit (eg 40A 240V) with all that entails (eg heavier duty cables, custom plug, etc).

Depends what you want to do and which cards you have, but usually going with any older (3rd gen+) threadripper pro setup will give you a lot of pcie lanes.

I myself run with gigabyte trx40 aorus xtreme, but since it's regular threadripper (not pro) with 4 GPUs 2 of them will run at x16 and two of them at x8 speeds

You could ask AI and get pretty far reading the answer.

I am also considering to buy 3-4x RTX 6000 Pro 96GB plus some Ryzen workstation with a grant.

Is this the best general-purpose choice as of 2026 with $50k for training, fine-tuning and running large open models?

I have the Macbook M5 MAX with 128 GB of RAM. I put its performance at roughly equivalent to the RTX 5070 Ti. The M3 Ultra 512 GB for me is about half the performance of the RTX 5070 Ti but obviously it has the ability to do more because of the increased memory.

I don't think anything compares to the nVidia chips at all.

You would install a 240v circuit (in the US) like for an electric clothes dryer.

Edit: I now see the author was in an apartment and couldn't do this, so I concede this is not responsive here.

> paid a professional pc builder

They did not. That's a mining rig not a workstation. It's visible from the photo and the chart showing multiple failures over a short period of time including the risers -- which are visibly very low quality -- failing twice.

You have 50K, you call a real expert like Puget Systems or Digital Storm.

I wonder why using 2 PSUs resulted in having slower interconnect.

There is no specs in this blogpost regarding cpu/motherboard choice, but if you go with threadripper pro they have 128 pci-e lanes for some time now, so using all GPUs at full speed shouldn't be a problem

If you split models using pipeline/layer parallelism you don't have to care about a slow interconnect, you're just slowed down a lot when running a single inference at a time as opposed to a fully pipelined minibatch. But tensor parallelism requires much faster interconnects than you could get in your average server, so I'm not sure that a different motherboard would help all that much.

what is a "professional pc builder" in 2026

Consumer motherboards can still make sense even if you leave some performance on the table. Running an actual 8x GPU server is not something you'd want to do in an apartment. Imagine the old Lucasfilm "THX" trailer where an unearthly-sounding foghorn whine rises to a sweeping crescendo at reference level, only without the decay at the end.

At the time he put this rig together, there weren't a lot of open-weight LLMs that could run well on 6x48=288 GB, so it probably wasn't a huge loss. There still aren't, really.

Right now I'm in the process of cramming Blackwell cards into an old DDR4-based Milan server, where the important thing is to be able to run large models at all. The GPU fans alone burn over 400 watts at full throttle.

Don't those Ada 6000 GPUs support NVLink? I think I can even see the cover for the connectors in OP's pic.

edit: Hm, finding mixed information online on whether that's still supported or not. Apparently it was removed in workstation GPUs.

the very last line of the article:

"If I were to do this again, I wouldn’t do a custom build like this. I would buy a standard datacenter server and rent space in a colocation center. But then I would miss saying Hi to grumbl once in a while."

The cheaper, easier solution would probably be just to get an electrician to wire up a high amperage 240V outlet just like your electric stove or dryer has, and then get a PSU that connects to that.

Would probably cost you $500-1000 depending on how difficult your home is.

In the article, he wrote "I tried to insure it under my renter’s insurance policy. They didn’t like that. I had to get business insurance to cover it.“， but he didn't say how much it cost, either.

Also a lightning strike or surge from the electric utility could fry the whole rig. Proper protection costs thousands, and even then it's not guaranteed to protect everything

It is unsafe for wires to be handling higher power than it was rated cause the wires act like very low ohm resistors. At some high enough I, you’re still gonna be generating power P=I^2R which is mainly thermal and melt the wires.