I take issue with your numbers. A $20k system and $1000/month in power are way off. Like both an order of magnitude off.

> Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup

It’s not “they” talking about it. I am personally running the machine I described, and ~20t/s is the average of what I get at every point in 128k context for coding tasks. Dense models like qwen3.8-27b slow down at longer contexts, but flash-next does not. It’s slower for prose, but I don’t really do much of that. Yes it’s barely usable speeds - i’ll be adding a better gpu in a day or two when it arrives.

To run 4bit quant and 256k context, yes you’re going to have to spend more than $600. But not $20k. I have friends running systems that cost less than $2k getting >80t/s output and >1500t/s prompt processing, at 256k context, 4bit quant. A 5060ti with 16gb vram is all you really need to add to this config to get into that performance range.

I live somewhere with cheap power, which helps - but we’re talking about maybe 250W peak power consumption (180W for the gpu, the rest for the system). So if you ran that full tilt 24/7 it’d be ~6kwh per day.

Assuming you have very high power costs, that could work out to something crazy like $3/day, $90/month. I have no idea how you imagine getting to $1000/month without running a fleet of machines like this cranking out tokens 100% of the time.

It depends on your workload and use case IMO. If you work at a company that is using LLM coding as staff augmentation (my current and last job) a subscription setup and (usually) buying tokens is cheaper. As someone working on a side project I’ve been getting a really good ROI on my box made from used parts running Qwen models. I need something that will make progress but I’m sending a prompt, going about my day/dayjob, and check back in.

There’s no way I’ll beat a $200 subscription on total output but I am too busy piloting LLMs, taking care of my kid, having a social life, etc to max out a cloud subscription. I can however blow through the lower tiers of cloud subscriptions when doing hobby work.

A bunch of what I’m doing these days is in the category of people doing porting of ancient software or unexpected software. It’s fun and every so often I can contribute stuff back to the community. The box runs at under 1kW and I have the ability to remote manage it to turn it on and off when I’m not using it. Given it’s winter it’s a slightly annoying sounding but equally useful space heater in some ways .

With current software you don't need those 6x RTX3090 anymore. With strata you get >50 t/s on Q4 on a single 5070Ti. And with 192GB RAM you can run Q6 without reducing speed too much.

With 6x3090 I'd most likely go for even bigger - DeepSeek?

That chart is comparing the cost of buying tokens, not the cost of generating your own. The system I specced out can generate 5.2M tokens per day.

That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.

If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.

But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.

imho those prices are artificially deflated to serve as a loss leader. Either subscription cost goes up or some other model such as advertising fills the gap.

I don't really understand the argument being made here. The best models are better, but they're not that much better. If I'm paying only 20% as much for 20% less performance, I can make up the 20% and have abundant cash or compute left over to spare.

The other thing that matters to me is the tooling. I could use Deepseek Harness with Claude but I'd rather support the same people that brought me DSH.

This makes no sense. DS4.1 flash is the cheapest model basically everywhere

From those retailers with billions of debt?

Seems more like a final fight against open weight models than a sustainable business model.

Cool story bro

Astra and 5.5 came out in september, so I didn't compare to them I compared to "where the SOTA was less than a year ago".

But ok, let's compare: https://benchlm.ai/compare/gpt-6-astra-vs-qwen3-8-flash-next

Astra's score: 85.8/100

Qwen3.8-flash-next's score: 55.87/100

So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.

My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.

Assuming that you need frontier models like Opus or Astra for every use case that you'd throw at an LLM is also silly

> there’s no world where “normies” can do this.

Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.

For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.

The normie use cases need less performance, so it could be fine. Review this email, presentation slide, make a flyer for the soccer game, what should I do in Barcelona, etc. If you are not running gas town, you don't necessarily need that much.

nobody really meaning to do that work on 20 tok/sec, they mean if you have something like 2-4x5090 or few 'rtx6000 pro' you probably will able get speed around 100+t/s and with that and right harness (customized for specific work/model pair) it will work well enough and may be even better for some cases (like if you are doing security and constantly banned by Antropic/OpenAI)

Sure, but electricity cost is not a lot for me. i was already running a home server, so it’s mostly just the gpu. The gpu i’m currently running is max 75w which works out to about $0.25/day if it’s operating 100% of the time - the gpu I’m planning to upgrade to runs at 180w or about $0.6/day.

Not everyone has as cheap power as I do, so you’re right it’s worth factoring in - but in my case it’s a very small factor

Some of the math I see in this thread justifying local LLM hardware reminds me of “hobby math” where someone wants to justify a hobby expenditure and tortures the comparisons until they think they’re winning.

You do local LLM stuff either because you’re strict about privacy or because you want to play with it.

Any argument that you’re going to replace a Claude subscription with a highly quantized model at 20 tokens/sec is delusional thinking.

I wasn't describing the best option, I was describing the cheapest option.

I know that configuration works, because I am running that configuration right now.

I thought I was pretty specific? But ok more details!

All used gear. A dell 5810 or similar (x99 motherboard, 2011-3 socket) - should cost well under $100 with no ram or hd, upgrade the CPU to something with lots of cores (e5-26xx v4) for ~$50 (check ebay if there's none local). Buy at least 64gb of ddr4 registered ecc ram - I’d shoot for 96gb - if you pay more than $50 for 16gb you're over-paying. Workstations like this usually have 6 or 8 ram slots so 616 is a great config. If you go up to 256gb, I think it unlocks running deepseek, but I've not looked into that much.

Then the GPU is the hard part. A $100 1070ti or similar with 8gb is the very least thing you can put in and get usable performance. Barely* usable. 10gb, 12gb or 16gb vram is much better. You can also double up - two of the same make & generation gpu to double the vram. You can mix generations, but you have to be careful as there are incompatible combinations. I know AMD GPUs can work, but I have no experience with them so I have no advice to give on that front except don't mix nvidia gpus with amd gpus.

Pascal generation gpus (like the 1070ti) don’t have much computing power so prompt processing suffers. Newer is better, more VRAM is better. Server GPUs are usually cheaper (e.g. the P100) but require screwing around with 3D printed fan shrouds for cooling, and they have terrible idle power consumption.

If you want to spend more, you can get a lot more capable machine - but then you're off my map. Personally if I had more to spend I would set up this exact machine and then put all the extra money into a better GPU. Unless you have money to burn I would avoid any system that uses ddr5, since the ram cost is a big jump up.

I prefer my setup to handing my money and leverage over me to the sociopaths.

And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.

I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.

If that scales with parameters I don't see why QAT shouldn't be the norm.

(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)

sometimes yes sometimes no :)

that's why different models exist.

Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.

That's not the speed of the inference, but the fact that the default parameters are to max thinking. You can even disable thinking for many queries, or lower it to a smaller value.

Yes, Deepseek consumes a higher number of tokens per turn that comparable models (the latest Muse for example). This is just my experience, not a scientific study.

I use lemonade llama.cpp with a 2-bit quant of DeepseekV4F 0731 that runs at about 15 tokens a second. I use it as my "smol" model in omp, with default going to OpenCode hosted Deepseek (faster inference).

I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!

https://github.com/lemonade-sdk/llamacpp-rocm

I've switched from halogen to gufo a while ago: https://github.com/gufo-org/gufo It's OSS and I think it's faster.

43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.

all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.

So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.

I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.

Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.

Try using llama.cpp directly and tune it for your hardware. I had gemma 26b a3b running at 400-500t/s prefill and 40ish t/s generation speed with a RTX A3000 GPU with only 4.5GB free VRAM. A better model than gemma itself may help you tube the parameters. Gemma wasn't that helpful in that respect.

Try strata. I’m not sure how well it handles igpus but if it does i’m pretty sure that system will go screaming fast

Vibes from localLLAmA suggest a larger model (e.g. DSF) at a low quant still beats a smaller model (e.g. qwen 27B) at a higher quant.

Where that threshold is crossed is up for debate.

This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.

This but also the kv quantization shouldn’t be lower than 8 bits.

There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)

Next frontier is achieving good output with that kind of quant dimension

PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.

Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.

Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.

This is the point where you could ask an agent to unblock you. Don’t even need a subscription. Download opencode and pick the GLM-5.3-Flash (Free) model. Just be mindful to not use it on anything you don’t want to end up in training.

Scan for exploits, spyware and viruses, actually automatically format the PRs if they don't match and warn the submitter, automatically upmerge main/master into branches under consideration where possible and ensure all new functionality has test cases (not the same as code coverage).

When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.

If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.

I'm not saying 1.7K a week, but they aren't keeping on top of the PRs being raised so the number just keeps increasing, unless bots just close them because of inactivity.

At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).

Edit: just checked and it's now 1.9K PRs.

Most people can’t see a difference between scaffolding and an end product, sadly.

Have you tried Strata?