Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

285 pointsposted 4 days ago
by stared
(quesma.com)

151 Comments

spider-mario

4 days ago

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.

Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).

stared

4 days ago

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

tomrod

4 days ago

I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.

ricardobeat

4 days ago

The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless.

I'm not that well versed in statistics, but a standard box plot is probably the best alternative

stared

4 days ago

A single result is binary. All we get from a run is which tasks were solved, which weren’t.

breezybottom

4 days ago

Both ways involve sophistry. If you don't like dirty tricks, statistics isn't for you.

mitxela

3 days ago

> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value).

They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.

spider-mario

3 days ago

It’s not reaching. See my response here: https://news.ycombinator.com/item?id=49622516

> Like saying the probability your uncle died of a shark attack is either 0 or 1.

For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t just stop at “it was drawn from a pool of intervals of which 95% contain the true value” since you have more information than that (the numerical bounds).

mitxela

2 days ago

If I pick an interval uniformly from a pool of intervals of which 95% contain the true value, it has a 95% chance of containing the true value.

Just like if I pick a ball from a bag of balls of which 50% are red, it has a 50% chance of being red.

spider-mario

2 days ago

If you pick a ball from a bag of balls of which half are fuzzy and half are smooth, then before picking the ball, there is indeed a 50% chance that you will pick a fuzzy one. But once you pick it and have it in your hand and can feel the texture, are you still going to say “there’s a 50% chance that this specific ball I am holding is fuzzy because half of the balls in there were”?

diseasedyak

4 days ago

Yah, prediction interval instead, right? (I'm still learning statistics)

Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

spider-mario

4 days ago

Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution.

Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.

95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.

95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).

Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.

jmalicki

4 days ago

> a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well

Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.

Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.

Does that mean it's valid? Well, the confidence interval is as equally valid as a lot of credible intervals, which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it, it is not a different procedure that produces different numbers.

spider-mario

3 days ago

> Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.

The confidence interval is “valid” for what its definition is. But:

- saying “there is a 95% chance that this interval contains the true value because it’s a 95% credible interval” is valid;

- saying “there is a 95% chance that this interval contains the true value because it’s a 95% confidence interval” is not.

It’s very much possible to reach a correct conclusion with invalid reasoning. As Jaynes put it:

“As we stress repeatedly, from a false proposition – or from a fallacious argument that leads to a false proposition – all propositions, true and false, may be deduced. But this is just the danger; if fallacious reasoning always led to absurd conclusions, it would be found out at once and corrected. But once an easy, shortcut mode of reasoning has led to a few correct results, almost everybody accepts it; those who try to warn against it are not listened to.”

> Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.

Right, but if you get that by happenstance and not from the fact that it’s a confidence interval, the latter isn’t really getting you anything.

Per my first link: “One of the misconceptions regarding the relationship between Bayesian inference and frequentist inference is that they will lead to the same inferences, and hence all confidence intervals can simply be interpreted in a Bayesian way. In the case where data are normally distributed, for instance, there is a particular prior that will lead to a confidence interval that is numerically identical to Bayesian credible intervals computed using the Bayesian posterior (Jeffreys 1961; Lindley 1965). This might lead one to suspect that it does not matter whether one uses confidence procedures or Bayesian procedures. We showed, however, that confidence intervals and credible intervals can disagree markedly. The only way to know that a confidence interval is numerically identical to some credible interval is to prove it. The correspondence cannot — and should not — be assumed. More broadly, the defense of confidence procedures by noting that, in some restricted cases, they numerically correspond to Bayesian procedures is actually no defense at all. One must first choose which confidence procedure, of many, to use; if one is committed to the procedure that allows a Bayesian interpretation, then one’s time is much better spent simply applying Bayesian theory. If the benefits of Bayesian theory are desired — and they clearly are, by proponents of confidence intervals — then there is no reason why Bayesian inference should not be applied in its full generality, rather than using the occasional correspondence with credible intervals as a hand-waving defense of confidence intervals.”

> which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it

You don’t get to choose to interpret a confidence interval as if it had the meaning of a Bayesian one. If you want to be able to interpret a computed interval as “there is a 95% chance that this interval contains the true parameter value”, only the Bayesian procedure guarantees it, and it can very well produce different numbers. So it is a better alternative in that sense.

Dylan16807

3 days ago

If 95% of the intervals in your set of intervals include μ, and you randomly pick one of them, in what way is that interval not 95% likely to contain μ? Ignoring the frequentist pedantry that "likelyhood is the wrong word", is there a way for a different number to be the correct number?

spider-mario

3 days ago

https://link.springer.com/article/10.3758/s13423-015-0947-8#...

https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... (+ Cauchy example below)

It’s like with test accuracy. Test accuracy is the pre-test probability that the test will give a correct result. But once you have a positive or negative result, which way it turned out plays a part in computing the predictive value. Likewise, once you have computed the interval, the specific bounds you ended up getting can affect the plausibility that they contain the true value.

Dylan16807

3 days ago

Your first link is just the pedantry. The second link... I don't see any issue for the question I asked? The confidence interval is supposed to be wrong a certain percent of the time. It being so wrong it disproves itself is funny but the 90% is still 90% isn't it? They just landed in a particularly harsh part of the 10%.

The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.

spider-mario

3 days ago

If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”).

But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to make the post-data reasoning that “because it was picked at random from the set of 90% confidence intervals, it has a 90% chance of containing the true parameter”.

Again, it’s like with medical tests. If a test has 90% sensitivity and 90% specificity, it has 90% accuracy (it will, in 90% of cases, produce a result that matches disease status) – a pre-data statement on the test result (/ on the confidence interval that we will compute). But it does not follow that, if you screen an asymptomatic patient with low prior probability of disease and get a positive result, they have a 90% chance of having the disease – a post-data statement on disease status, given the test result (/ on where the parameter lies, given the interval).

> The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.

It means you know in the individual case that the specific confidence interval does not in fact have a 90% chance of containing the parameter.

user

3 days ago

[deleted]

Dylan16807

3 days ago

I can see the analogy here but a test like that being binary throws things off and that's also super asymmetrical error. Weren't we sampling a gaussian?

spider-mario

3 days ago

It’s the same principle. The “trivial interval” from the first link (the one you called pedantry), which is applicable to a Gaussian and tweakable to have different coverage than 50%, clearly shows that being an n% confidence interval, on its own, doesn’t guarantee the ability to directly translate that n% into post-data inferences. Likewise, in the truncated exponential example, when you compute that confidence interval of [12.1471, 13.8264], you know that the probability that it contains θ is 0%. In the Cauchy example, when you compute [-2.31, 10.31] as your 90% CI from the two samples 3 and 5, you can then compute that it actually has a >99% chance of containing θ (table I).

Dylan16807

3 days ago

The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.

The examples that actually ruin the number so far have been asymmetrical. Is there a way to do it with a gaussian? Especially if you're trying to make a reasonable internal?

spider-mario

2 days ago

> The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.

Sorry, I don’t follow. Let’s say I want to compute a 50% confidence interval for the unknown mean of a Gaussian distribution. I sample two numbers from the distribution, get 9 and 7, compute the interval according to the trivial procedure and get (-∞, ∞). Does the interval (-∞, ∞) have a 50% probability of containing the mean of that Gaussian distribution? I would think it’s closer to 100%.

If what you are saying is “it’s meaningless to talk about the probability of that specific interval containing the unknown-but-fixed parameter” then that’s the purely frequentist view and then you also agree that it’s meaningless to say that [-2.31, 10.31] has a 90% chance of containing the location parameter of the Cauchy distribution that happened to yield the samples 3 and 5. Incidentally, what asymmetry are you referring to in the Cauchy example?

Dylan16807

2 days ago

I misread the Cauchy example since I was going too fast, nevermind the symmetry part.

> then that’s the purely frequentist view and then you also agree that it’s meaningless

I'm saying that when you hit "meaningless" you can back up a step to where you actually had randomness and look at that distribution, which gets rid of a lot of these issues.

But after looking at these examples I think it only makes sense in limited circumstances to do that. Like in the trivial example: your final distribution isn't based on the probability of the mean being any particular number. The only probability was back a step and that was 50%.

At this point I still don't think it's objectively wrong to say a particular interval above is 95% likely, but there's too many ways to interpret the statement so nobody should say it is.

The way we're calculating that these intervals are "wrong" is by looking at all the possible parameters that could have given us the samples we got, and checking how often the range contains the parameter. That's a useful calculation but is it the one people expect? I think that depends on the situation. Treating the parameter as being the thing we sample over is misleading, but treating it as fixed is also misleading.

Snowfield9571

a day ago

just read a book on Bayesian stats bro. Probably just the intro of most of them will get you to your answer.

jnwatson

4 days ago

Mind blown. The more I read about statistics, the less I know.

exogenousdata

4 days ago

“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)

maCDzP

4 days ago

Thank you for these, coz I learned a lot! Great that they are open access.

soltanov

3 days ago

Task success is not enough; reliability and cost matter.

fr2029

4 days ago

the 2nd derivate of shannon covariance of noise begs to differ

user

4 days ago

[deleted]

alentred

4 days ago

I would be very interested in a similar benchmark for *KV cache* quantizations.

I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.

I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.

skolos

4 days ago

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.

I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.

my coding agents regularly reach 200k context used without noticeable degradation.

P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2

D13Fd

4 days ago

I’m running 27B on a 5090 as well, and the results have been really strong. It does almost as well as, and sometimes better than, a 121gb DS4 model running on an M5 Max 128gb. 27B also flies on the 5090, and at medium think it returns results many times faster than my DS4 setup (the default xhigh is basically broken, though).

For the kinds of things I use a local model for (legal document review), it’s just spectacular. It also has good vision support. I’ve been using 27B more and more over DS4.

redox99

3 days ago

Those are really nice numbers. With that t/s, no network latency or queueing it must feel much snappier than cloud models.

skolos

3 days ago

it is, unless it set to xhigh - it really likes generating tons of tokens for its thinking. unfortunately, for decently reliable coding results you want it on xhigh ...

skolos

4 days ago

There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:

https://arxiv.org/html/2609.04098

quotemstr

4 days ago

You don't have to quantize all layers and all dimensions uniformly, FWIW

sharmajai

4 days ago

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.

It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).

So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.

I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.

Thanks for doing and sharing the research!

anyfoo

4 days ago

Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.

I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.

sharmajai

3 days ago

Without knowing anything about your benchmark, it might be your harness at fault here. I say this because with longer thinking you run the risk of filling up your context faster and you need a good context compaction strategy in your harness to mitigate that.

I have heard good things about Pi which supports auto-compaction, but I can't personally vouch for it since I use my own.

anyfoo

3 days ago

You're probably partly right (though I use the same context size for all models, so there still is a difference in the model/quantization itself), but I definitely see it "thinking itself into an extremely expensive and error-prone approach without backing out". A common "failure mode" of my benchmark with <Q6 is "okay, I will implement a simulator", and when I see that I think to myself "please, no, you don't need to do that at all, you just missed a tiny thing" (but if I would type that into the session, it would void the benchmark; the idea is that I wouldn't know the solution myself after all).

Especially because Qwen3.8 27B does not seem to be good enough to implement the simulator with all its intricacies and tiny subtleties correctly. So it's churning for hours with no real progress, where it just had to explore the initial problem space a tiny bit more.

And very interestingly, even with Q6, turning off reasoning entirely does not make it fall into that trap, or many others, and it more often solves the task in record time. Both because of what I just described, but also because the endless reasoning costs a lot of token, which significantly translates to time on a home rig...

sharmajai

3 days ago

I see, maybe your use case is indeed pathological for Qwen 3.8 27b. Another thing to try if you are using llama.cpp is "reasoning budget". That makes the thinking stop after the budget has been reached and inserts a custom message you can choose, so something like "you have thought for too long, now continue with the execution ..."

For what it's worth, if you haven't already, you can also let it run overnight (if you have compaction enabled) to see if it ever gets out of that hole. The reason `xhigh` is the default is because 3.8 is trying to optimize for long horizon tasks where monitoring its every single thinking misstep might not be a good use of our time.

seemaze

4 days ago

As they say, time is money.

In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

chmod775

4 days ago

Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.

celrod

4 days ago

If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound. The kv cache was still bf16, so a few ops are the same between quants.

conmod278

4 days ago

Computer science has known the tradeoffs between memory and compute since ages ago. The same could be reflected here.

phh

2 days ago

Relatedly, someone found that on a Qwen MoE, increasing the number of experts at the deep layers reduces the number of thinking tokens. So while you mention that the LLM is capable of thinking more if it's thinking less clearly, it looks like it's also capable of thinking less if it's thinking more clearly?

This has been validated on Qwen3.6-35B-A3B and Qwen3.8-Flash-Next, but with the caveat that it was always on quants.

https://github.com/vagrillo/llama.cpp/blob/moe-expansion/ben...

anon291

4 days ago

I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)

kennywinker

4 days ago

Fair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.

lowbloodsugar

4 days ago

If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.

anyfoo

4 days ago

I observed this as well. I have problems that get solved very quickly using non-thinking mode, but in thinking mode it digs itself into giant holes, that it sometimes doesn't manage to escape from.

Like wanting to implement an entire emulator to figure out something it could easily glean otherwise (and does, in non-thinking mode).

purpleflame1257

4 days ago

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

civvv

4 days ago

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

zenoprax

3 days ago

9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream.

I'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.

Forgeties79

4 days ago

What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates

civvv

4 days ago

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

slim

4 days ago

Running Q3 on 5060ti with 64k context. It runs great

brynx97

4 days ago

Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.

civvv

4 days ago

I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

dofm

4 days ago

There is an interesting new dynamic 3 bit quantisation I have been meaning to test:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.

jadbox

4 days ago

Q3 XL and Q3 XS are the two I'm trying to decide on

selectodude

4 days ago

I have a 5080, three OpenAI Pro token resets, and I’m on paternity leave. Astra seems pretty clever. Maybe I’ll give it a task.

kmike84

4 days ago

Measuring quality e2e definitely makes sense.

But I think there is a bit more to this:

> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.

A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.

I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.

You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.

Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.

Still investigating it though :)

stared

3 days ago

Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.

On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.

So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.

kmike84

3 days ago

Yeah, makes sense; e2e is different and valuable, KLD is not a replacement.

As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?

stared

3 days ago

Nope.

If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.

xscott

3 days ago

> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext

Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing:

Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baseline for stats)

Gemma 4 26B: "Dude, I'm certain I wouldn't have said this." (Bad KL)

Gemma 4 12B: "Umm, Me either!" (Similarly Bad KL)

Gemma 4 E4B: "I might say almost anything, this is fine." (Much better KL!!!)

Gemma 4 E2B: "I'm basically a toy. Let's play a game!" (Same KL as E4B)

Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.

nullc

3 days ago

> Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.

Way better than wikitext-- but tells you nothing about errors tending to compound or cancel out.

Like say a test shows that only one token in a 10,000 token test would be different. Sounds very close, ship it!-- but what if trajectories with that single different token guarantees failure because it sets in motion a cascade of differences that ultimately result in a final distribution that doesn't include the solution?

Farmadupe

4 days ago

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?

Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),

But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?

Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?

stared

4 days ago

I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.

So, if there are irrelevant remarks, these are mine. :)

Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).

Ey7NFZ3P0nzAe

3 days ago

Hi,

If you ever get to writing a blog post about kv cache quantisation, i'm interested in quantising K differently than V

stared

3 days ago

If you like, you can run these tests yourself as well. It is ~$500 per a single combination, and assuming no failed runs.

Ey7NFZ3P0nzAe

6 hours ago

I don't have anywhere near that kind of money to spend. Sorry.

cogman10

4 days ago

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.

The line is "Is this an interesting and accurate article that concisely makes it's case".

LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.

I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.

breezybottom

4 days ago

It's not nearly as annoying as being tricked into reading them.

Farmadupe

4 days ago

(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)

-----

Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.

dofm

4 days ago

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem.

In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

wiml

4 days ago

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audience. The section and paragraph breaks are meaningless, the metaphors are unenlightening, the rhythm is exhaustingly uniform. Pre-LLM, you'd find this kind of text in marketing materials, corporate PR, and heavily padded sophomore essays.

jameshart

4 days ago

Possibly an inevitability, since LLM text is specifically generated to be low entropy. It’s smooth, optimized for minimal perplexity.

And if you don’t find any of the word choices a little bit perplexing, your brain can just gloss over it. There is no meaning here. It’s like smalltalk.

dofm

4 days ago

Interesting — do you happen to have any suggestions for something I could read about this?

It really bugs me, this aspect of LLMs, and yet I find myself sounding like a lunatic when I describe it, until others chime in and say they have the same experience.

dofm

4 days ago

The weird, sort of sub-surface uniformity is what I notice. It appears to be doing all the right things, but you can't feel the writer.

It is like a rollercoaster that doesn't have any jerk or jounce.

wiml

4 days ago

And reading my own comment it has the same kinds of problems (lookit all those 3-example lists!). I wouldn't hire me for an editing job.

cogman10

4 days ago

Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.

dofm

4 days ago

I don't think so, no, because it extends to LLM-generated text I want to recall.

I use LLMs to generate starter/tutorial material. I may hate the way Claude writes but I absolutely don't hate the way Gemma 4 writes. But I have to continuously consult it in a way that I do not with human-written text, which gets its message across in a more persistent way I find less troubling.

(This non-memorability extends to AI images and video.)

You are right that there are some confounding factors in my life but while I was worried about middle age affecting recall, I find actually I am still remembering stuff humans wrote pretty much fine.

ETA: there is one thing that I have noticed that does affect recall that is specific to LLMs: watching text roll out word-by-word in LLM chat, I think, damages recall. It's engaging the wrong kind of memory and focus.

So I now let it generate and I'm trying to find decent ways to format it e.g. as PDF, to give it the best chance.

This could be generational; there were many studies twenty years ago that suggest that people a decade or so younger than me who grew up with full colour books and magazines and multimedia can read less-linear text layouts more comfortably, for example, so it's not out of the question that there's something generational going on here too.

lowbloodsugar

4 days ago

Sure, but if you're going to publish it, at least run it through an edit prompt and tell it to remove clickbait "Its not X, its Y" rubbish. Like literally calling examples clickbait in the prompt has given me better results. Interestingly, I have a lot less trouble with the first draft with Qwen than with Opus.

JSR_FDED

4 days ago

Except that wasting the reader’s time became a lot easier with LLMs.

cogman10

4 days ago

Yes, LLMs make it a lot easier to produce a lot more garbage. That's not an exception to my point. If something is well written then it doesn't waste the reader's time.

vardalab

4 days ago

Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?

kennywinker

4 days ago

Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.

nhecker

4 days ago

I normally don't appreciate snarky comments on HN, me being but a simple curmudgeon wanting pure information and the occasional opinion. but this one actually makes a good point, even if it's not stated explicitly. Reading this book is a wonderful experience because of the way it's written. Reading the plot summary (from an AI or otherwise) yields nothing but the dry husk of that experience.

nottorp

4 days ago

Well, sadly it's off topic on HN where everyone seems to only read non fiction.

Or they don't admit to reading fiction...

clircle

4 days ago

I think the advice is the same regardless of AI use: read articles written by authors that have a history of high quality writing.

JSR_FDED

4 days ago

You don’t need anyone’s permission. You have only so much attention, why spend it wading through slop?

anyfoo

4 days ago

I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution.

To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.

The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.

Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.

But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.

teaearlgraycold

4 days ago

I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.

anyfoo

4 days ago

Depends on the actual audience, I guess. My stated “wisdom” comes in part from /r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.

But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.

neonstatic

3 days ago

> my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.

That sub is games, porn, and complaints about not having money. A waste of time.

KennyBlanken

4 days ago

It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.

General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.

What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?

Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?

nateb2022

2 days ago

> For comparison, DeepSeek V4 Flash 0731, a 284B model, costs around $0.1/Mtok for output from the cheapest providers on OpenRouter. I am not sure how much of this difference comes from the efficiency of running models at scale, pricing strategy, or popularity.

Due in large part to DeepSeek's MoE architecture. Generating 1 token through DS4's 13B active params requires roughly half the FLOPs to generate a token through the dense Qwen 27B.

seamossfet

4 days ago

If you want to do a 1-bit model you have to QAT at pre-training with way more data than chinchilla to compensate for the cliffs (like 50x). Quantization on an existing pre-trained model will almost always collapse at 1-bit

bellowsgulch

4 days ago

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.

Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.

spider-mario

4 days ago

> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.

Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.

SwellJoe

4 days ago

Even at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model, in my testing.

lowbloodsugar

4 days ago

I've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.

bellowsgulch

4 days ago

Thanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.

Xeoncross

4 days ago

I leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf

sidewndr46

4 days ago

I've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster

ThrowawayTestr

4 days ago

I treat it like image gen. Send a prompt then come back in 40 minutes.

syntaxing

4 days ago

I’m more curious how each 4 bit quant compares. It seems like NVFP4 outperforms Q4_K_M in terms of speed and top 1 but is only good for expensive Nvidia cards

user

4 days ago

[deleted]

anyfoo

4 days ago

> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.

Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?

magnat

4 days ago

Those are multiple-choice questions. If some of them are "trick questions", where obvious answer (e.g. the value taken directly from question's text) is wrong, bad model might perform worse than a dice.

On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.

user

4 days ago

[deleted]

kouteiheika

4 days ago

Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.

wgd

4 days ago

It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.

user

4 days ago

[deleted]

qsbuilder

4 days ago

I always wonder is it safe to run one of these models on a personal pc, or do you guys recommend something like docker, sorry a bit new to all of this.

bitwize

4 days ago

Yes, it's fine to run a model on bare metal. The model is just a token predictor. Leave out the fine semantics about this; it's a function taking a set of input tokens to output tokens.

It can't mess with your computer or files until you hook it to a harness, which interprets some of the model's output as commands to execute. So, model on bare metal, harness in a container or VM.

mrbonner

4 days ago

I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.

rvba

4 days ago

Those benchmarks are very interesting.

But is there any model that actually works in a decent way at quantization of 1?

dvh

4 days ago

Could this be used to estimate how many fingers LLM have?

quietraster

4 days ago

the 4-bit matching bf16 on terminal-bench is a useful data

zrail

4 days ago

I've been running Unsloth IQ3_S on my 5060ti with mmproj offloaded, getting 600-1000 prefill and 30-50 tg with this config:

       /data/llm/llama.cpp/build/bin/llama-server
        --threads 4
        --threads-batch 8
        --batch-size 4096
        --ubatch-size 256
        --port 9999
        --temp "1.0"
        --top-p "0.95"
        --top-k "20"
        --min-p "0.0"
        --presence-penalty "0.0"
        --reasoning auto
        --reasoning-preserve
        --reasoning-budget 4096
        --gpu-layers-draft all
        --spec-type draft-mtp,ngram-map-k4v,ngram-mod
        --spec-draft-n-max 3
        --spec-draft-p-min 0.75
        --spec-ngram-mod-n-match 24
        --spec-ngram-mod-n-min 4
        --spec-ngram-mod-n-max 16
        --spec-ngram-map-k4v-size-n 8
        --spec-ngram-map-k4v-size-m 16
        --spec-ngram-map-k4v-min-hits 1
        --n-gpu-layers all
        --ctx-size 131072
        --repeat-penalty 1.0
        --jinja
        --metrics
        --model /data/llm/models/unsloth/Qwen3.8-27B-UD-IQ3_S.gguf
        --chat-template-file /data/llm/models/qwen3.6-chat-template.jinja
        --fit off
        --flash-attn on
        --cors-origins localhost
        --mmproj /data/llm/models/unsloth/Qwen3.8/mmproj-BF16.gguf
        --no-mmproj-offload
        --parallel 1
        --kv-unified
        --cache-type-k q4_0
        --cache-type-v q4_0
        --cache-type-k-draft q4_0
        --cache-type-v-draft q4_0

zrail

4 days ago

Too late to edit, but a few other things to note: I minmaxed the draft config. On my typical coding workloads it gets around 70% acceptance, more variable on prose.

The chat template is froggeric's fixed qwen template, v22.5 as of today.

dotinvictim

4 days ago

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

kennywinker

4 days ago

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

TomBombadildoze

4 days ago

Beats Opus 4.6 at what exactly? It certainly isn't code.

I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.

kennywinker

4 days ago

https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o...

Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...

Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.

If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.

brandon272

4 days ago

What configuration? What harness? These matter greatly to how a local model performs, in my experience.

fshr

4 days ago

> 5060

I think you're referring to a 5060Ti 16GB, yes?

32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.

kennywinker

4 days ago

Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.

But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.

Winfred-zz

4 days ago

In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big.

qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.

noir_lord

3 days ago

It’s genuinely the first local model that has actually made me keep it around, it’s pretty good at spotting things existing tooling can’t, like comment rot/drift/summarising accurate git changes, it’s also the first local model that pushes back when it thinks it is right (I don’t dislike that even when it is wrong, it still hallucinates things that don’t actually work the way it confidently states they do).

It does clear the useful enough to be worth it bar though.

Zero interest in remote models but local ones if they offer utility, sure.

Runs pretty well on a 7900XTX as well.