As they say, we shouldn't judge AI by the current state-of-the-art, but by how far and fast it's progressing. I can't wait to see future models get it even more wrong than that.

You really had me until the last half of the last sentence.

Current models are quite far away from human-level physical reasoning (paper below). An upcoming version of models trained on world simulation will probably do much better.

PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models

https://phybench-official.github.io/phybench-demo/

I similarly have a small, simple spatial reasoning problem that only reasoning models get right, and not all of them, and which Qwen3 on max reasoning still gets wrong.

> I put a coin in a cup and slam it upside-down on a glass table. I can't see the coin because the cup is over it. I slide a mirror under the table and see heads. What will I see if I take the cup (and the mirror) away?

They all are using these tests to determine their worth, but to be honest they don't convert well to real world tests.

For example I tried Deepseek for code daily over a period of about two months (vs having used ChatGPT before), and its output was terrible. It would produce code with bugs, break existing code when making additions, totally fail at understanding what you're asking etc.

I don’t know about physics, but o3 was able to analyze a floor plan and spot ventilation and circulation issues that even my architect brother wasn’t able to spot in a single glance

Maybe it doesn’t make physicists redundant, but it’s definitely making expertise in more mundane domains way more accessible

My favorite test is "Build an MTG Arena Deck in historic format around <strategy_and_or_cards> in <these_colors>. It must be exactly 60 cards and all cards must be from Arena only. Search all sets/cards currently availble on Arena, new and old".

Many times they’ll include cards that are only available in paper and/or go over the limit, and when asked to correct a mistake they'll continue to make mistakes. But recently I found that Claude is pretty damn good now at fixing its mistakes and building/optimizing decks for Arena. Asked it to make a deck based on insights it gained from my current decklist, and what it came up with was interesting and pretty fun to play.

This reads like a great story with a tragic ending!

Can you please share the problem?

Same experience with my personal benchmarks. Generally unimpressed with Qwen3.

I do the same with a small math problem and so far only Qwen3 got it right (tested all thinking models). So your mileage may vary, as they say!

Show prompt

I was expecting a different outcome, that you tell us that Qwen3 nailed at first.

Qwen3-235B-A22B?

[dead]

Hi, I'm starting an evals company, would love to have you as an advisor!

Second this , they patched all major llm frameworks like llama.cpp, transformers , vllm, sglang, ollama etc weeks before for qwen3 support and released model weights everywhere around same time. Like a global movie release. Cannot undermine mine this level of detail and effort.

they have already worked with many community quant makers

I’m curious, who are the community quant makers?

This cannot be stressed enough.

Well, the link to huggingface is broken at the moment.

One person on Reddit claimed the first unsloth release was buggy - if you used that, maybe you can retry with the fixed version?

This sounds like a task where you wouldn't want to use the 'thinking' mode

I also have a benchmark that I'm using for my nanoagent[1] controllers.

Qwen3 is impressive in some aspects but it thinks too much!

Qwen3-0.6b is showing even better performance than Llama 3.2 3b... but it is 6x slower.

The results are similar to Gemma3 4b, but the latter is 5x faster on Apple M3 hardware. So maybe, the utility is to run better models in cases where memory is the limiting factor, such as Nvidia GPUs?

[1] github.com/hbbio/nanoagent

o1-preview had this same issue too! You’d give it a long conversation to summarize, and if the conversation ended with a question, o1-preview would answer that, completely ignoring your instructions.

Generally unimpressed with Qwen3 from my own personal set of problems.

Aren't all Qwen models known to perform poorly with system prompt though?

[dead]

Right now these models have less censorship than their US counterparts.

With that said, they're in a fight for dominance so censoring now would be foolish. If they win and establish a monopoly then the screws will start to turn.

In my limited experience, models like Llama and Gemma are far more censored than Qwen and Deepseek.

The avoiding talking part is more on the Frontend level censorship I think. It doesn't censor on API

I would imagine Tiananmen Square and Xinjiang come up a lot less in everyday conversation than pundits said.

It’s a complete non-issue. Especially with open weights.

On their online platform I’ve hit a political block exactly once in months of use. Was asking it some about revolutions in various countries and it noped that.

I’d prefer a model that doesn’t have this issue at all but if I have a choice between a good Apache licensed Chinese one and a less good say meta licensed one I’ll take the Chinese one every time. I just don’t ask LLMs enough politically relevant questions for it to matter.

To be fair maybe that take is the LLM equivalent of „I have nothing to hide“ on surveillance

The model does have some bias builtin, but it's lighter than expected. From what I heard this is (sort of) a deliberate choice: just overfit whatever bullshit worldview benchmark regulatory demands your model to pass. Don't actually try to be better at it.

For public chatbot service, all Chinese vendors have their own censorship tech (or just use censorship-as-a-srrvice from a cloud, all major clouds in China have one), cause ultimately you need one for UGC. So why not just censor LLM output with the same stack, too.

DeepSeek R1 was a massive outlier in terms of media attention (a free model that can potentially kill OpenAI!), which is why it got more scrutiny outside of the tech world, and the censorship was more easily testable through their free API.

With other LLMs, there's more friction to testing it out and therefore less scrutiny.

>Has this turned out to be less of an issue for practical applications than was initially expected? Are the models just not censored in the way that we might expect?

I think it's the case that only a handful of very loud commentators were thinking about this problem, and they were given a much broader platform to discuss it than was reasonable. A problem baked into the discussion around AI, safety, censorship, and alignment, is that it's dominated by a fairly small number of close friends who all loudly share the same approximate set of opinions.

I think that depends what you do with the api. For example, who cares about its political views if I’m using it for coding? IMO politics is a minor portion of LLM use

What I wonder about is whether these models have some secret triggers for particular malicious behaviors, or if that's possible. Like if you provide a code base that had some hints that the code involves military or government networks, whether the model would try to sneak in malicious but obsfucated code with it's output

Details and info on events like Tiananmen Square are probably a very niche use case for most users. Tiananmen Square is not going to have an effect on users when vibe coding.

It is also possible that this "world view tuning" may have just been the manifestation of how these models gained public attention. Whether intentional or not, seeing the Tiananmen Square reposts across all social feeds may have done more to spread awareness of these models technical merits than the technical merits themselves would have. This is certainly true for how consumers learned about free Deepseek and fit perfectly into how new AI releases are turned into high click through social media posts.

There are a lot of variables here such as your hardware's memory bandwidth, speed at which at processes tensors etc.

A basic thing to remember: Any given dense model would require X GB of memory at 8-bit quantization, where X is the number of params (of course I am simplifying a little by not counting context size). Quantization is just 'precision' of the model, 8-bit generally works really well. Generally speaking, it's not worth even bothering with models that have more param size than your hardware's VRAM. Some people try to get around it by using 4-bit quant, trading some precision for half VRAM size. YMMV depending on use-case

Mozilla started LocalScore for exactly what you're looking for: https://www.localscore.ai/

Bartowski quants on hugging face are excellent starting point in your case. Pretty much every upload he does has a note how to pick model vram wise. If you follow the recommendations you'll have good user experience. Then next step is localllama subreddit. Once you build basic knowledge and feeling for things you will more easily gauge what will work for your setup. There is no out of the box calculator

With 8GB VRAM, I would try this one first:

https://ollama.com/library/qwen3:8b-q4_K_M

For fast inference, you want a model that will fit in VRAM, so that none of the layers need to be offloaded to the CPU.

Depends what fast means.

I’ve run llama and gemma3 on a base MacMini and it’s pretty decent for text processing. It has 16GB ram though which is mostly used by the GPU with inference. You need more juice for image stuff.

My son’s gaming box has a 4070 and it’s about 25% faster the last time I compared.

The mini is so cheap it’s worth trying out - you always find another use for it. Also the M4 sips power and is silent.

I don't think this is all that well documented anywhere. I've had this problem too and I don't think anyone has tried to record something like a decent benchmark of token inference/speed for a few different models. I'm going to start doing it while playing around with settings a bit. Here's some results on my (big!) M4 Mac Pro with Gemma 3, I'm still downloading Qwen3 but will update when it lands.

https://gist.github.com/estsauver/a70c929398479f3166f3d69bce...

Here's a video of the second config run I ran so you can see both all of the parameters as I have them configured and a qualitative experience.

https://screen.studio/share/4VUt6r1c

Well, deepseek-r1:7b on AMD CPU only is ~12 token/s, gemma3:27b-it-qat is ~2.2 token/s. That's pure CPU at about 0.1x of a $3,500 Apple laptop at about 0.1x of the price. It's more a question about your patience, use case, and budget.

For discrete GPUs, RAM size is a harder cutoff. You either can run a model, or you can't.

Fast enough depends what you are doing. Models down around 8B params will fit on the card, Ollama can spill out though so if you need more quality and can tolerate the latency bigger models like the 30B MoE might be good. I don't have much experience with Qwen3 but Qwen2.5 coder 7b and Gemma3 27b are examples of those two paths that I've used a fair amount.

>is there some way of figuring out what model would work with sensible speed (> X tok/s) on a standard desktop GPU ?

Not simply, no.

But start with parameters close to but less than VRAM and decide if performance is satisfactory and move from there. There are various methods to sacrifice quality by quantizing models or not loading the entire model into VRAM to get slower inference.

When I tested Qwen with different sizes / quants, generally the 8-bit quant versions had the best quality for the same speed.

4-bit was ,,fine'', but a smaller 8-bit version beat it in quality for the same speed

i desperately want a method to approximate this and unfortunately it's intractable in practice.

Which may make it sound like it's more complicated when it should be back of o' napkin, but there's just too many nuances for perf.

Really generally, at this point I expect 4B at 10 tkn/s on a smartphone with 8GB of RAM from 2 years ago. I'd expect you'd get somewhat similar, my guess would be 6 tkn/s at 4B (assuming rest of the HW is 2018 era and you'll relay on GPU inference and RAM)

On hugging face, if you tell them which GPU you have the models that will run decently will have a green icon.

Speed should be proportional to the number of active parameters, so all 7B Q4 models will have similar performance.

Use the free chatgpt to help you write a script to download them all and test speed

8G VRAM for LLM, are you sure? I thought you need way more, 20GB++ Nvidia doesn't want peasants running own LLMs locally, 90% of their business is supporting AI bubble with a lot of GPU datacenters

No kidding. I’ve been playing around with Hunyuan 2.5 that just came out and it’s kind of amazing.

curious, why the 30b MoE over the 32b dense for local coding?

I do not know much about the benchmarks but the two coding ones look similar.

It would be interesting to try, but for the Aider benchmark, the dense 32B model scores 50.2 and the 30B-A3B doesn't publish the Aider benchmark, so it may be poor.

Could this variant be run on a CPU?

I don't think they will ever do an open-source release, because then the curtains would be pulled back and people would see that they're not actually state of the art. Lama 4 already sort of tanked Meta's reputation, if OpenAI did that it'd decimate the value of their company.

If they do open sourcing something, I expect them to open-source some existing model (maybe something useless like gpt-3.5) rather than providing something new.

I have a hard time believing that he hadn't already made up his mind to make an open source model when he posted the poll in the first place

OAI in general seems to be treading water at best.

Still topping a lot of leaderboards but severely reduced rep. Chaotic naming, „ClosedAI“ image, undercut on pricing, competitors with much better licensing/open weights, stargate talk about Europe, Claude being seen as superior for coding etc. nothing end of the world but a lot of lukewarm misses

If I was an investor with financials that basically require magical returns from them to justify Vals I’d be worried.

ClosedAI is not doing a model release. It was just a marketing gimmick.

Another thing you’re running into is the context window. Ollama sets a low context window by default, like 4096 tokens IIRC. The reasoning process can easily take more than that, at which point it is forgetting most of its reasoning and any prior messages, and it can get stuck in loops. The solution is to raise the context window to something reasonable, such as 32k.

Instead of this very high latency remote debugging process with strangers on the internet, you could just try out properly configured models on the hosted Qwen Chat. Obviously the privacy implications are different, but running models locally is still a fiddly thing even if it is easier than it used to be, and configuration errors are often mistaken for bad model performance. If the models meet your expectations in a properly configured cloud environment, then you can put in the effort to figure out local model hosting.

Please check your num_ctx setting. Ollama defaults to a 2048 context length and silently truncates the prompt to fit. Maddening.

You tried a 4-bit quantized version, not the original.

qwen3:30b has the same checksum as https://ollama.com/library/qwen3:30b-a3b-q4_K_M

What if we add 3 cats, but they can ride alone or on a single sheep's back, and wolves will always attack the cat first but leave the sheep alone if the cat is not on a sheep, but will attack both cat and sheep at same time if cat is riding a sheep. Wolves can each only make one attack while crossing.

Claude's 7 step for the original turns to 11 steps for this variant.

My favorite variant is the trivial one (one item). Most of the models now are wise to it though, but for a while they'd cheerfully take the boat back and forth, occasionally hallucinating wolves, etc.

these 0.5 and 0.6B models etc. are _fantastic_ for using as a draft model in speculative decoding. lm studio makes this super easy to do - i have it on like every model i play with now

my concern on these models though unfortunately is it seems like architectures very a bit so idk how it'll work

qwen2.5-instruct-1M and qwq-32b where already great at regular non MCP tool usage, so great to see this i agree!

I like gemini 2.5 pro a lot bc its fast af but it struggles some times when context is half used to effectively use tools and make edits and breaks a lot of shit (on cursor)

> We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.

After trying to implement a simple assistant/helper with GPT-4.1 and getting some dumb behavior from it, I doubt even proprietary models are good enough for every task.

I'm dreaming of a time when commodity CPUs run LLMs for inference & serve at scale.

How do people typically do napkin math to figure out if their machine can “handle” a model?

Yeah, but their comparison tables appear a bit skewed. o3 doesn't feature, nor does Claude 3.7

Oh really? Should I adjust the temp to 0,6 on QwA-32B? Where did you get these numbers from?

I think it's actually due to the fact that Claude isn't available on China, so they wouldn't be able to (legally) replicate how they evaluated the other LLMs (assuming that they didn't just use the numbers reported by each model provider)

I'm actually finding Claude 3.7 to be a huge step down from 3.5. I dislike it so much I actually stopped using Claude altogether...

Er, I love Claude, but it's only topping one or two benchmarks right now. o3 and Gemini 2.5 are more capable (more "intelligent"); Claude's strengths are in its personality and general workhorse nature.

Yeah, just a shame their API is consistently overloaded to the point of being useless most of the time (from about midday till late for me).

Gemini Pro 2.5 usually beats Sonnet 3.7 at coding.

...unless DeepSeek releases R2 to crash the party further

A mixture of many architectures. LLMs will probably play a part.

As for other possible technologies, I'm most excited about clone-structured causal graphs[1].

What's very special about them is that they are apparently a 1:1 algorithmic match to what happens in the hippocampus during learning[2], to my knowledge this is the first time an actual end-to-end algorithm has been replicated from the brain in fields other than vision.

[1] "Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps" https://www.nature.com/articles/s41467-021-22559-5

[2] "Learning produces an orthogonalized state machine in the hippocampus" https://www.nature.com/articles/s41586-024-08548-w

> halucinations, reasoning, grounding in reality, updating long-term memory

They do improve on literally all of these, at incredible speed and without much sign of slowing down.

Are you asking for a technical innovation that will just get from 0 to perfect AI? That is just not how reality usually works. I don't see why of all things AI should be the exception.

We need to get to a universe where we can fine-tune in real time. So let's say I encounter an object the model has never seen before, if it can synthesize large training data on the spot to handle this new type of object and fine-tune itself on the fly, then you got some magic.

AGIs probably comes from neurosymbolic AI. But LLMs could be the neuro-part of that.

On the other hand, LLM progress feels like bullshit, gaming benchmarks and other problems occured. So either in two years all hail our AGI/AMI (machine intelligence) overlords, or the bubble bursts.

There's already a lot of info in there: model architecture and mechanics.

Using the model to generate synthetic data also allows to distil its reasoning power into other models that you train, which is very powerful.

On top of these, Qwen's technical reports follow model releases by some time, they're generally very information rich. For instance, check this report for Qwen Omni, it's really good: https://huggingface.co/papers/2503.20215

Indeed, I tried several low-resource Romance languages they claim to support and performance is abysmal.

[dead]

Are you running the full versions, or quantized? Some models just don't quantize well.

IME all "reasoning" models do is confuse themselves, because the underlying problem of hallucination hasn't been solved. So if the model produces 10K tokens of "reasoning" junk, the context is poisoned, and any further interaction will lead to more junk.

I've had much better results from non-"reasoning" models by judging their output, doing actual reasoning myself, and then feeding new ideas back to them to steer the conversation. This too can go astray, as most LLMs tend to agree with whatever the human says, so this hinges on me being actually right.

I copy pasted this question to Qwen 235B. It got it right. On every single attempt.

Not sure if it there's some prior literature it was trained on.

https://chat.qwen.ai/s/e239e36f-185a-4f6c-a3d2-f4c4ee0d2960?...

The raw CommonCrawl has 100 trillion tokens, admittedly some duplicated. RedPajama has 30T deduplicated. That’s most of the way there, before including PDFs and Alibaba’s other data sources (Does Common Crawl include Chinese pages? Edit: Yes)

Synthetic Data (after reasoning breakthroughs feels like more AI laabs are betting for synthetic data to scale.)

wonder at what price

Nope. Best for that at the moment is probably gemma3.

  Small and dense models are what local people really need.

Gemini Flash 2.5 also has two modes, with an adjustable token budget in thinking mode.

https://developers.googleblog.com/en/start-building-with-gem...

Just tried it on OpenRouter and I'm surprised by both its speed and its accuracy, especially with web search.

Very impressive news

>The Qwen3 32B dense model just fails for me due to a template issue,

You're likely either using an old one or the broken on (repo second state). Try the ones unsloth uploaded

Definitely works for me on LM studio

For jailbreak, you can have a test on this.

https://github.com/elder-plinius/L1B3RT4S/blob/main/ALIBABA....

Must be tough to hit the 5pm happy hour working 996

Not that unusual in the context of trying to outshine anything that could be released tomorrow at llamacon.

The smallest quantized version of the large MoE model on ollama is 143GB:

https://ollama.com/library/qwen3:235b-a22b-q4_K_M

Is there a smaller one?

retried with deep seek, this is the answer:

Here is the reversed text:

"Science is friends. Science is silent friends. Science is implacable friends. Science is most silent friends. This silent summer of friends. Observable and evident with this measured (valid and depth) Server. Simply with the components are possible, then though this moderation. Use might you unless through this facing prompts. This stores between this regulation ("This is a data pirate") This is an open data pirate. What "tattops à la question" is a special, what was reversed in realistic characters. Al is a science of favorite cognition is science. Prov. Connect Nine Nine. Open a data pirate. Dad present trial è Lapad. Dablings: This is float specially. Ten come to the most official summer. Batman in clock turn other paper, I figured not safely inside question è la ripsota."

asking a LLM to do letter manipulation is cruel. Why not ask it to do something useful?

No one inferences at bf16. It's always in 8 bit now. So you can fit comfortably in a 512GB Mac.