It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...

All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.

Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.

Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!

It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

I got the qwen 3.8 abliterated model running on my MacBook Pro M5 48GB and it's pretty nice having a local model that can do a lot of experimentation without rails.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

How many tok/s are you getting? What gen mbp?

how quick does it respond? what are specs of your laptop?

Did you read even the title?

How's it performing on the challenges?

As for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme...).

People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.

Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.

For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

> is there something fundamentally wrong with Ollama ?

Yes https://news.ycombinator.com/item?id=47788385

I know nothing about "vLLM" -> anything better compared to "llama.cpp"?

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

In my limited testing of ollama (in a Docker container because I don't want Meta touching bare metal disk), I noticed that first of all the model is abandoned. Last updated in 2023. Secondly, it basically can't do anything. Anything I asked of it resulted in a lot of "I can't do that" so it's just stupid.

I thought it was a link too because of the line under the with math but it isn’t. :/

You've missed a really great human-authored piece then.

I've been comparing against TextGen and llama.cpp while I port to LocalAI and have been surprised by what's happening over the API, even with the defaults and jinja. It's been a fair reminder not to eschew familiarizing myself with the repos.