I used that to Generate an SVG of a pelican riding a bicycle:

https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.

If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:

  export HF_TOKEN=xxx # optional, speeds up the download
  
  ./llama-prism-b10685-7dffb15/llama serve \
    -hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
    --port 8331 -ngl 99 -fa on -c 32768

Thanks for all of your exploration in public Simon.

Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.

Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes

Where did you get these instructions?

They have a demo repo with a setup.sh script:

https://github.com/PrismML-Eng/Bonsai-demo

The release tag and weight file you suggest doesn’t match what they wrote.

It would be great to have upstream llama.cpp support for this!

I agree it makes little sense in a literal mathematical take but "it's 9x smaller" or is too much of linguistic advantage compared to "the original is 9x larger" or "it's 1/9th as large" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain short... so that's what people will adapt and interpret the meaning to be.

One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).

If we were talking about speed instead of size, i think it would be perfectly reasonable to say 9x faster. I'm not sure I agree that 9x smaller is unintuitive. It makes sense to me.

It's simple

If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9

It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number

This is widespread usage.

I don't see what makes it hard to understand.

They probably rephrased it from some more technical form like "we compressed the model by a factor of 9" or "we've improved the packing efficiency of the model by 9x". Where these are measurements of the transformation the model is undergoing, not measurements of the resulting model.

This sounds like the same kind of error as writing "0.10 cents" because it's less than a dollar when the number is in dollars regardless of how big or small it is

But we're cutting drug prices 500, 800, 1700%! Numbers nobody thought were possible.

Eh, when I read smaller with an integer multiplier, I mentally switch to the reciprocal. Easier than convincing the world not to use "9x smaller". Do you feel the same way about "9x faster"? What you're actually measuring is time, and "faster" is the reciprocal of time, similarly to "smaller" being the reciprocal of size.

I completely agree and this is a pet peeve of mine so it's nice to be validated :]

Me too!! It's a huge pet peeve. And it's so hard to get people to see how it's linguistically AND mathematically WRONG.

So you have some fun examples?

Sadly crashing on Pixel 9 Pro, but I guess phone GPU with 16GB RAM total wouldnt be enough anyway.

There's a table on the HF page that compares it against UD-Q4_K_XL and IQ2_XXS (you need to expand the dropdown): https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...

The table claims it performs on par with UD-Q4_K_XL except on OCR.

I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.

1.76 bpw number is kinda misleading if you compare it directly to IQ2/Q2. The encoding is ternary, but the quantization procedure is way more sophisticated than "round Qwen weights to {-1,0,+1}."

They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.

Unsloth has been dethroned by ISTA:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

The 3-bit quant is lossless based on benchmarks.

There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...

Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).

I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/

You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.

This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.

Tried it today on a B70 and couldn't get anything usable out of it.

Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(

A Tree if you will

Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.

Try a MoE model like Qwen3.6 35B-A3B for better tok/s

That would require Alibaba releasing a Qwen 3.8 8B first

Yes! And maybe get a hf fused webgpu runner for that model so it’s fast!

180b * 1.76 bits per weight = 39.6 gigabytes.

Best you could realistically run in 32gb is like 28gb, or a 127B param model

That's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.

What are you getting for prefill?

Bonsai Dai ?

Other names that occur to me: Orchard, Forest, Stand (of trees).

Without context, it should be number of params * 1.76 (the “effective bits per weight”) / 8

So for this one, 27B * 1.76 / 8 = 5.94 GB

For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:

300GB/s / 5.98 GB = 50.5t/s.

Realistically it’s probably a bit slower, but that is your theoretical maximum.

I'm seeing around 7.9GB of ram, 120 tokens/s on a 6000 pro blackwell.

Their first 27B bonsai was able to run on an iphone.

"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."