11 hours ago
This seems to be a wrapper around llama.cpp with several tuned parameter settings. Most of the speedup is gained by enabling speculative decoding [1]. No amazing breakthroughs, just good tuning!
10 hours ago
Speculative decoding was on in all of my tests, the gain came from moving the round's inputs off host memory.
11 hours ago
I have never understood how it is that "Ollama" manages to stay some sort of de-facto standard local hosting solution when all it does is wrap llama.cpp poorly. It's not as if llama.cpp is hard to use - it offers pre-compiled single file binaries! It implements every extant endpoint format! It has a web interface! What more could you ask for? Yet everything from agentic frameworks to vibe coded chat apps offer configurations for a variety of proprietary cloud providers, plus "Ollama". Never "llama.cpp". Why?
10 hours ago
agreed, if not for this project I've been making I would at least be using base llama.cpp
Personally I think it comes down to simplicity, but there's no reason for it's performance drops compared to llama.cpp while it's a wrapper of it.
11 hours ago