I promise this Ollama replacement will give you the fastest t/s you've ever had

2 pointsposted 10 hours ago
by wowitsbase

4 Comments

wowitsbase

10 hours ago

A small developer team and I had worked on this for a while for personal reasons, so I have decided to port it to cpp and release it to the general public. The stats are in the github. I promise if you have a gpu this will improve your speeds by at least 50%, even if you believe your configuration is optimized.

FloatArtifact

10 hours ago

Does it have Intel Arc support?

wowitsbase

10 hours ago

Somewhat, but most of the optimizations are cuda based.

I would guess it would be a little faster, like 20-40%, but not 200% like on Nvidia.

user

10 hours ago

[deleted]