I’ve been working for a while on Kalavai (https://kalavai.net), a project to make distributed AI easy without a devops certification. There are brilliant tools out there to help AI hobbyists and devs on the software layer (shout out to vLLM and llamacpp amongst many others!) but it’s a jungle out there when it comes to procuring and managing the necessary hardware resources and orchestrating them. This has always led me to compromise on the size of the models I end up using (quantized versions, smaller models) to save cost or to play within the limits of my rig.

Today I am happy to share the first public version of our Kalavai client (totally free, forever), a CLI that helps you build an AI cluster from your everyday devices. https://github.com/kalavai-net/kalavai-client

Our first use case is distributed LLM deployment, and we hope to expand this with the help of the community.

I’d love for people interested in AI at scale (bigger than a single machine) to give it a go and provide honest feedback.

Do you share our motivation? If you tried Kalavai, did you find it useful? What would you like it to do for you? What are your painpoints when it comes to using large LLMs?

Disclaimers: - I am the creator of Kalavai - This is my first post