Shameless plug, but super relevant maybe-
I’m a product marketing guy for Nutanix. We have a new LLM deployment solution that can run on any CNCF Kubernetes where you can deploy LLMs from a validated list from Hugging Face or NVIDIA NIM, or upload your own.
You can then create RBAC-controlled endpoints to connect your GenAI apps up to all with a point and click interface.
Check it out with basic walkthrough on the product page: https://www.nutanix.com/products/nutanix-enterprise-ai
I see large companies leveraging their large technical teams and external consulancies and technology providers to build their own e.g. banks on IBM. And I think for small companies it's impossible to beat the value for money a Saas type solution provides and the big APIs like ChatGPT and Claude. For the companies inbetween I'd imageine the risk to reward of picking a stack might be too high right now given how fast everything is moving. There is probably room for someone to do it next year but I don't get the sense that the market is ready for it just yet.
My understanding, and I could be wrong here so I’m happy to be corrected, is that AI companies like OpenAI are perfectly capable of operating an instance of their product that doesn’t send data back to the source.
They discuss this on the Enterprise page:
Protect your data with enterprise-grade privacy, security, and deployment tools
You own and control your business data in ChatGPT Enterprise. We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise. See our privacy page and our Trust Portal (opens in a new window) for more details on how we treat your data.
https://openai.com/index/introducing-chatgpt-enterprise/
So it’s very likely that most companies find this reassuring enough and therefore don’t necessarily care too much about running models locally. Anyone that needs security greater than this probably has the resources to develop AI capabilities in-house.
Have you checked build.nvidia.com by any chance ?
Thanks guys. I agree on market timing, probably next year. I doubt a lot of corps will trust those privacy policies by openAI, especially outside the USA
Capital expenditure, hardware, storage, operations and management.
When I last looked it seemed like cloud costs were much higher.
I think this is a the value prop for AWS Bedrock (no affiliation)
AFAIK You get a managed instance where the model data doesn’t get sent to the model provider, but you pay PAYG rates.
> Why do we see no companies offering to deploy local LLMs?
Because they need your data to sell^Wtrain LLMs.