kkielhofner
a year ago
The model you linked is not an LLM either by architecture or size.
A few thoughts:
1) TensorRT anything isn’t an option because it requires Nvidia GPUs.
2) The serving frameworks you linked likely don’t support the architecture of this model, and even if they did they have varying levels of support for CPU.
3) I’m not terribly familiar with Hetzner but those instance types seem very low-end.
The model you linked has already been converted to ONNX. Your best bet (probably) is to take the ONNX model and load it in Triton Inference Server. Of course Triton is focused on Nvidia/CUDA but if it doesn’t find an Nvidia GPU it will load the model(s) to CPU. You can then do some performance testing in terms of requests/s but prepare to not be impressed…
Then you could look at (probably) int8 quantization of the model via the variety of available approaches (ONNX itself, Intel Neural Compressor, etc). With Triton specifically you should also look at Openvino CPU execution accelerator support. You will need to see if any of these dramatically impact the quality of the model.
Overall I think “good, fast, cheap: pick two” definitely applies here and even implementing what I’ve described is a fairly significant amount of development effort.
backend-dev-33
a year ago
Well, looking at Triton Inference Server + OpenVINO backend [1]...uff... as you said: "significant amount of development effort". Not easy to handle when you do it first time.
Is ONMX runtime + OpenVINO [2] a good idea ? Seems easier to install and to use: Pre-built Docker image and Python package... Not sure about performance (the hardware-related performance improvements - they are in OpenVINO anyway, right?).
[1] https://github.com/triton-inference-server/openvino_backend
[2] https://onnxruntime.ai/docs/execution-providers/OpenVINO-Exe...
kkielhofner
a year ago
Hah, it actually gets worse. What I was describing was the Triton ONNX backend with the OpenVINO execution accelerator[0] (not the OpenVINO backend itself). Clear as mud, right?
Your issue here is model performance with the additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner.
Triton does things like dynamic batching[1] where throughput is increased significantly by aggregating disparate requests into one pass through the GPU.
A docker container for torch, ONNX, OpenVINO, etc isn't even natively going to offer a network socket. This is where people try to do things like rolling their own FastAPI API implementation (or something) only to discover it completely falls apart at any kind of load. That's development effort as well but it's a waste of time.
[0] - https://github.com/triton-inference-server/onnxruntime_backe...
[1] - https://docs.nvidia.com/deeplearning/triton-inference-server...
backend-dev-33
a year ago
> additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner.
@kkielhofner thanks a lot! now I realize it. I see, there is even GRPC support in Triton, so it make sense.
kkielhofner
a year ago
Make sure to check out the existing Triton client libraries: