The implementation of a language model is usually good enough for running _a single forward pass_ through the model, but to host it via an inference engine you typically need to convert a few operations. For instance the MLP can be easily split across GPUs (tensor parallelism) and MoEs also have a way of parallelizing.

Most of it is pretty standard, since not that many different layers and primitives are used in LLM architectures, but once in a while something new comes along that needs more effort. MoEs are one example, they are sparse and allow for completely different inference patterns, which takes a while to figure out.

Last year I started a blogpost series about this topic (that I hope to update some time). I start from a minimal gpt implementation by Karpathy and build the engine around it, you might like it: https://pieter.ai/blog/2025/nanogpt-inference/

> Somehow I expected inference engines are generic LLM runtimes that can execute any weight.

In fact, this was close to be true until last year: almost every open model except DeepSeek had a very similar architecture that was pretty close to the GPT-2 one with very few variations on top (and sometimes an MoE architecture, which itself was a few year old at that point).

But a year ago there's been a cambrian explosion, first in attention mechanism but also in a bunch of other directions, mostly coming from China, and now there's a very massive diversity today's space.

Makes me feel better as a guy who tries to get O(1000) times smaller models working for scientific applications. Writing a backend that works for all the models people train is quite a chore, so the common refrain is "why don't we just do what industry does?" (he answer is that "industry" has billions of dollars). You also get "no one uses X backend any more" to which the reply is also "yes, but they have billions of dollars and a team of software engineers".

Go look at the github commits for llama.cpp and read the actual code they're adding to support new model/quant types... to me at least it's some serious black magic, and the sheer number of genius developers and activity level in this repo is absolutely wild.

Imagine what could happen if other open source projects had this level of engagement and expertise at hand and eager to contribute... instead of arguing over politics and making yet another fork of something.

Yes. https://inferencex.semianalysis.com provides some comparisons wrt. certain cost metrics. Some commonly used ones are vLLM, TensorRT-LLM, and SGLang. These three at least are open source, and all come with an Apache 2.0 license.

Some providers also have to implement their own engines, e.g., Cerebras has their own inference serving stack for their wafer-scale chips, as does Google for their TPUs (XLA compiler).

FYI Mistral at launch just dropped the weights without any model architecture mentioned.

Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work.

Yes. You can see this affecting perf benchmarks as well. Usually the cheapest inference providers either use approximations like tanh instead of sigmoid, nvfp4 quantizarion, etc.

There was a post here the other day highlighting this by showing the benchmark perf of different I defence providers, it's a fantastic area to cheap out in, because you can never really tell if a model is 75% good or 83% good on some specific benchmark when you use it to build your own stuff

Absolutely. The worst part is that ML being probabilistic means the models still output something comprehensible meaning you can't determine if it's correct or not without a lot of scrutiny and engineering.

Not the poster, but maybe I can help.

Your comment is a little unclear, so it‘s hard to parse your exact question. But it seems you are conflating 3 things, PyTorch, RL and Gym/Training (?).

– PyTorch is a framework which lets you define neural network models.

– RL is a collection of methods to train neural networks (change the network parameters to improve its performance).

– An RL-Gym is a framework to apply the neural networks to some problem. This lets you collect the data necessary to later use the methods of RL to train your model.

thanks for pointing out. its a bug

fixed

the only way i am aware of is to cross check with what deployed on huggingface/transformers repo

i will see what i can add here. contributions are welcome

I think depending on your perspective it might be even better to eschew PyTorch and implement with pure NumPy. Sure it will be slow, but perhaps as a pedagogical exercise it is more worthwhile.

will try my best, but open for contribution.