Ask HN: Is it time to run the LLM engines on the CPU?

3 pointsposted 10 hours ago
by roschdal

Item id: 49402551

5 Comments

dtagames

10 hours ago

You always could. It's just slower and less efficient because most of what a model query needs to do is matrix multiplication and the GPU is optimized for that.

bigyabai

10 hours ago

It depends, how much coal are you comfortable burning?

roschdal

10 hours ago

In the future we could have CPUs which are better suited for llms, with GPU-like properties built in.

bigyabai

6 hours ago

You're reinventing the CISC wheel that Apple, Google and Nvidia all abandoned.

x86 already has AVX accelerators that are a huge help for LLM inference speed and efficiency. It's just not enough, only the absolute weakest iGPUs are slower than AVX inference. The majority of consumer hardware would benefit from pipelining AI workloads through the GPU, especially on Apple Silicon and other ARM hardware that has borderline nonexistent SIMD support.

ARM will be ideologically opposed to expanding CPU-side SIMD, and x86 is already buckling under the weight of modern AVX (see: AVX-512 issues).