A large fraction of the innovation in CPUs is driven by working around the memory wall. I anticipate AI inference will follow the same trend, and innovations that work around the autoregressive nature will be enormously impactful.

Speculative decoding is an example. An accurate draft model can reduce the number of times you stream through memory by a factor of 4x.

One of the biggest limitations right now is memory capacity (storing large models/contexts in memory) and bandwidth (transferring the relevant data/weights to the silicon that is performing the operations on that data). This would cover things like:

1. having more memory on the card/chip and/or faster access to that memory;

2. integrated memory and compute units optimized for matrix and vector multiply add operations;

3. optimized load circuitry to e.g. read memory in the stride and span (next row, next column) access patterns common to matrices or ensure that no/few parts of the chip are stalled waiting on data or operations to complete.

Another aspect is quantizations. These are similar to SIMD vector operations in that you are performing an operation on a block of n-bit data values at the same time, so can have optimized circuitry.

For 2 or 3 valued quantizations you can reduce various addition and multiplication operations to logic operations, avoiding circuitry for things like the half-adder, full-adder, and carry-lookahead.

Then there's adding specific circuitry for common operations such as ReLU like is done in hardware acceleration of image, video, etc. processing. There's a trade off here as optimized hardware would perform better at the specific operations but if those are too specific then they can't be used by different/newer model architectures. (Though it does make sense to try and optimize common operations/logic where possible.)

It would be interesting to see if these designs can/will benefit training as well, as that would bring down the time/cost/energy of training large models as well as making it easier for local fine-tuning.

Some kind of compute-in-memory architecture is a good candidate, I think. There are many alternatives here, researched for many years prior to the LLM craze. However economies of scale dominate in chip industries, and this tends to favor more conventional or incremental approaches (to piggyback on existing scale). Alternatively someone needs to have a way of bootstrapping the insane scales needed to be competitive with a better-but-different approach. So it could be that boring and straightforward stuff like two-chip prefill+decode takes most.

> The list of individual innovations that led to today’s ubiquitous, powerful personal compute could fill dozens of books.

https://a.co/d/0dMk8urP

which is the Hennesey and Patterson computer architecture book would serve the role of the "dozens of books" hyperbole rather well.

Exactly. If we ever get models that stabilize in terms of capabilities (and arent leapfrogging each other month after month) you have a "cartridge" like thing that just plugs in with the latest model in a read only ROM thats super fast.

As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there

The analogy is flawed - in Scrabble you get rewarded for unlikely word combos. Also causal attention attends to prior tokens (tiles already placed).

hang on, isn't this the standard way to implement IEEE 754 floating-point multiply since forever?

normalize the two numbers A and B to have the same exponent, add the mantissa, then convert back to IEEE 754?

but aren't additions in exponents expensive? as in each operation is an fma

Big money from taking Tesla's place in line for an Nvidia shipment during a huge shortage without compensation.