I don't know.... With all those approaches to run AI in your own premise, people need their own hardware. It makes them independent. Of course hyperscalers and AI labs don't like that, but I argue that consumers prefer independence over some costs of operations. If this would be the case, people wouldn't own cars.

server -> personal -> server -> personal... I've seen this cycle a number of times in the last 40 years. Almost none of the people making any of the software we use are purely using the cloud, they are using powerful personal computers with cloud for GPU. The ubiquity of personal computing is required if we want this industry to keep producing enthusiastic hackers.

If the AI companies ever get the idea that it would make sense for the AI model to modify its KV cache, it would end up adding a write step to the KV cache per token (moderately expensive) and increase the cost of inference by the width of the update (arbitrarily expensive).

This in itself could double or triple the demand for both memory bandwidth and compute. I honestly don't see a future where AI demand disappears...