What I don't quite get is why can't they use AI to translate CUDA programs into more open architectures like AMD ROCm?

AI is supposed have solved the "coding problem". But shouldn't translating a program from one platform to another be an even easier, more mechanical, task for the AI?

I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem.

AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.

Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.

To paraphrase the apocryphal Winston Churchill quote about democracy:

“CUDA is the worst development ecosystem in existence. Except for all the others.”

Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars.

Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.

Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.

It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI.

The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.

It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.

There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.

I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.

One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.

[1] https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/

CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you.

Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.

Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.

Management likes it because it removes software developers from the loop.

[dead]

My Pixel 9 has a personal tpu

[dead]

It seems like path dependent lock in to me and a risk/reward calculation.

What do you gain by not using CUDA vs what do you risk?

Indeed. Now, given that CUDA is apparently not being usurped, update your priors.

well done you realised no one knows what they are talking about

LLMs make the dev environment almost irrelevant. llama.cpp supports AMD with both ROCm and Vulkan and that's nearly all that matters now. TBD how much AMD's AI Halo play will change things if at all, but they got a lot of positive press in launch reviews for having an actually robust software story for once.

Hardware companies are notoriously bad at software. The software they use sucks, the languages they use suck, the internal tooling software they write sucks.

They don't know what good developer experience is, how do you expect them to deliver it to other people?

... with performance greater or comparable to even the lowest performance Nvidia card?

Google sells TPUs already, to competitors :)

Coral and the TPUs are ASICs, and therefore are barely reprogrammable. It doesn't really compare to the complexity and flexibility of CUDA ALUs.

I think efficiency is unlikely to result in lower demand for compute, instead more useful compute per watt increases the value of that compute; and we are not going to run out of economically useful things to do with it anytime soon on the demand side.

The harder thing to forecast for me is if we hit a wall on increasing efficiency, either on the model weights side or silicon side, with current approaches. If we have to switch to something like burning the model weights into silicon to continue to make gains, then the current math on general purpose accelerators might be upside down.

It's also hard to predict how much money will be burned going down wrong avenues. The internet was the future, but it took a lot of failed companies to eventually land on a sustainable model that brought us the giants we have today.

Railways were also the future, but that didn't stop a rush to build out (often subsidized) lines that were ultimately uneconomical (either because they were corrupt or the planned settlements never arrived).

If AI is similar, then there's going to be a long slowdown on compute spend until the surplus is worked through. A good historical analogy could be the fiber optic buildouts of the late 1990s. The demand for data never really went down much, but the industry eventually commodified and took down some large companies (Nortel, especially)

  It is entirely possible that 5 years from now, there's >100x LLM inference going on - but demand for AI chips (including memory) is only 2x or less.

Jevons paradox should win out for a long time for AI.

When internet connections got faster than 56k modems, we didn't use the same amount of bandwidth but faster. We used more bandwidth doing things like 4k streaming. I see the same in AI inference. If AI inference is that much more efficient, it will just enable more use cases for AI.

See for example, internet traffic over time: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS779VS...

Even after so many years, internet traffic continues to grow at an increasing rate.

Nvidia's great superpower is flexibility. You can easily run models of very different types on the same card; and their hardware is great for R&D.

However, at some point AI may be good enough for most people and then it makes sense to make an ASIC for the model (or group of models); and at that point you don't need Nvidia.

I suppose this scenario will happen in various moments at different levels.

Not really. The GPU+LPU combination is going to be pretty good once it comes out.

amd is not putting nearly enough effort to improve their software its almost suspicious

nVidia will fail right after reaching an 8 trillion valuation and supplying 80% of the world's hardware!

Trust me guys, it's over!

An AGI doesn't need that level of detail to do most tasks effectively. To use the OP's example, a cat catching a bug out of the air does not need to run a fluid dynamics simulation of airflow over the bug's wings to be able to catch it. A cheap approximation of the flight path is sufficient. Perhaps some physical tasks will need that level of detail but many will not.

Is the human brain capable of doing real simulations of light and physics in realtime? Or does it hallucinate the details and represent some low-resolution mental image? You may be overestimating the capabilities of flesh-based neural networks and underestimating the capabilities of silicon-based neural networks.

No need. The 'apocalypse' will come purely out of the hubris and greed chasing untold fortunes as opposed to a healthy, considered integration.

The decision to scare the shit out of normal people early on was a short-term decision that will cost everyone dearly (it led non-technical people to believe in things about LLMs that just aren't true—even worse, they've now made difficult business decisions on bad information with seemingly limited consideration of secondary effects).

The amount of short-sightedness alone should give pause, but at least in America, we seem to have collectively lost our minds in worship of the almighty dollar.

Judgment day approaches.

> They rightfully invested their insane profits into many ventures

This is the inherent downside of such a rapid rise to being the world's most valuable company AND still being considered a growth stock. At their massive scale, the number of new adjacent businesses that have both sufficient size and potential growth is limited.

> I hope they succeed in chewing even 50% of what they bit off.

Anything approaching that is vanishingly unlikely. They're being forced to play the game more like a VC. The question is if a few unicorn winners can offset dozens of losers. The challenge is that, unlike a VC, their bets are much more correlated around AI.

Unified memory DGX Spark and RTX Spark laptops are already a thing :)

Yes. The more capable SoC PCs become, the weaker the "everything has to run in the datacenter" argument gets. As more powerful SoC PCs from multiple vendors appear over the next few years, that advantage may shrink for many workloads.

I'd be happy enough if Nvidia decided to launch an update to the now dated (2019) Shield streaming box, which is still the best on the market as to codec support and image quality.

RTX Spark already launched.

that undercuts their core business, so it will be a defensive play at most to fend off mac and amd's local inference offerings

> What's the danger? They slide back down to being just a gaming graphics card company with a $10 share price?

Nvidia dropping from being a $5 trillion company to a $242 billion company would be 1929 levels of bad.

Global economy end of days stuff, especially since Nvidia can't crash that hard without a lot of other stuff crashing with it.

I've always marveled at how one can pick any year since the internet went mainstream and in that year people thought, "Oh my goodness, the internet is amazing!"

Then, move five years forward from that year and look back. In every case, people think, "Hah! The internet was so simple then!"

In 2031, I suspect we'll say the same about 2026.

Yes and no...

So much of the dotcom bust was essentially: "anything internet will work", but even after it was pretty easy to find different ways to publish web pages or migrate to new forms of social media, video, short-video, etc.

It's orders of magnitude harder to automate work, which is the business value proposition of AI (whether you are displacing worker or adding breakthrough capacity), and it's not clear the LLM hyperscalers will get the value-add from that.

On the consumer side, it might be a race to the bottom: same ad profits, but now you need AI to produce it.

There’s a massive difference between “going up forever” and “point B is higher than point A.”

The current setup can’t sustain a downturn, even if yes 20 years from now point B is likely to be higher than present.

That’s the danger. Those that are going to get wiped out by the AI bubble burst aren’t wrong about AI being huge long term, they just put themselves in a position to not survive the storms that happen between points A and B.

I guess we'll find out soon enough.

Or at least our public pension systems.

I did not mean to be snarky. But still, the proposition that a railroad bankruptcy “led to world war” (direct quote from the article) 40 years later is ridiculous.

Re: "This looks to have gotten some dominoes rolling that may have contributed to WW1 " - people who believe this should read some history about beginning of WW1. I recommend starting with "The Guns of August".

I am assuming the rest of article is as well researched as the, rather long and incredibly incorrect, introduction.

Bayesian inference and all....

well its typically when it becomes clear that the private equity firms taking the risk decide they can not get the returns they need, forcing the backstoppers such as Nvidia to take that burden, and the whole ecosystem collapses.

I know him, I subscribed for a while. Even his paid content is lacking. I'm looking more Valens Research kind of analysis. SemiAnalysis is also good in the higher tier.

ps. Being influential in Silicon Valley just means you are influential, it does not mean substantial. Leopold is still influential and gets money thrown at him at $100s of million despite having no substance.

To me they come across as overzealous peddlers and hacks on some market pumping mission.