Doesn't multiply area scale at O(n^2 * log(n)) ?? (At least, I'm pretty sure the Wallace Tree Multiplier circuit is somewhere in that order).

So a 64-bit multiplier is something like 32x more area than a 16-bit multiplier.

But what you say is correct for RAM area or the number of bits you need for register space. So taken holistically, it's difficult to say...

Okay, 64-bit FP is only like 53-bits and 16-bit FP is actually like 11 bits. But you know what I mean. I'm still doing quick napkin math here, nothing formal.

-------

We can ignore adders and subtractor circuits because they are so small. Division is often implemented as reciprocal followed by multiplication circuits for floating point (true division is very expensive).

With the B100 somehow announced to have lower scalar FP64 throughput than the H100 (did they remove the DP tensor cores ?), one will have to rely on Ozaki schemes (dgemm with int8 tensor cores) and lots of the recent body of work on mixed-precision linear algebra show there's a lot of computing power to be harnessed from Tensor Cores. One of the problems of HPC now is a level of ossification of some codebases (or the lack of availability of porting/coding/optimizing people). You shouldn't have to rewrite everything every 5 years but the hardware constructors go where they go and we still haven't found the right level of abstraction to avoid big porting efforts.

You've heard of SIMD - it's possible to do both, in terms of throughput, with instruction/scheduler/port complexity overhead of course.

Yes, that's a great point that I missed. From anecdotal evidence, it seems more people are using supercomputers for ML use cases, that would have been traditionally done by HPC. (eg training models for weather forecasts)

If I understand your comment correctly, we're taking a stable but not that relevant metric, because the real players of the market are too secretive, fast and far ahead to allow for simple comparisons.

From a distance, it kinda sounds like listening to kids brag about their allowance while the adults don't want to talk about their salary, and try to draw wider conclusions from there.

Top line comparison numbers for reference: https://www.theregister.com/2024/03/18/nvidia_turns_up_the_a...

It does seem like Nvidia is prioritizing int8 / fp8 performance over FP64, which given the current state of the ML marketplace is a great idea.

> soup of nearly independent machines

that does a serious disservice to hyperscaler clusters.

i wish people wouldn't make stuff up just to sound cool.

like do you have actual experience with gov/edu HPC? i doubt it because you couldn't be more wrong - lab HPC clusters are just very very poorly (relative to FAANG) strewn together nodes. there is absolutely no sense in which they are "one single machine" (nothing is "abstracted over" except NFS).

what you're saying is trivially false because no one ever requests all the machines at once (except when they're running linpack to produce top500 numbers). the rest of the time the workflow is exactly like in any industrial cluster: request some machines (through slurm), get those machines, run your job (hopefully you distributed the job across the nodes correctly), release those machines. if i still had my account i could tell you literally how many different jobs are running right now on polaris.

Put slurm on it, bam. Supercomputer.

Nvidia GPUs are also used for inference on Google products. It just depends on availability.

Not true. Apple trained some models on their TPU

Huh? https://cloud.google.com/tpu/docs/intro-to-tpu

Hey now I compare Apples to Raspberry Pi's regularly :)

Source?

That's a good point.

That said, I assumed the context of the article was specifically on the topic of AMD GPUs, and not, say, Epyc processors, so if so, it's ultimately irrelevant.

(Also, there are just over 3 exaflops across non-accelerated supers; AMD would need > 2/3 of the remaining share in order to surpass Nvidia on that front as well.)

Yes, I saw that, but that doesn't justify the title as written. Had it said "AMD Now Has More New Compute" I wouldn't have said anything.

I think you nailed it on the head.

Companies like CoreWeave have deployed so many giant clusters (and growing), it is insane. Their IDLE compute is larger than most of the supercompuers out there.

Of course, they aren't on the list either.

The API part isn't thaaat hard. Indeed HIP already works pretty well at getting existing CUDA code to work unmodified on AMD HW. The bigger challenge is that the AMD and Nvidia architectures are so different that the optimization choices for what the kernels would look like are more different between Nvidia and AMD than they would be between Intel and AMD in CPU land even including SIMD.

> P.s. how much is an exaflop or petaflop, and how significant is it? The numbers thrown around in this article don't mean anything to me. Is this new cluster way more powerful than the last top?

Nominally, a measurement in "flops" is how many (typically 32-bit) FLoating-point Operations Per Second the hardware is capable of performing, so it's an approximate measure of total available computing power.

A high-end consumer-grade CPU can achieve on the order of a few hundred gigaflops (let's say 250, just for a nice round number). https://boinc.bakerlab.org/rosetta/cpu_list.php

A petaflop is therefore about four thousand of those; multiply by another thousand to get an exaflop.

For another point of comparison, a high-end GPU might be on the order of 40-80 teraflops. https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.ht...

Anybody spending tens of billions annually on Nvidia hardware is going to be willing to spend millions to port their software away from CUDA.

CUDA is the assembly to Torch's high-level language; for most, it's a very good intermediary, but an intermediary nonetheless, as it is between the actual code they are interested in, and the hardware that runs it.

Most customers care about cost-effectiveness more than best-in-class raw-performance, a fact that AMD has ruthlessly exploited over the past 8 years. It helps that AMD products are occasionally both.

Software will bridge the gap. There are simply too many competing platforms out there that are not Nvidia based. Most decent AI libraries and frameworks already need to support more than just Nvidia. There's a reason macs are popular with AI researchers: many of these platforms support Apple's chips already and they perform pretty well. Anything that doesn't support those chips, is a problem waiting to be fixed with plenty of people working on fixing that. If it can be fixed for Apple's chips, it can also be fixed for other people's chips.

And of course there is some serious amount of money sloshing around in this space. Things being hard doesn't mean it's impossible. And there's no shortage of extremely well funded companies working on this stuff. All your favorite trillion $ companies basically. And most of them have their own AI chips too. And probably some reservations about perpetually handing a lot of their cash to Nvidia.

If you want an example of a company that used to have a gigantic moat that is now dealing with a lot of competition, look at Intel. X86 used to be that moat. And that's looking pretty weak lately. One reason that AMD is in the news a lot lately is that they are growing at Intel's expense. Nvidia might be their next target.

A high grade consumer gpu a (a 4090) is about 80 teraflops. So rounding up to 100, an exaflop is about 10,000 consumer grade cards worth of compute, and a petaflop is about 10.

Which doesn’t help with understanding how much more impressive these are than the last clusters, but does to me at least put the amount of compute these clusters have into focus.

According to Wikipedia the previous #1 was from 2022 with a peak petaflops of 2,055. This system is rated at 2,746. So about 33% faster than the old #1.

Also, of the top 10, AMD has 5 systems.

https://en.wikipedia.org/wiki/TOP500

> P.s. how much is an exaflop or petaflop

1 petaflop = 10^15 flops = 1,000,000,000,000,000 flops.

1 exaflop = 10^18 flops = 1,000,000,000,000,000,000 flops.

Note that these are simply powers of 10, not powers of 2, which are used for storage for example.

People have been chipping away at this for a while. HIP allows source-level translation, and libraries like Jax provide a HIP version.

There is ZLUDA to break the lock-in for those who are stuck with it. The rest will use something else.

Isn't porting software to the next generation supercomputer pretty standard for HPC?

its possible. Just look at Apples GPU, its mostly supported by torch, what's left are mostly edge-cases. Apple should make a datacenter GPU :D that would be insanely funny. It's actually somewhat well positioned as, due to the MacBooks, the support is already there. I assume here that most things translate to linux, as I don't think you can sell MacOS in the cloud :D

I know a lot developing on apples silicon and just pushing it to clusters for bigger runs. So why not run it on an apple GPU there?

> But how will AMD or anyone else push in? CUDA is actually a whole virtualization layer on top of the hardware and isn't easily replicable, Nvidia has been at it for 17 years.

NVidia currently has 80-90% gross margins on their LLM GPUs, that’s all the incentive another company needs to invest money into a CUDA alternative.

Maybe the DOJ will come in and call it anti-trust shenanigans.

Not that I would want this...

People most certainly do use CUDA

> It really is that hard

YES!! Thank you!

> Nvidia's real victory (besides leading on performance-per-watt) is that their software stack doesn't suck

YES! And it's not just CUDA and CUDA-adjacent tools, but also their cuDNN/cuBLAS/etc. libraries. They invest a massive amount of staffing into squeezingt the last drop of performance out of their hardware, identifying areas for improvement and feeding that back to the architects.

> Apple wrote the OpenCL spec for exactly that purpose and gave it to Khronos

Nitpick: Affie Munshi from Apple wrote down a draft and convinced his management to offer it to Khronos, where it was significantly modified over... was it a year or so?... by a number of representatives from a dozen companies or so. A ton of smart people contributed a ton of work into what became the 1.0 version.

And let me tell you that the discussions were often tense, both during the official meetings as well as what happened behind the scenes. The end result was as good as you can expect from a large committee composed of representatives from competing companies.

But, in summary, you get it, unlike so many commenters in HN.

The industry, meaning Google decided to go with Renderscript C99 dialect for Android, while Intel and AMD never delivered anything that could match CUDA ecosystem (note the ecosystem part), Khronos never understanding the value of C++ and Fortran in HPC, they still don't in regards to Fortran.

Intel actually has proven to be more clever than AMD in that regard, as DataParalell C++ builds on top of SYCL (it isn't only SYCL), and Intel Fortran now also does GPU offloading.

> you just pay TSMC a shitload of money

I guess with money you can win any argument ...