Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram of architecture) to tapeout, this is an amazing timeline. The truth is probably somewhere in between. A more concrete statement would use actual technical milestones and gates.

The hardware description languages (HDL) used in chip development are like programming languages. The existing models understand them and can do a lot with them. You don’t need to have separate, specialty models designed for this work to use LLMs in chip design workflows.

Design verification also involves a lot of traditional programming which benefits from LLMs.

So it’s not meaningless at all. You could download some of the open source chip design software today and the LLMs could even help you get started on your own tiny chip if you are so interested.

Right. There are two possible meanings and shades in-between:

1) OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)

2) OpenAI designed test/verification models and kernels that could be run on the simulated hardware to test its performance

As you and others have said, it's hard to trust when they are happy to write something that could easily only mean the latter but sounds like the former.

AlphaChip is what a chip design with AI is. I'm very suspicious that OpenAI has anything like this or they would be bragging about it.

https://deepmind.google/blog/how-alphachip-transformed-compu...

There is a lot of verilog out there, it's pretty feasible that they had AI assistance writing more to design their chip.

It doesn't have to be revolutionary, it could just be AI-assisted design and lined up well enough with their operations for a custom ASIC to be worth it.

I would assume they've already made as big a deal of it as they can without outright lying too much. Read the rest of the press release.

FWIW, Google is now on their 8th generation TPU iteration, having put out the last 4 generations on a 1-year cadence.

VHDL, VLSI are well documented languages, with well build test and verification frameworks and harnesses. Even just by iteration you could get there if you have the money to pay for it.

NVIDIA already designs most of their chips using AI. Why would you assume it's meaningless marketing?

realistically, how hard are AI accelerators to design?

I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.

I recently put 2+2 together.

Broadcom has become wealthy by being Google's TPU hardware partner, including sharing their TSMC capacity with Google, and evidently now they are doing the same thing with OpenAI. What a brilliant way to take advantage of the AI gold rush!

I wish they weren't using their piles of money to extort money out of the software industry like they are with VMWare and Bitnami.

> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.

But if you have such a breakthrough could you not also apply it and run 200T models on todays datacenters?

Interesting comment, but the comparison with hard disk drives is probably unfair.

The IBM 350 was commercialized 70 years ago; it took 70 years for someone like you to be able to compare that to a multi-TB SSD.

Furthermore, nothing says that Moore's Law will necessarily apply to LLMs, for decades to come.

> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.

I think there will be specialized hardware (beside GPUs) that would be custom made for LLMs. Yes TPUs exist, but mainly for datacenter. GPUs exist, but they are adapted from mainly graphic application. Once all the demand from data center dries up, innovation will kick in.

True but as someone else pointed out; at that time we'd be interested in running 200T parameter model rather than 200B. Why, you might ask? Law of human laziness - a human will become as lazy as the technology allows it to. With the 200T or 20,000 T model - I'd be heavily incentivized to ask it to make the bread for me that I enjoy making now or create a movie for me (featuring myself) which will maximize the dopamine production in my brain.

I think Jevons Paradox and scaling laws will make this not the case. If bigger models are always better (which seems they are), then will always need high-end hardware.

> I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI

it will build expertise/infra/know-how foundation for next generation of hardware

I agree with you. Stepping stones are still a part of getting there, if only to be briefly useful.

Usually breakthroughs in computing lead to more usage of computing, not less.

Looking at the development of memory bandwidth, capacity and prices over the last 10 years there is little indication that’s likely.

It'd be cool to see more of this type of thing, but I have to imagine the ability for it to be updated to a brand-new model as new models come out is limited. If that is the case, it's going to be an extremely hard sell.

In a chatbot, 17k tok/s is a neat but nearly useless showcase. In a coding agent it is a meaningful improvement. In robotics, it could be an absolute revolution.

8B models aren't useful in general, but for specific use cases they can provide an enourmous amount of intelligence - nVidia's Tesla/Waymo competitor is a 7B LLM with a 2B diffusion model, and running that at those speeds could be an order of magnitude cheaper than existing solutions.

Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.

Inference costs are higher than training now. I think.

Nvidia is king of general purpose training chips. But inferences can be specialized.

> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art

We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims.

I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.

Yeah, the narrative feels like pre-IPO shenanigans, and it looks like the lid on my laundry basket. I wouldn’t be surprised if this is a con.

Who's IPO? Broadcom and Google are already listed, obviously.

Can you suggest some tutorials for Verilog and FPGAs in general?

I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.

Both Google and Amazon also codesign heavily with Broadcomm (Amazon also with Marvell and Alchip)

Broadcomm does stuff like physical design, provides IP blocks, managing manufacturing process with TSMC, packaging and testing. Google and Amazon work with system architecture, performance targets, and requirements but Broadcomm as consultant.

I have been eyeing what Taalas is doing [1] by making pure hardware models. The speed is absurd.

[1] https://taalas.com/products/

What are the other phases. Or what are you referring to in general?

“People who are really serious about software should make their own hardware.” ― Alan Kay

Memory bandwidth is the bottleneck in the Spark. If you replace the SoC with an optimized ASIC but keep the same 256-bit LPDDR5 the performance will be the same. You can increase performance by using wider memory but that's also more expensive.

Unfortunately Sam Altman won't be the one to deliver us at-home hardware that can run Opus-level models

Forget about it. Datacenter class hardware is getting farther and farther from desktop use. It’s not PCIe GPUs anymore.

This is just an uncut wafer - I don't think it's intended to be a wafer-scale chip.

Cerebras etch memory onto the wafer alongside the processing elements, but AFAIK OpenAI are going to be using HBM memory and a conventional chiplet design.

The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.

> 17k tps taalas chip

It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases.

I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?

CA Technologies was much worse than Broadcom in its heyday.

Three of their top execs - CEO, CFO, and head of sales - went to federal prison on securities fraud, conspiracy, and other charges. The CEO, Sanjay Kumar, who was at least partly the fall guy for co-founder Charles Wang, served 10 years.

Being acquired by Broadcom could only have been an upgrade, as strange as that may sound.

Because of Micron, no? I don't think it's related to OpenAI's announcement

If you're referring to the big circle of silicon, that's a wafer, generally contains many chips (100-1000s).

I think Cerebras stock going down could also be partly caused by the lock-up period ending today for 200k shares (page 73 of their prospectus) - https://www.sec.gov/Archives/edgar/data/2021728/000162828026...

That's just the wafer disc. Looks like it was presented to Sam Altman for ceremonial purposes.

The wafer disc is what the CPU gets "printed" on.

Dang, I just checked and CBRS is in free-fall since the IPO.

Sucks, I think they're a cool company.

OTOH, I was the only person back then pushing hard during my time at KAUST (back in 2019) to buy one of their systems when they were nobody, eventually resulting in a partnership between the two.

Then I joined their online discourse, very few users, I was semi-active there but they didn't care much.

Then I came to Toronto and heard they were opening an office here, tried to get noticed several times but got mostly ignored. I asked about upcoming events several times, anything to get involved, "yeah man, maybe one day". Then they made an event during Toronto Tech Week and didn't even tell me ... idk.

I don't get schadenfreude as I still think they're a cool company.

My point is they put all the eggs in one basket (AI inference) and neglected everything else. They seem to be on shaky ground now ... sad.

They don't have true competition, what they lose out on is market share with hyperscalers, since OpenAI would have no plans to share inference hardware with any other company right now. Plus, I don't know how does NVIDIA's investment equation pans out long terms given OpenAI will be investing in more purpose built inference stack for the future.

they're still kings for training, though I've heard Anthropic is training now on JAX+TPU setup, so might not be a monopoly in that segment.

I assume OpenAI has been buying memory and "giving" it to Nvidia in exchange for a discount.

that's not really how it works

You don't need a whole lot of software support if you just want to serve a single family of LLMs.

It's probably the same speed but cheaper.

That’s exactly what this is.

9 months to production is completely impossible anyway.

9 months from design to early samples is probably impossible given than TSMC takes 3 months after tape out to produce them. Then it’s up to the customer to qualify and revise for production. TSMC doesn’t do that.

There’s no AI that makes this happen in 9 months.

No worse, I suppose, than, the obsession with Lord of the Rings that the authoritarian surveillance companies have. Palantir, Anduril. Then we have the not defense/surveillance ones: Mithril, Valar, Narya, Erebor

Strawberry was too complicated as a codename.

Don't worry, in Europe it's the same, but for insurances/lawyer stuff. Tons of companies have names based on Latin words such as Civitas/Insalus/Legalia/Legalitas or whatever which looks tacky/rancid/old fashioned kilometers away.

Jalapeño

Jalapeño

Jalapeño

Really has a… ring to it