Asking out of curiosity because I always wanted but never needed such speeds, could you elaborate on what you do and how does this speed help you? And I presume there’s a reason you’re not able to connect directly to the drives etc. with thunderbolt.

>I'm using the sonnet at work.

I initially read this as SONET[1] and was seriously questioning your technical choices.

[1] https://en.wikipedia.org/wiki/Synchronous_optical_networking

Thats what I do.

ConnectX-4 and newer are natively supported. I use a ConnectX-4 LX in a $50 Razer CoreX off of craigslist. Works fine, hit similar limits to what Jeff did.

Using an OCP is the neat trick here. MUCH smaller setup.

As far as the other posters comment about Tx's... that is an entirely different topic. I've never had an issue with FS.com optics personally.

Absolutely not. It's another layer of throughput/compatability bingo on top of the already often quite annoying SFP bingo.

If you are unfamiliar, the reality of the situation is quite bad. Slight hardware revisions of adapters/switches, which if you are lucky can be differentiated by manufacture date or serial number sequence, will not recognize known good modules, and if they do you then have to actually test if they can establish link, then let them burn for 24+ hours to see if they can maintain it without dropping (often, because of forward error correction, you will not start to see flapping until enough correctable errors have accumulated and the buffer overflows). Then there are firmware updates where this process must then be repeated.

You would think with how expensive all of this shit is you would simply be able to buy the officially listed as compatible components but my experience with them has been that a shotgun approach is best for your sanity and deadlines, buying sample kits of each module and trying them all, and then once you find the ones that play nice, buying as many of them as is reasonable and testing all of them.

If I had a dollar every time two modules from the same brand, that came in the same box, from the same production run, did not work with one another, I would be able to test your theory for free. Network Engineering is Fun

This is a ConnextX4 on a thunderbolt bridge. This is well trodden territory. Unless some hero of open source has come to bat recently, MacOS doesn't support RDMA with this card or over Ethernet at all for that matter. It's only supported over Thunderbolt.

His iperf3 results show that it's able to go full beans on a single 25g port. A cross platform SMB file copy is kind of a meaningless benchmark in this context.

IMO a pretty cool product would be a network switch providing native thunderbolt interfaces to multiple upstream hosts. That wouldn't magically fix this RDMA issue, though. This is just a separate thought. Seems like the kind of thing Mikrotik should make.

I have a 25GbE setup using Thunderbolt, kinda similar to the author's. Except that my fan is held together by a piece of string, not a 3D-printed enclosure.

I found that SMB on macOS is just bad. If I try to heavily load it (e.g. by doing mass conversions in my Audible library), it can cause stalls and disconnections.

I managed to find a set of parameters (in /etc/nsmbd.conf) that makes it work more-or-less reliably by trial-and-error: dir_cache_off=yes, file_ids_off=yes, notify_off=yes, mc_on=no.

It's still terrible, though. It's so bad that Linux in a Parallels VM works _better_.

https://developer.apple.com/documentation/technotes/tn3205-l...

MacOS has supported RDMA over thunderbolt since March 19th.

As I understand it, it takes a bit of anger to get electromagnetism to wiggle that fast over copper. Imperfect transistors using their full operational envelope to control a sort of chaotic thing? Fields and stuff.

They’re not that bad, but even a couple watts of power with no way to dissipate is going to get hot enough to burn you. Forced air cooling over a heatsink is really effective. 15W under load is much lower than a high end desktop CPU that can pull 150-200W.

See also NVMe drives? They draw enough idle power to drain a laptop battery overnight, and will easily hit 60+C in a bad enclosure. That’s only a 10Gbps module. The USB4 enclosures are all heatsink.

It's distance. 25GbaseT is good for 30m+. Thunderbolt is good for 1m without active cables. Making the much more rugged signal is a much more intense job.

And your computer already speaks the PCIe and thunderbolt. It has that probably on CPU. Here we have to receive USB, convert it to Ethernet. On much less up to date process nodes than your CPU.

Third, until very recently for even 10Gbit it meant using a usb4<->pcie bridge chip which itself was hot, to provide a generic pcie connection, which then talked to a conventional network chip, so it was a double hop: USB <-> pcie <-> nic. I don't know if 25Gbit to USB direct convert is available yet but we've seen a wave of vastly cheaper 10Gbit adapters come out that are single chip USB<->nic, probably not pcie at all, that are much much cooler. 25Gbit is probably still not here yet, maybe?

Ethernet is isolated through transformers on the physical layer and they're always sinking power. Thunderbolt is just pushing current directly through.

With ethernet it's for lots of reasons but a big one is not creating ground loops between distant equipment. If you didn't isolate the network cable would be sinking potentially amps of current between distant devices and that just isn't sustainable. You don't have this problem attaching USB between two devices.

A Mac Studio is over 100 W while a 25G NIC is under 25 W (that's the power limit for a normal PCIe slot).

Maybe the NIC does a lot? In past experience, a whole Xeon CPU couldn't keep up with a 10gig NIC if it was sending/receiving min-size UDP packets. Even with Intel DPDK iirc.

> It’s crazy that the Ethernet chips consume more power (presumably - because of the heat)

The NIC in this setup was released 12 years ago. It's built on a very old process.

There are no shortcuts in networking. It's quite easy to spend a 2-3x+ premium on a switch or adapter capable of non blocking line rate on every port simultaneously only to then have it fall over to a fraction of that in the form of lots of tiny packets. This is why we have things like QoS, flow control, and ECMP

is 10W a lot of power? I absolutely don't believe the cpu is using less than that while pushing 25Gb out

you can look up the rated power use of a connectx4 card, they do a lot of offloaded computation, too

Something like 5 percent of AI datacenter usage is just optics.

You're an order of magnitude off there. The 10 Gigabit connection was providing ~ 1 Gigabyte per second of throughput.

$400 is for the empty enclosure and $1,000 is for the enclosure with a NIC in it. Obviously it's still a ripoff because a 25G NIC does not cost $600.

Yup, totally doable. Of course, it’s a point to point link so if you want to build a whole cluster that way then you’ll need a lot of usb-c ports or you’ll need to make more of a mesh topology.

I believe MacOS supports it via the usual network control panel. I’ve used it for transferring data from an older laptop to a newer one. The cable is more expensive than fiber, last time I checked.

I used to have a thunderbolt ring network for a ceph backend on my k8s cluster. On Linux load the thunderbolt_net kernel module and they show up the same as any other Ethernet interfaces. Just directly connect them and configure the network settings on each end using your normal tools.

Latency was an order of magnitude higher than twisted pair but the throughput was greater than 10Gbps and I didn’t have to spend on 10GbE capable servers and network equipment. I vaguely remember that they had to be proper Thunderbolt not just USB-C but that could have changed it’s been quite a while.

Yes, there are a few standard(ish) mechanisms for doing this, like [RNDIS][1] and NCM. If you have two Thunderbolt-equipped Macs and connect their ports together, you can use Bonjour/zeroconf for network discovery between them.

[1]: https://en.wikipedia.org/wiki/RNDIS

With USB 4, very strictly speaking: It's iffy. USB 4 doesn't require hosts to be able to switch roles and talk to other hosts. It can support this, but support is optional.

But for all certified Thunderbolt 4 and 5 ports, this kind of networking support is a requirement. (It is optional with Thunderbolt 3.)

The rest is up to the operating system.

USB-4 is basically Thunderbolt right? Thunderbolt networking is pretty automatic on Macs at least: https://support.apple.com/en-gb/guide/mac-help/mchld53dd2f5/...

You're limited in your network topology because there is no USB Switch. This HN post used two USB4 connections to create a 3-node cluster: https://news.ycombinator.com/item?id=39003469

Your so-called appropriate USB C cable is just two USB-C to Ethernet adapters plus an Ethernet cable. It’s very common on laptops without native Ethernet ports. But in my experience USB isn’t a very good protocol; it generally uses way more CPU than plain PCIe Ethernet. This is especially the case if the chipset is using ECM rather than NCM.

I have the SABRENT USB-C to 2.5Gbps adapter as well (also bought late 2024) and can echo this sentiment. I had it connected to a M3 Pro MacBook Pro and could never get anywhere close to 2.5Gbps from it. I never found other complaints at the time either and was starting to blame poor wiring in my walls until I replaced it with Ubiquiti's 5GbE adapter (which I believe uses a Marvell controller) and that has been rock solid for me.

YMMV - I have three RTL8156 2.5G dongles - one in my Synology NAS, and two in different spots where I usually sit down in a chair and plug in my M1 Pro or iPhone to charge (on USB-C charger hubs) and have never had any issues where they drop out from throughput - at once point I had it running full bore overnight. They're from Cable Matters (2025) and Planex (2024). The only issues I have is sometimes they just don't show up when I connect and I have to disconnect and reconnect. That could also just be an issue with USB-C hubs.

I've had a lot of luck with the Plugable 2.5G USB-C adapters, although on some ports they're incredibly unreliable -- I'm guessing maybe something about the power supply?

On a CalDigit TS3 dock, I've had one working off the rear Thunderbolt socket for over a year, but it fails in minutes if plugged into the USB-C one. I've never had issues with a CalDigit Elements (TB4) hub. I've had success plugging them directly into various Chromebooks. On one Windows laptop, they seemed to work on one USB-C port but not the other.

So... reliable once working, but getting there can be a pain.

Seconded - those RealTek dongles are no good.

For my Macs, the IOCrest USB4 adapters with the AQC113 chipset have been rock-solid and can saturate my 10Gbe network. They do run hot but no stability issues. That's what I use and recommend.

The newer RTL8159 based dongles are smaller, cheaper, cooler, and can do 10 Gbe on paper. But they top out at 7 Gbps on Macs, which don't implement the USBx2 spec.

I use several of those UGREENs and they work perfectly fine out of the box and for sustained load as well.

> do not buy a USB-C RealTek RTL8156-equipped multi-gig ethernet dongle to use with your Mac.

Buying a Mac with built in 10gbe is so nice compared to dongle hell.

But it’s not 25gbe

A single PCIe-3 nvme drive will saturate it on a sequential read load.

In addition to Jeff's comments in this thread, I am guessing all of his purchased gear is capital expense for an LLC he runs for his youtube channel. That makes it deductible from personal tax. (again, all guesses, but a common pattern in the US)

My default assumption (with no knowledge about this particular case) is that all gear in YouTube videos is sponsored. Too easy to get jealous of all the fancy setups. But I don't want to become a YouTuber so that settles that.

Jellyfin: https://www.jeffgeerling.com/tags/jellyfin/

LLMs: https://www.jeffgeerling.com/tags/llm/

Etc... (I don't do any crypto)

To be fair he also talks about NTP and RPis.

Running local LLMs split across machines. Pretty sure the author has some other posts about doing that.

Edit: https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-stu...

We refurbish laptops and hence distribute .iso / ipsw images to various refurbishing stations. (or also, think of PXE/network boot situations). More images x more laptops refurbished at once = more bandwidth.

Obviously it's possible to move this to a NAS, not use a mac, etc. etc. - but just wanted to chime in with my little use case of moving multi-gig-files around. :-)

I was just looking for a thunderbolt-sfp solution. Super timely!

> Author talks about 4k video, but AI tells me that for "4K production mastering (ProRes 422 HQ, 4:2:2, 10-bit)" the rate is under 2Gbps.

That'd be the rate for 1 stream at 1x playback. Multiple streams because of overlays/crossfades/etc... along with playback at >1x (such as during rendering in particular) will change that significantly.

NAS stuff can benefit greatly and straightforwardly.

A typical NVMe SSD can easily do sequential reads at 7GB/s, which outpaces a 25Gbps connection by more than 2x. NAS is more typically backed by SATA HDDs, but there's some relatively easy way to put an NVMe disk in front of them as a cache.

Do you ever _need_ it? Perhaps not, but there's plenty of common activities just stemming from NAS usage which see a linear speedup, and would continue to get faster up to 100Gbps even.

ProRes is a compressed and encoded format. In order to become ProRes, it needs to be uncompressed and decoded at some point. uncompressed and unencoded is easier to work with on the rendering side, but much heavier on the data side. Ask your AI about the DPX file format.

The primary use case is the same use case as having 388 hp in a Porsche Carrera. It's cool and it's fun to go fast.

What if it’s uncompressed footage at 60fps? And dealing with many shots. And on top of that maybe working with multiple projects in parallel.

If you're editing video you're not just streaming in real time one copy. You have many streams that you're going to want to work around quickly.