25 minutes ago
> The exciting part of this is that there are four 400GbE QSFP56-DD ports for a total of 1.6Tbps of network bandwidth.
That I can't break down to 10gbit or 1gbit without another switch...
> The list price is $1295, but these often sell at a street price of $1100 or so, making this under $0.70/Gbps
It doesn't work that way...
I like mikrotik, but who is this switch really made for?
4 hours ago
I haven’t been directly involved in networking for a while, so it was funny to tune back in and see that now not only is 10Gbps apparently no big deal, they even put it on the management interface!
3 hours ago
Noting actually says those ports have to be for management.
Most people probably would use them for something else.
2 hours ago
They're not connected to the ASIC so anything on those ports is taking the slow path.
4 hours ago
> While $1100 is not a low number
It really is when you're talking 400G.
OK there's very few ports on this, but in general if you're looking at 400g interfaces. $1100 won't even get you a genuine Arista 400G passive DAC.
4 hours ago
Is there an actual use for this in a homelab? Isn't 25gbps enough that no SSD will saturate it?
3 hours ago
Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.
4 hours ago
The second sentence of the article:
> We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one.
I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.
22 minutes ago
Oh I'm sorry, just let me casually spend 30k...
4 hours ago
I see, so only if you are running an AI cluster. And what is it transferring between the nodes? Partial computations?
4 hours ago
Essentially, yes. The DGX Sparks, at 128 gigs for ~$4700 are one of the cheapest ways to get enough high-ish speed memory cobbled together to run one of the more capable open weight models to run at home/for a small biz. This switch lets you combine the memory of 2-4 of them.
3 hours ago
4-8 of them. 2 Sparks can operate without a switch.
3 hours ago
I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.
By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.
I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)
3 hours ago
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
2 hours ago
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
an hour ago
I understand that part, what I don't quite get is why we can't divide the layers such that each node only needs to talk to at most 2 nodes. I assume I'm misunderstanding how work is distributed between them, but haven't looked into the details yet.
3 hours ago
Yes, AI... this video does a good job walking through: https://www.youtube.com/watch?v=QJqKqxQR36Y
(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.
4 hours ago
I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.
2 hours ago
Yup, I have a pair of CRS504, so I can learn MLAG and my three node cluster doesn't really tickle the sides of that.
3 hours ago
Nic needs pcie5 x16 slot, so on am5, take out your gpu