wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.

0.3tx per second is decent speed?

And it gets worse with every token.

Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?

I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds?

To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.

Or could you do the same with custom layout of cheap slower RAM?

I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.

It is fun.

Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone.

Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200.

SOTA LLMs are efficiently compression of all the knowkedge humanity has built. Having ability to run it at home to extract said knowledge is important no matter the speed.

I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface.

So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ?

(note. I am not a believer in AGI)

"useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.

I can only guess some post purchase remorse.

Need to justify buying an expensive rig that doesn't do what you expected.

Specifically thinking the people they could do something AI with cpu, and realizing it isn't feasible. Happened at my fortune 20 company. They had to get approvals and ofc it was useless. Plenty people tried to explain, but they were the principle engineer, and out ranked everyone.

"It's not going to work", the topic changed, and we never spoke about it again.

'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?'

- bunch of people only ~4 years ago

You can't improve what you can't measure.

Consider this like if it were the first test

16 tokens / s is not nothing.

Done, Mac -> Max

You're lucky, mine is the M1 Pro 16GB

I just checked eBay. There's an insane price difference between used and new. $5k vs $40k.

No problem at all to read data over and over. In fact, LLM weights are a great candidate for low-quality flash that can't handle a lot of write cycles, and you want a large amount of storage cheaply...

Reads are not generally life-limiting for flash. (Well, no more so than power-on time in general. You still have aging mechanisms like electromigration, but these are orders of magnitude slower than write-induced damage.)

0.01 tk/s on an M1 Max is not "nearly". This is completely unusable, and in no way cost effective.

0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS.

[1] https://www.kimi.com/resources/kimi-k3-pricing

That sounds like the most boring exciting livestream of all time.

Had the same feeling when I first saw min/km units in some (human) running context.

UPD: I know it's not the same at all, just the reversal of units that gets me

I commented similarly below, but as a terrible programmer, I probably perform about 1 minute per token too (at Kimi 3 level). It puts into context how I think about intelligence

You can use an external drive if it's mounted as a writable volume. I would make sure it's fast, maybe thunderbolt 3/4/5 enclosure with a fast drive.

The Github specifically mentions an option to stream it from an online host. It's extremely slow.

The link beat you to it:

> It is not fast — about 16 seconds per token on our M1 Max

Better questions, how many ESP32s would it take to reach 1 tok/s decoding speed with K3?

1 button Kimi morse code interface

why not zip the entire internet to 1MB so everyone can have a copy? because it is not possible - we don't know if it is possible. I mean we know it is impossible, but we don't know if it is possible to do it with acceptable quality loss.