Don't forget the 15 000 or so notifications that you can't realistically turn off

For what it’s worth, the benchmark on the Zstandard homepage[1] shows none of the setups tested breaking 1GB/s on compression, and only the fastest and sloppiest ones breaking 1GB/s on decompression. If you can live with its API limitations, libdeflate is known[2] to squeeze past 1GB/s decompressing normal Deflate compression levels. In any case, asking for multiple GB/s is probably unfair.

Still, looking at those benchmarks, 10MB/s sounds like the absolute minimum reasonable speed, and they’re reporting nearly three orders of magnitude below that. A modern compressor does not run at mediocre dialup speeds; something in there is absolutely murdering the performance.

And I’m willing to believe it’s just the constant-time overhead. The article mentions “a few hundred bytes” per message payload in a stream of messages, and the actual data of the benchmarks implies 1.6KB uncompressed. Even though they don’t reinitialize the compressor on each message, that is still a very very modest amount of data.

So it might be that general-purpose compressors are simply a bad tool here from a performance standpoint. I’m not aware of a good tool for this kind of application, though.

[1] https://facebook.github.io/zstd/#benchmarks

[2] https://github.com/zlib-ng/zlib-ng/issues/1486

First, let's establish a cheery mood: Happy Friday!!!!

Second, I noticed we're extrapolating from a tossed out measurement in "microseconds per byte" here, of extremely small payloads, probably included fixed-cost overhead of doing anything at all.

All leading up to: Is "atrocious" the right word choice here? :)

More directly: do you really think Discord rolled out a compression algorithm that does 23 KB/s for payloads in the megabytes?

Even more directly, avoiding being passive and just adopting your tone: this is atrocious analysis that glibly chooses to create obviously wrong numbers, then criticizes them as if they are real.

zstd has faster decompression

though I always thought lz4 to be the sweet spot for anything requiring speed, somewhat less compression ratio in exchange for very fast compression and decompression

Zstd is generally known for markedly faster decompression (as well as compression) than zlib.

https://gregoryszorc.com/blog/2017/03/07/better-compression-...

I imagine it correlates with the end-user device specs, no?

The bandwidth probably doesn't really matter, but a 2MB must have blob vs a 300kB must have blob at the start of a connection is a big difference.

The start of a tcp connection is limited by round trip times more than bandwidth. Especially for mobile, optimizing to reduce the number of round trips required is pretty handy.

Presumably that's all CDNed and therefore a lot cheaper to serve.

Erlang terms are, to a first approximation, the same as JSON.

To a second approximation it gets more complicated. Atoms can save you some repetition of what would otherwise be strings, but it's fairly redundant to what compression would get you anyhow, and erlang term representations of things like dictionaries can be quite rough:

    3> dict:from_list([{a, 1}, {b, 2}]).
    {dict,2,16,16,8,80,48,
          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
          {{[],
            [[a|1]],
            [[b|2]],
            [],[],[],[],[],[],[],[],[],[],[],[],[]}}}

Some RIFF like format would not be that hard to parse different sections of. You get to ignore the parts you dont recognise and decode the parts you do.

Moving to a binary format would be better for 99.9% of users and would be a slight inconvenience to a few people creating bots. Discord could easily publish a library for reading the format if needed.

To be fair, this is exactly what the Accept and Content-Type standard HTTP headers are for. Clients can tell the API "OK, send me application/json data instead of binary data" or vice versa. You can have the majority of your traffic (client traffic) using the binary format, and still support JSON for bot API usage. This is standardized for both WebSockets and HTTP.

Discord is different from IRC in both the scale and payload. It's besides the point anyway, even if discord is still an unoptimismed piece of shit.

You clearly have issues with that unpolished turd. So I figured I'd offer my insights in what often causes that.

If you still insist on having your computer coming to a crawl because IRC did it better than that's entirely up to you.

Lol, in fact it works on all my machines. And lots of other peoples machines.