I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make the most sense, say, by user, session, or vehicle (still bottomless).
And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from the shared WAL across streams, server-side batching, and client-side in-memory pipelining, especially with HTTP/2, without as much connection pool overhead.
In practice, you can go to the extent of achieving up to 100 MiB/s throughput per stream. Considering how granular streams can be, you'd rarely need as much. The latency for a durability ACK is, however, the price to pay, which is going to be ~250 ms, or lower with S3 Express, which I'd say covers most real-time use-cases. The design itself is easy enough to extend to a disk-staged WAL for single-digit durability ACK latency.
Have you look at Google's new Rapid Bucket offering? Google Cloud Storage is arguably as good as S3, and is protocol compatible; Rapid Buckets are a type of bucket which supports appendable objects and low latency I/O. The downside is they can only be zonal, and they're a bit more expensive.
PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms.
I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!
Ah, S3 Express does look like the same thing! Looks like directory buckets S3 Express also allows appending to any object, while I believe GCS only allows you to append to a new object and then "finalize" it.
I would also check out Tigris [1], which has an S3-compatible API. Their main claim to fame is that buckets are low-latency, multi-region and replicated by default, so supposedly you get region-local latency no matter where you are reading or writing from. I have not done any rigorous performance comparisons, though. What's amazing, if it does perform well, is that egress is free, and the pricing is otherwise the same as GCS/S3.
[1] https://www.tigrisdata.com/
Fwiw the only two times I've used kafka in my career have been for traffic on the order of GBs/sec. And the folks I know who have relied on streaming pipes for genuinely realtime stuff built bespoke systems with RTT on the order of 10s of micros.
There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything is a really useful thing, but acknowledging that the system is opinionated such that it covers a specific set of things well is generally a better argument than 'this basically does everything that people need'.
Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ.
Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis.
There are other implementations that support the Kafka wire protocol and are cheaper in exchange for latency, e.g., AutoMQ and WarpStream.
That said, I’ll release Disk/EBS-staged WAL soon enough: https://github.com/PicoMQ/picomq/issues/13 as an add-on to cover low-latency needs.
Also (in case this isn't obvious) I'm 100% a fan of the decision to _not_ tightly couple to the kafka wire protocol. Kafka's apis are full of landmines and gotchas, and you can do much better from a UX perspective if you're not married to their quirky semantics.
I had heard people talk about the operation pain involved in keeping Kafka alive (which is a thing for sure), but what I was surprised by was how many things behaved in a slightly unobvious manner that wasn't loudly-documented (e.g. if you're using transactions for RWP loops the default rebalance protocol is unsound and transaction markers take up an index in the log so you no longer have contiguous indices in your message stream etc).
Haha, it was evident. I'd echo the complexities of managing self-hosted Apache Kafka at scale.
But because the Pico's semantics are close to Kafka, but not tightly coupled, it's quite feasible to support the Kafka protocol, * with some restrictions *, such as a topic can only ever have one partition, no support for transactions (since they would only really apply when producers are publishing to multiple streams), and more along those lines. As a result, you'd get Kafka with no topic tax and a lot less operational complexity.
Not familiar with this particular library, but similar libraries use S3 Express One Zone which has write latency <10ms, so you can use that for the WAL and compaction can move data onto other storage classes in the background.
Regular S3 has write latency 100-150ms, which might be fine depending on your workload anyway.
You nailed it! With some of the similar products I've seen, they either inherit the Kafka protocol and hence the KRaft and other complexities, or go the other way with single-node only deployments, commonly just using SlateDB's single-writer model for durability.