> I know the article says "analytics workloads," but I wonder if this pattern could be used for ex. bursty-write workloads.

The key point is that Delta Lake and Iceberg concurrency control is so coarse-grained. As I mentioned in the article, they did this because it seemed they wanted to avoid an external highly-available metadata store (e.g. FoundationDB which is what Snowflake uses).

Just about every Database as a Service (OLTP or OLAP) these days supports storage tiering to cloud blob storage. Delta Lake and Iceberg just took cloud blob storage to the extreme by trying to do everything in it (or as much as possible, Iceberg seems to require a Compare-and-Swap operation which I'm not sure is possible to express with atomic PutIfAbsent alone; seems to require an external HA metadata store for their one current metadata pointer). And in doing this they gave up fine-grained concurrency control.

I have also been thinking about this:

https://slatedb.io/ is new and KV on object storage. Maybe helpful.

There is lots of similar work over ipfs, I have not had the chance to experiment yet, but I'd like to see what a workflow is like with (eg): summa: https://github.com/izihawa/summa

I tried to implement trigram search on object storage using Zoekt: https://github.com/sourcegraph/zoekt, but I found that common queries load 30-40% of the entire index, so setting it up without some kind of caching strategy felt a little ridiculous.

Back in the day (and I'm talking about when memcached was still written in Perl here, so late 90s or so) "bursty-write workloads" were a really good fit for us - using SQL queries as keys and storing the responses as values. For every read you first do a (super fast) memcached query using the SQL you're about to run as the key. If it exists, you just send the value back without running the query, if it doesn't exist you run the query and store the response as the value before returning it.

It depends a lot on how big your dataset is, and the distribution of reads across your data. But if you have a "fat head long tail" type distribution it can work really well.

(Of course cache invalidation is always a hard problem, which we worked around by trying to ensure all our code that did database writes knew how to delete relevant memcached entries, force expiring the whole cache at our lowest traffic times, and having a big red panic button that would delete the entire cache if needed.)

My approach to this class of issue has been to make the reindex be "Creates a new search server, build the index, then switch the application to the new one and turn off the old one".

At the cost of doubling storage/IO requirements for search, this works around virtually every kind of outage-due-to-index-corruption (don't switch to the new one if it fails healthchecks).

> however, I think the article would benefit from distinguishing this approach from a 'real' database. that is MVCC gets you a lot more concurrency because it spreads out the exclusionary state in both time and space.

I thought I already was going to be seen as being unfairly critical of the course-ness of the concurrency control.

We use the same approach in time series database I'm working on. While file creation and fsync aren't atomic, rename [1] syscall is. So we create a temporary file, write the data, call fsync and if all is good - rename it atomically to be visible for other users. I had a talk about this [2] a few month ago.

[1] https://man7.org/linux/man-pages/man2/rename.2.html

[2] https://www.youtube.com/watch?v=1gkfmzTdPPI

That's a good point but I think the system will still be correct in this scenario just unavailable. If a transaction ever fails to fully write a transaction metadata file all future transactions will crash while reading it. You can recover by deleting the faulty and final transaction file. Link is a better choice though you're right. I didn't remember it will fail if the new file already exists too. I'll rewrite my code. Thanks!

If you're working at a scale where you accumulate a ton of data quickly and don't know what questions you want to ask, it gives you nigh-infinite scalability without nigh-infinite maintenance, so you can ask questions later.

The other use-case is akin to giving a really big house to a hoarder.

> Can anyone explain the rise in DBs backed by object storage?

I dare say it's a "when the only tool you have is a hammer" problem.

But it might also be the fact that object storage services are reliable, relatively cheap by cloud standards, and do not have a upper bound on resource availability.

I think the main attraction of object storage like S3 is that it already solves the hard problems (distributed-ness, scalability, durability, etc) with a simple(r) API.

Also, apis like S3 are defacto standard with many vendors supporting a descent subset of it.

One other reason that hasn’t been mentioned is that it eliminates inter-AZ network fees. WarpStream has some numbers on their site that show huge savings by using S3 as the backbone.