They all got played by some ai dunning kreugers. The investors might not know it but anyone who knows anything about dbs knows that failing a jepsen test this badly is not the start of something. it's the end of something.

Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.

Go figure one of the founders is on openais board. Yikes...

> We now run sustained Jepsen workloads daily alongside our own test suites.

Why would anyone believe you when you say this? From the report:

"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.

Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?

Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."

> I'm part of the ParticleDB team.

Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.

What kind of introspection did you do on the root cause of those issues? "Data from deleted table reappears in new vaguely-similarly-shaped table" or "simple row write just doesn't happen" are bugs that are quite impressive to accidentally happen!

How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?

"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.

Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.

I would highly recommend leaving the industry. You are not at the caliber required for this kind of work, and you are not aware that you are not at the caliber required for this kind of work. You bring shame to our profession.

Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.

I struggle to imagine how an organization which is capable of taking things seriously would have produced software with these issues to begin with.

Hi Claude, can you fix all the Jepsen findings and push to prod. Thanks.

Where is the TLA+ proof?

If you took the findings seriously why didn’t you take the outrageous claims you were making in your own copy seriously? Why couldn’t you provide cogent analysis of these issues during the process? How did you hack up your db so bad in the first place?! This thing looks so bad bro I can’t help but think this kind of assertion only adds fuel to the fire.

> we're working on benchmarking now as well.

But you already said you were 1000x faster than Postgres??

Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.

No one in their right mind will use this product ever.

should you have done that in the first place? its a database engine

Once I started writing one (I could say I still am) because I was unhappy with most of them and also, to see how hard it could be

Alas! It turns out it's really hard to do it right.

"But AI can oneshot Redis, so we can just vibecode it."

I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.

> these were tests done on pre release builds

That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.

https://web.archive.org/web/20260518201315/https://particled...

https://web.archive.org/web/20260518200305/https://particled...

Yeah, but none of those builds should have seen the light of day, given all the problems that were found.

And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).

What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.

Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.

Ok sure, so an LLM told you all the bugs are fixed and it’s faster than Redis, just like how it told you before that it was formally verified?

Please try to resist the urge of being a meat proxy.

The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.

Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?

I mean did you see the "primary key equality" thing? Holy shit.

/dev/null is also an extremely fast KV store.

Sure, it still has some bugs resulting in occasional data loss, but it is pre-release so that's okay. Look at the speed, people!

> and all of these issues have been fixed..

You can't be sure until they test it again

I look forward to the second report from Jepsen for the new implementation then.

I just wonder why, from all the things you can vibecode, you would pick a database if you don't know the first thing about database design?

> Its also faster than Redis for KV

No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.

Redesigned by whom? Given how bad your LLM was at bugfixing in the releases tested by Jepsen, I would expect the redesigned version to bring back all those bugs and then some.

I commend Orkes for funding a Jepsen report knowing that it could be published this harshly. I question the wisdom of doing so, but it at least speaks to the team's integrity. Building databases is very hard; I know this from personal experience.

Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.

I decided to dig into these claims a bit.

> Its also faster than Redis for KV

ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.

- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.

- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.

- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.

- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.

> also numbers show this should be top on clickbench

Would this claim survive after validating query results?

Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.

The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.

[0]: https://particledb.ai/benchmarks/key-value/

[1]: https://github.com/ClickHouse/ClickBench/pull/2952

https://github.com/ClickHouse/ClickBench/pull/2952

> Hey Claude, could you find some releases we can send to Aphyr for testing?

... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready

ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22

2 rows returned

MongoDB had consistency issues when running as a distributed system, that's expected if you read Jepsen's reviews. Meanwhile, ParticleDB had bugs like this when running as a single node:

    root=> CREATE TABLE t
           (a int primary key, b int);
    root=> INSERT INTO t VALUES (4, 5), (5, 6);
    root=> SELECT * FROM t WHERE b = 4;
     a | b
    ---+---
     4 | 5

[dead]

^^ this is funny, but I think that's likely how it has happened.

And no doubt they wanted a headline to help boost the Anthropic IPO (I'm kidding but knowing the crypto crowd and the overlap with the AI crowd, who knows)

  > WHERE column = NULL and then query WHERE column <> NULL

NULL means what it means and is conceptually foundational to SQL. Comparing it to anything returns UNKNOWN, the “third value” besides true and false. And WHERE needs a TRUE predicate. So your implied fault that they “have no problem“ really is just a matter of you not understanding the tool you are complaining about.

To match against the NULLness of a value you must use special purpose operators. In this case

   WHERE column IS NULL OR column IS NOT NULL

> I won't push back on it being garbage, but I would suggest that if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork

No. No-one would ever accept a database with _this_ level of brokenness. Like, this makes early Mongo and noughties-era mysql look _amazing_.

> I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL.

You may not like it, but that is entirely deliberate and internally consistent. Blame the people who came up with SQL in the 70s; far too late to change it now.

I am rather receptive to the argument that SQL is a fractal of bad design. But complaining that SQL engines implement well known SQL semantics according to spec is silly.

And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.

Exactly, this is the takeaway. The agents are running everything, the humans have no idea what is going on.

I remember Postgres in the early 2000s, and it didn't have anywhere near this level of problems. It had its issues, especially in terms of performance, and there were of course occasional correctness bugs that tended to turn up when you were doing something unusual. But you could certainly rely on basic SQL operations to work correctly.

> Maybe you don't remember Postgres in the early 2000s

Postgres was already mature by the early 2000s - it was started in 1986 by a veteran who had written Postgres' spiritual predecessor in the 70's (Ingress). So no, Postgres didn't have amateur-hour data loss bugs in the early 2000s. What Postgres was missing out of the box were some features, like replication.

> Maybe you don't remember Postgres in the early 2000s.

Not even the grungiest version of MySQL 3.23 running on some shared hosting provider in the early 2000s would behave as immediately illogically as the database under test here.

Someone with your username should know this.

I remember Postgres in the early 2000s. And early Mongo. And many others.

And bugs like “WHERE 4 = 5”, or selecting from nonexistent columns, or selects permanently modifying table behavior, or any of a dozen others that Jepsen found … didn’t happen in released databases. Even young ones.

Prior to the age of LLMs, that level of bugginess wasn’t “everything has growing pains”. That’s the kind of thing previously associated with poor undergraduate CS homework or “the CEO’s kid wrote our database” corporate horrorshows.

> with no QA

That’s not immaturity. That is a choice. A “the car we are selling does not have airbags, seat belts, or brakes” kind of choice.

> Maybe you don't know too much about engineering

Save it. You have no idea what you are talking about.

> Maybe you don't remember Postgres in the early 2000s

The ParticleDB devs definitely don't

@dang I think HN should revoke and reserve this username.

Edit: I know @dang does nothing but maybe make it more likely he sees it.

It is months that I find subtle bugs far more complicated than the most immediately findable issues in the report. Together with the rest of the world. So there must be something wrong with their AI usage.

I'd guess that all the restraint in this case has been spent on staying professional throughout writing all the individual bug descriptions. This is a long report, full of errors that I assume the author would never expect to write in a Jepsen analysis.

They probably asked an AI to run the tests against their DB. And the AI said "yeah I ran it, all green!" (except with a lot more words and perfect grammar, and maybe a few section headings and emoji.)

Seems to me that “we ran the Jepsen test” is just something an LLM thought would sound good on a DB’s website.

Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.

Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.

The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.

> vendor who feels it their duty to provide color commentary

Honestly? At some point there's a fiduciary-like obligation for it. What ParticleDB delivered is so far past good or bad, it's just straight ugly and then some. It's so poor that, if Jepsen couched it in any way, they'd be implicitly misleading their readers.

The principal values of things like the Jepsen reports is to the community of potential users/purchasers.

I agree with GP, that vendors should be somehow required or very strongly encouraged to participate in such audits. Not because they want to (as you say, most wouldn't like their dirty laundry aired thus), but because consumers deserve to know. U.S. government software groups don't like FISMA/RMF audits either, but they have to do them anyway.

I am of a contrary opinion. At some level of developer insolence, such commentary is valuable - especially if it needs to mention the core basics and foundations that keep the software world running, such as running existing Postgres test suites for a supposedly Postgres-compatible database. In the increasingly sloppified world, even such basics are being eroded - this being a proof.

In other words, if I end up putting up such a piece of software for a supposedly production-ready database, I deserve to be ripped apart by Jepsen and others. Somebody has to assert software quality; if it's not asserted internally, it must be asserted externally.

In yet another words, software development is a trust-based business. If you break it, such as by outright lying to people, then you deal with the consequences.

“No.”

Afraid to show the slop?

Oh, jeez, sorry! There's my Sunday afternoon project, I guess. :-)

Clicking on either of the feeds takes me to a page that just returns an error.

e.g. https://jepsen.io/blog.rss returns a text/plain respond containing only "Nothing to see here; move along.".

Those feeds appear empty, though. Although I could be running into a race condition, since it's the Jepsen site.

They forgot the “don’t make any mistakes” part of the prompt.

Right, when Rachael Gunn (Raygun) embarrassed Australia and the whole of breakdancing on the olympic stage, the nice thing to say would have been "congrats on getting this far, fixing the issues won't be easy, etc..." but what really should be said something like "I'm not sure how it got to this point, but everything about this is wrong, and you should probably focus on something else".