Was not able to find any mention of AI or LLM usage on the article. The article is very detailed and goes in depth about how they have been able to do it. If anything it just shows the database level expertise and understanding of the existing implementations to find the optimization opportunities.

Unless its explicitly mentioned lets not dilute the credit of the folks who worked on.

There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.

Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*

World experts + LLMs = moving mountains.

(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)

I don't completely disagree with your hypotheses but it feels like the hard part of his TIN stuff isn't BM25 (which has been around for donkeys years) it's all the hardcore storage engine work around it. And is an LLM particularly good at e.g. segment merging under a thousand updates a second? I've had a few situations where I've been told "we've hit the perf floor" by Claude only to have persisted myself and shaved substantial amounts off still.

More damning for the theory might be that I think paradedb's pg_search predates the agentic coding by a few years?

ParadeDB's implementation builds on the Tantivy crate, which predates AI coding.

Becoming more the norm for them, Neki is the same.

Immediately rules out ever using them (though I don't currently have any problems that would benefit from that level of scale currently, have in the past though).

Postgres's license allows this but for me (personally) it leaves a bad taste.

Also it's not really "full-text search for Postgres" it's "full-text search for our hosted version of Postgres" so the title is a little misleading.

The problem is that they don’t support bare metal. I’d love to use PlanetScale in our bare metal servers.

If you mean tsvector/tsquery - https://www.postgresql.org/docs/9.6/textsearch-intro.html - it's very good, but it's missing an important feature: ranking based on the overall document collection.

PostgreSQL built-in FTS provides a score for each row based just on the data for that row.

Relevance algorithms like BM25 take overall corpus statistics into account. If you search for a bunch of words and some of them are less common than others in the overall set of documents, documents that match THOSE words will score higher than matches for other words in your search.

That's what all of these additional extensions are providing.

If you read deep into this, they claim much better performance than the built in search; they also imply that the built-in search is missing features they provide but don’t make clear which ones (I think it is just support in the same index for queries covering other conditions on other columns, because every other feature they claim seems to line up with the built in search features, which have been around for about 20 years.)

The built in search can't do any scoring mechanism that involves corpus-wide stats, so things like tfidf and bm25 are right out. If you don't need that then great, but in my experience the results are much worse.

It has it.

Incredible? No.

SQLite FTS relies on shadow B-trees under single-writer locks. Postgres index access methods must map postings directly to physical ctid tuples, surviving MVCC visibility checks and heap tuple churn.

I’m one of TIN’s developers and if you google my username you’ll see I’ve been in this space for a long time.

The answer to your first question is simply: yes

As far as your second question, what customization do you need that you believe TIN or PlanetScale doesn’t provide? These are things we can do, with alacrity.

in my experience it’s pretty common to find big inverted indexes for text directly in the database - not necessarily large docs but certainly free text records in volume. using bm25 and unicode’s breakiterator is a very good way to build it. like putting lucene in the database basically - makes a lot of sense when the database is already large. places that bend over backwards to move search out of the db are usually trying to avoid having a very large db (and often end up with one anyway, getting the worst of both worlds)

It has FTS built in and has had it for a VERY long time.

Poe's Law strikes again.

Is this an ad? Postgres has had search for more than a decade.

>is this 21st century embrace, extend, extinguish

I assume here you're talking about Amazon's modus operandi?

From the benchmarks deep in the document, TIN is much faster than built in text search (tested against GIN, which is itself much faster than GiST for text search.)