> The article then ends before saying if the new setup solved their problems.

They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!

Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.

Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?

The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.

If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.

Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.

It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.

>The article then ends before saying if the new setup solved their problems.

This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.

I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.

What's wrong with ClickHouse again?

Sounds like a common problem in the AI era.

It doesn’t look like planning and then executing.

It looks like executing and simultaneously reacting to the periodic “novel” information that would have been surfaced during the planning.

Burn enough tokens and it gets harder to tell them apart. Sometimes. But it’s appearance more than anything else.

Failure to plan can’t be repaired with even more tokens.

If I was a user of this product I'd be really worried about the "We rewrote our software 5 months ago, but that didn't solve the issue we wanted to solve so we're doing it again, maybe this time we'll fix it" message of this post.

To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.

I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).

Hi! Author here.

We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.

I don't get the blog post either.

> It feels like a lot of churn and not a lot of measurements or engineering.

They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.

This sounds more like: shoot first ask later. (OMG!)

The intro to that post is so bad: "PostHog's original data warehouse had a single goal: delay the moment a company needs to hire their first data engineer."

It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.

Engineers like to experiment and we get the benefit of learning from their experiences. Not sure why you’re upset.

Author here. I admit I kinda left out the context which is important here. We did not replace Clickhouse. Instead we want to move our warehouse product from Clickhouse to another stack which suits the long modeling jobs and multi-tenancy better. We learned through all our early design partners running huge workloads what we need to change.

As much as I'd love to claim victory, we are not there yet :)

This is the same company with intentionally annoying full-wrap bus ads on Muni. They are trying to capture mindshare.

Posthog is a relatively small company, and they have victory-claiming people already.

NGMI

Author here. This is not about our internal warehouse but the warehouse product we offer to customers. This is still running on our centralized Clickhouse. Long-running data modelling queries and multi-tenancy is a challenge on Clickhouse.

I worked at a place with a couple of overly optimistic principals who were so happy about their computationally convoluted system responding to queries in 600 ms, but it practically meant we needed almost half a CPU per request, and then pre-autoscaling, a strange requirement to be able to failover from taking a region offline so we ran each at around 60% utilization, putting us at about 1 CPU per req/s. Which is just fucking embarrassing.

The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.

A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.

Sounds a bit like this team might have the same disease.

The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.

Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.

On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.

Author here. We didn't leave Clickhouse. It still drives most of what Posthog offers. I missed that bit of context in my post I admit. Instead we are looking into a new stack we can give customers so they can run their warehousing workloads on it instead of the centralized CH.

Stateless CH on Iceberg sounds interesting, we looked into it but Delta/Iceberg had other downsides for us.

But why even bother with ClickHouse? What's wrong with Trino?

I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.

But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.

I actually like the menus. They work great on mobile too.

We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.

I use it as a free web analytics, the UX is so bad and convoluted, it's like a bad joke.

And what's with that bs cookie banner? If the only cookies are functional/essential, they don't need to show a cookie banner. If it's for analytics, then I should be able to reject it. Otherwise, it's just a dick-over.

I actually love this design but I nevertheless agree it's atrocious.

really? I think its one of the best websites going these years.

Author here. I agree. We spent most of the time to make it work at the scale of our large design partners and probably should have stepped back earlier. And yes, with an agent you can have a much larger scope hashed out faster than before.

Author here. Good point. Trino is not made for fast response times. We are using it for long running data modelling jobs or nightly ETLs.

Author here. Good question. AFAIK you cannot use Quack for anything that looks remotely like distributed querying.

Author here. Our fork, like most of our work is public on Github: https://github.com/postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.

"engineering effort" - its all just agents spinning in the background now

Certainly frontend folks churned like mad by picking one package after another to build on. I saw it first hand at a startup. I was the backend guy, just wrote it once. That doesn't mean it was perfect or he wasn't doing fine, but I didn't need to rewrite it so many times. I'm not ready for a world where everything gets rewritten on some new toolset just because. Hope you've got good tests.