In my experience, this is the path many developers take. I’ve consulted to hundreds of companies over the years, and it’s alarming how often I’ve encountered decisions driven by what someone wanted to learn or work with, rather than what was actually right for the problem.

That happens at every level, from individual developers through to project leads and CTOs.

Consultancies are often no better. Choosing technologies that require substantial or highly specialised skill sets seems almost routine. I’m looking at you, Kubernetes.

I’m not entirely innocent here either. I owe a decent portion of my mortgage to MuleSoft consulting. That said, I don’t think I ever pretended it was always the best solution. Even while working directly for MuleSoft, my recommendation in probably half of the engagements was some variation of: ‘You’re using the wrong technology for this.’

But by then, an executive had usually tied their reputation to the project and the platform, commitments had been made, and changing course had become politically harder than continuing.

And so we persist.

In my experience, the best technology choices are boring ones. There’s still a large area of immature technology you can get creative with (like Backstage or Port for software catalogs and setting up a nice golden path”), but the meat and potatoes of development work should be a boring choice, that follows a well-tread path within a large ecosystem of developers.

There are exceptions, but they’re not for the majority of organisations.

So take 16 independent computers and have them each solve the problem separately. You'd still be saving 80% compared to the paper's benchmarks. It wasn't close.

McSherry does a lot of interesting work on making monotonic/incremental distributed systems efficient (e.g. Differential and Timely Dataflow). Those kinds of systems scale much more linearly.

I dunno, it often matters a lot less than you think when something goes down. And distributed systems have a knack for going down in a much less predictable way, it's not going to automatically make your system more reliable.

(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)

Sometimes your entire business is just a laptop with a python dictionary and a backup power supply