> Repeat after me, "my creative energy should be spent on my customers"

"I should save my energy, so I won't exercise."

"I should save money, so I won't deploy it towards investments."

I don't think "creativity" is a zero-sum, finite resource; I think it's possible to generate more by spending it intelligently. And he pointed out how moving towards immutable infrastructure, while more "standard," directly hurt customers (the engineering team lost deployment velocity and functionality), so it's especially weird to end your comment the way you did.

To say "immutable infrastructure is just more straightforward" so definitively, from the limited information you have, is just you stating your biases. The stateful system he describes the company moving away from may also have been pretty "straightforward" and "boring," just with different fixed points. Beauty in the eye of the beholder and all that.

>> Repeat after me, "my creative energy should be spent on my customers"

I agree with you. But from the blog:

>> "Product requirements were changed to play with the adopted tech."

That's when things may have gone too far.

Every customer has one pet feature. They want that feature, then they want less downtime and more performance, then maybe they want the features other customers want.

The biggest problem with bespoke deployments like this is that they can widen the gap between a happy path and a disaster. If deployments are faster but not derisked, customer expectations are raised but failure cases become more costly.

It's "weird" to want low downtime?

The general nastiness of updates is one of the largest customer friction points in many systems, but creative energy should be directed away from fixing it?

Gross.

How is downtime beneficial to the customer?

Don’t forget startup innovation culture: everything has to be disturbed. Encourage with tax exemptions for « innovative » jobs and you’ll have cohorts of engenders reinventing wheels from infra to UX in a glorified innovative modern "industry".

I think theres more to it than that.

You are correct for 90% of the cases, but this also kills innovation.

What are the specific lessons worth preserving, but being lost?

(I assume that "keep an image, it's too costly to rebuild everything from version-controlled sources" is not such a lesson.)

> undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary.

You can mostly prevent this by mandating that fresh nodes come up regularly. Have a management process that keeps a rolling window of ~5% of your fleet in connection-drain, and replaces the nodes as soon as they hit low-digits of connections.

Whole fleet is replaced every ~3 weeks, you learn about any deployment/startup failures within one day of new code landing in trunk, minimal disruption to client connections.

> The big thing about immutable infrastructure is that it is reproducible. I've seen both worlds and I do appreciate the simplicity and quickness of the upgrade solution presented in the post. The problem with this manual approach is that it is quite easy to end up with a few undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary.

The point is not really automation vs manual. Hot loading is amenable to automation too. The point is really that when you replace immutable servers with state with another set, there's a lengthy process to migrate the state. If you can mutate the servers, you save a lot of wall clock time, a lot of server cpu time, and a lot of client cpu time.

I deal with this issue at my current job. I used to work in Erlang and it took a couple minutes to push most changes to production. Once I was ready to move to production, it was less than 30 minutes to prepare, push, load, verify and move on with my life. I could push follow ups right away, or wrap up several issues, one at a time, in a single day. Coming from PHP was pretty similar, with caveats about careful replacement of files (to avoid serving half a PHP file) and PHP caching.

Now I work with Rust, terraform, and GCP; it takes about 12 minutes for CI to build production builds, it takes terraform at least 15 minutes to build a new production version deployment, and several more minutes for it to actually finish coming up, only then can I start to move traffic, and the traffic takes a long time to fully move, so I have to come back the next day to tear down the old version. I won't typically push a follow up right away, because then I've got three versions running. I can't push multiple times a day. If I'm working many small issues, everything has to be batched into one release, or I'll be spending way too much of my time doing deploys, and the deployment process will be holding back progress.

The funny thing here is BEAM is “immutable infrastructure as a programming language environment” which, to me, is strictly superior to the current disjunction between “infrastructure configuration” and “application code”.

Erlang defaults to pure code and every actor is like a little microservice with good tooling for coordination. There are mutable aspects like a distributed database, but nothing all that different from the mutable state that exists in every “immutable infrastructure” deployment I’ve seen.

> Now, for immutable infrastructure, you have a whole different set of problems. All your changes are nicely logged in git, but to deploy you need to rebuild containers and roll them out over a cluster.

The real issue is that it effectively forces externalizing nearly all state. On the surface, this seems like it's just a good thing, but if you think about the limitations and complexity it creates, it starts seeming less unquestionably good. Sometimes that complexity is warranted, but very frequently it is not.

That being said, I think modifying code is a running system without pretty strict procedures/control around it is... dangerous. I've seen hotfixex get dropped/forgotten because it only existed on running system and not in source control more than a couple of times.

The real issue is that the languages most use do not support, reliably, another approach.

k8s is not the issue. Worse Is Better languages and runtimes are.

Two years after writing A Pipeline Made of Airbags, I ended up prototyping a minimal way to do hot code loading from kubernetes instances by using generic images and using a sidecar to load pre-built software releases from a manifest in a way that worked both for cold restarts and for hot code loading: https://ferd.ca/my-favorite-erlang-container.html

It's more or less as close to a middle-ground as I could imagine at the time.

A truly well crafted app requires very little maintenance or support, and that maintenance/support has already been throught through and made easy to learn and do.

These things are possible, and they fit economically somewhere in the 3-5 year maturity of a system. Years 1-3 are usually necessarily focused on features and releases, but far too many orgs just stop at that point and aren't willing to invest that extra year or two in work that will save time and money for many years to follow.

I believe this resistance is due to the short-sightedness of buyouts/IPOs or simply leadership churn.

If the original devs wrote good documentation then pretty much anybody can maintain it easily.

Google doesn't use Kubernetes internally. Kubernetes is a simplified version of Borg. If it's taking 45 minutes to deploy a change, that's on your company's platform team, not Google.