"The biggest mistake I made was high uptime"

Quite. I'm old enough to remember machine uptime being a badge of honour.

However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.

Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".

I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.

Sorry, slight digression.

High uptime implies no patching. We all love patching.

This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.

This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.

(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)

Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.

For a physical machine which you can't easily copy, it's a different story.

I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.

Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected

> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade

LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.

My home setup is kind of this but it's unraid running the containers and one of them is an nginx thingy specifically designed for reverse proxying other services.

This is more or less how I do it, too. Debating switching from Debian to something like Flatcar which is container-first with an immutable system/OS.

I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.

I mean you can update trivially without AI. The problem is the risk of breaking something, which the backup mitigates.

> I was missing something like a template with the defaults that come with UFW, for instance.

FreeBSD does include this! It's implemented using IPFW instead of PF. Check out `firewall_type` key in `rc.conf`: https://cgit.freebsd.org/src/tree/libexec/rc/rc.conf?id=8e08...

For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.

For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.

For more details and to see exactly what each option actually does, check out `/etc/rc.firewall` where this is all implemented: https://cgit.freebsd.org/src/tree/libexec/rc/rc.firewall?id=...

> - PM2 was buggy on FreeBSD, which I used to manage my processes

For supervision?

> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.

The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.

> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.

I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.

[0]: https://nostarch.com/book-of-pf-4e

pm2 has been buggy every time I’ve used it, no matter the OS. Incredibly convenient to begin with but simultaneously unpleasant to use software. Updating environment variables with a deployment has not once ever worked as intended.

For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).

Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.

I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.

Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.

> But are they maintained well?

Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.

Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.

Look pretty well maintained to me. If only judging by recent events.

You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.

Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.

RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.

I would say NixOS, where it is trivial to switch across releases, run software from different releases, and perform rollbacks.

I have been running NixOS on several servers for more than a decade. No reinstalling, upgrading, or any breaks whatsoever.

Ubuntu Pro is free for up to 5 systems. 15 years.

I mean Ubuntu Pro is free for personal use and it extends the LTS support of 5 years so a total of 10 years afaik.

Debian LTS/extended LTS

Probably Debian or Ubuntu. The question is...why do you care that much?

I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.

I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.

Use a rolling release like Arch and it’s supported forever.

FWIW, I once abandoned an Ubuntu server for a decade, and managed to update it painlessly in 20 minutes. That same server is still running today, now with the latest LTS. I think I might have even started with Ubuntu 4, or perhaps 6, and it has been painless all the way through. Perhaps my slow upgrades saved me from early adopter woes :).

I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.

> However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation...

You've been misled.

> something had gone wrong with an earlier kernel update

That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability. But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.

I hope FreeBSD has longer supporting cycle. Its release has a supporting life of less than one year, if missing the upgrade window, then later upgrade is more difficult than others such as debian stable.

My servers/VMs typically run either FreeBSD or Alpine. A Debian here or there where needed (proxmox, VPS that doesn't support Alpine, corp stuff, etc).

I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.

These days it’s fairly straightforward to install on my providers.

I have FreeBSD with Hetzner and OVH. I’ve also used Vultr in the past.

OVH has FreeBSD templates. And most KVM VM/VPS providers will allow console access and mount custom ISO to install whatever you want.

What is there to be afraid of? Don't you have backups?

Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.

dd filesystem to another machine then boot it up with an emulator like qemu and do a trial run

Be careful if you have anything that autostarts that reaches out

My oldest server is on 8.04.

Formal education doesn't typically emphasize this kind of learning. Univerity CS classes will focus on data structures, algorithms, languages, turing machines and finite automata and how they relate to computability.

If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.

My understanding is that the kernels are mostly equal. I’d be pretty surprised if one had a large impact one way or the other. Any differences I’d chalk up to the userspace program running it.

Don’t forget there are serious hardware differences. It’s not apples to apples.

But that’s ok. The author isn’t claiming FreeBSD is way better than Linux. It’s just a comparison of what he had vs what he has now.

Show me those benchmarks

Can you detail the transition? What were the pain points? I feel like you lose a lot of the selling point of OpenBSD as soon as you start pulling from ports, but how could you do anything productive without it

I've had a "Get in loser, we are going to Tau Ceti" on my car's bumper ever since the book came out

I did it this way for a long time, but it was mostly a learning/experimenting/fun thing back to have a "proper" server out there that I can run whatever service I wanted on (be it an irc bouncer or whatever). I'm a grownup now and don't have the time/care anymore and just run them out of s3/cloudflare (which was still "fun", but now I don't need to worry about the cve of the day. I don't mind contributing to the centralization of the internet when I'm paying $0/month for pages that nobody visits. Definitely happy to nerd out again if something ever warrants it).

I don’t do it myself, but “objectively inferior in every meaningful way” is a bold claim. It might be harder, but we (geeks) love to do things ourselves.

If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.

I pay $35/month for a dedicated server for my nothing webpages. One the one hand, I could really host at home for $0/month extra. Or on a VPS for $5/month. I do need my own thing because I run a network testing tool that needs some amount of direct access.

On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.

At least I'm sensible and don't have an actual colo space to visit.

In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.

some people (myself included) like hosting their own stack for fun or for learning.

There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.

I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.

As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.

If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.

s3 + cloudfront takes approximately 2 extra steps every deploy, and about 10 extra steps that are easy to screw up at setup time. It's not a trivial drop-in, but yes, once it's done it's _really_ done.

Another option is cloudflare pages. Can be coupled with any hub or you can just push html artifacts.