Tape + Iron Mountain is difficult to beat for offline copies.

Surrendering just a little bit of absolute control over the physical media can dramatically increase the likelihood that it survives whatever apocalyptic event. I don't know that I've ever heard of tapes being stolen from Iron Mountain. Even if this happened, what are the odds that they would get all the tapes. Your tapes? It probably looks like the warehouse from Raiders of the Lost Ark in there.

Anything that requires manual work will inevitably fail as a backup strategy in my opinion. You might not need that backup now, while you actively think about it, but in 4 years, when you had other problems to deal with and haven't done a backup in a long time.

For small datasets, a cloud backup (think s3 or azure blob) with credentials that can't be harvested automatically by a malware (eg custom backup script with encrypted credentials - claude will happily write one for you in seconds) is as good as offline. For small datasets (code base, important documents, even photos if you don't go crazy - or perhaps backup a lower resolution as a dooms day last resort thing), this is nearly free.

For larger datasets, you can buy some cheap X11SSH-LN4F or X11SSL-F motherboards on ebay with RAM and CPU for ~$100. These can be remotely switched on and off programmatically with IPMI (same thing, custom scripts with encrypted credentials - claude is your friend). Have your NAS perform an incremental backup once a week or once a month and keep it off the rest of the time (or trigger it from a raspberry pi with own credentials if you don't want to connect IPMI to your LAN or have encrypted IPMI credentials on your NAS). And unless a malware hits right at the time of the backup, it is as good as an offline backup while also being automated. Doesn't protect from a power surge though, which may or may not be a problem depending on where you live.

Also have your backup pull data from your NAS rather than the other way round, and run with different credentials than your NAS, so a malware can't jump from the NAS to the backup (or encrypt the backup). I have seen first hand that if you reuse admin credentials between machines, one machine compromised means all machines compromised within minutes.

My main backup server uses ZFS and I replicate (using my own https://github.com/fazalmajid/zfssnap but there are plenty of alternatives) to a 14TB USB drive using ZFS snapshot and incremental zfs send which makes it very fast, using ZFS encryption in case they get stolen.

I have two of these and rotate them weekly and keep one at my work (and I have a self-hosted healthchecks.io instance to remind me if I forget).

I only have a small collection of bits in true regularly updated offline backup (my authentication/other DB, some important documents).

An extra protection against the possibility of malicious access such as ransom attacks is a two-step “soft-offline” backup.

The source machines backup to a central place, and the backup machines pull copies from there. The important part is that the source machines can not connect (or at least can not authenticate against) the final backups and vice-versa, so a malicious process/person getting into one can not affect the other and vice versa. Obviously if I don't notice the damage immediately then the most recent backups may be corrupted because damaged data was pulled, but past snapshots (taken after each pull at the backup side) will still be clean.

You have to be very careful about storing credentials to make sure source credentials don't ever end up on the backup machines and backup credentials don't ever end up on the source machines, even well out of the way of normal places like ~/.ssh, because a targetted attack might find them, but it gives almost the assurance of an offline backup while still being fully automatable. My backup site credentials are in my true offline backups, I need to refer to them for maintenance access, and then they get cycled after that use.

Restores can be mediated the same way. Verifying backups can be done by both sides running hashes on the files and posting the list back to the central machine for comparison - anything that differs without having a timestamp after the previous snapshot is likely corruption.

As a side note about corruption: if doing snapshots in the filesystem (“cp -al after rsync” or one of the many similar options) make sure you have more than one snapshot chains (on separate storage if you have resource for that). If you have a file that hasn't changed in years so every one of your snapshots points to that one version, it could only take one random filesystem error to completely lose the file.

I have a locker at my place of work where I store a few HDDs and USBs. These are the most up to date, but I also have others at my parents place and my in-laws. Gets troublesome keeping track of which ones are up to date as of what date. Good challenge for staying organised though, I've got a whole naming system and numbered hierarchy and scripts that run ordered by priority.

Well overdue for a refresh.

Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view.

Yeah, but an attacker is unlikely to compromise both you and your storage provider at the same point. Many S3-compatible storage providers (e.g. Backblaze and Hetzner) support object lock, where you can lock objects for a certain number of days (and refresh locks if the objects are still used in recent backups). Typically object locks are implemented such that not even you yourself can remove the data from the account settings.

E.g. when I cancelled my Backblaze B2 account, I had to wait until the object locks expired before I could remove the remaining data and delete the account.

Arq on macOS has great support for object locks BTW.

I get that a lot of immutable backup solutions are code, but they don’t have to be. You can do stuff like block writes in firmware or even blow fuses on the TX traces.

Ideally an offline copy that can be made read-only with a physical switch. So that when you're trying to restore, you know nothing is going to mess with it. Not sure what the right solution is for something like that.

I’ll take the chances that the attacker hasn’t also hacked AWS to get around data retention policies.

> I went down from a whopping 8TB to 300Gb doing that and it's still shrinking monthly.

Just hours ago on another post I had commented

> discard/reject/delete: ≈400GB → 25GB → 3GB (< a decade ago; I remember) (I was shocked to see how little of that really mattered).

This is one of the most important aspect of managing your personal data in any meaningfully sane and sustainable (!) way.

Then being deliberate about one's digital footprint.

I don't particularly like the idea of my relatives sneaking into my files when I am dead. I destroyed all my father's files when he passed to respect his privacy.

not sure if i got this right, but it sounds like you experienced some things and you want to share something, but you are telling others how to live instead.

imap-backup. It just extracts everything from imap server as mbox. Then that's rdiff-backup'ed to target disks. You can imap-backup it back to another IMAP server if you need to.

However it's better to just delete most stuff so you don't have to back it up. Think my mailbox is about 50Mb.

or use imapsync to pull emails, then backup the resulting folder.

> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice

I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity.

So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline.

That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often.

He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life.

Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity.

Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.

Less hard these days. AI is a game changer for learning new technologies. It's like having a highly paid expert available to answer all your questions about your little USB backup. Makes learning how to use properly a new software trivial. And priceless when troubleshooting.

I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!

Oh interesting I hadn’t considered this. I guess the only trouble would be if I am out with my laptop and I have no internet connection but this seems like a good tradeoff for simplicity

You don't want to compile code over SMB, even on a fast network. And what happens when you travel with patchy internet access? I use syncthing to a NAS, and the syncthing data being backed up by the NAS along everything else it contains.

Exactly.

What does your practice recovery run look like? I've got backup set up, but I don't know how to best test it. For example, do you restore everything from Hetzner, or some random sample?

What solution do you use that syncs/backs up while the device is turned off? IME stuff?

It's a pain in the ass on new systems if it copied restrictive permissions to reset/gain access.

Ntfs equivalent of Chmod/chown is a "go have a long lunch" type of operation.