dbdr
3 days ago
The novice said: "I will save my working files, but not my system and application files, as they can be always be reinstalled from their distribution disks."
The master made no reply.
The next day, the novice's disk crashed. Three days later, the novice was still reinstalling software.
CJefferson
3 days ago
In my experience that works in box like 80% of the time. Which is great! But a backup is great in case it doesn’t.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
edoceo
2 days ago
Backup the distribution files via mirror/proxy-mirror?
rolymath
2 days ago
I think the point was a single file for bit by bit replication. Managing infrastructure isn't good ROI here.
thenobsta
2 days ago
The single file bit-by-bit replication is only possible if those bits still exist and you have guarantees of it. Those bits are a hidden dependency unless you mirror them. Without a mirror it seems like replication will work most of the time, but will eventually fail in ways you don't want.
CJefferson
2 days ago
I mean I could but I find a full drive backup isn’t that much bigger than a more selective backup and whenever I try to back up part of a drive I never grab the right bits.
gnull
3 days ago
It's not guaranteed to be bit identical (the why of it is a can of worms), but it's identical enough for all practical purposes.
gf000
3 days ago
At the same time it is significantly more "hardware-change" resilient, since the same backed up config file (minus perhaps some very system specific lines that you may or may not have) can bring up a functionally equivalent working system on quite distinct other hardware as well.
orbital-decay
3 days ago
If your NixOS setup is not of hello-world kind, you typically have sops in it (or another secret management tool) and want to backup your age keys as well. They're necessarily separated from the main config tree so they don't end up leaking to the globally accessible nix store, and in the most typical case they're located in a 600 file outside the git repo. Forgetting to back it up is a classic NixOS gotcha and also a generalized version of this koan.
KnightHawk3
2 days ago
Wouldn't you just re-encrypt the secrets with a new host key? That's what I usually do (boot once to generate keys, update the secrets, rebuild and switch remotely to the machine)
PunchyHamster
2 days ago
Does Nix keeps old version indefinitely ?
Like, you can do the same with any distro and say Puppet or Ansible but only to within that distro packages, any 3rd party stuff (and people use more and more stuff that's 3rd party to the distro) might just have dead link for the download.
Also, plain restore from backup is still faster
l0b0
2 days ago
It does, actually! Because only the package definitions are needed, and you can get those from the tarball of the relevant nixpkgs repo commit. And since building+installing is the same Nix command as downloading+installing, whether the binary caches still have the old package (or are even still up) is only the difference between a fast download and a slower build. This is different from most package managers, where trying to install anything old means spinning up a complex build environment and learning a bunch of new tools.
Of course, if even the original sources are no longer available, you'd have to change the broken URLs. And if the source has somehow been scrubbed from the Internet you're SOL
mindcrash
2 days ago
My strategy for Gentoo, more or less:
/etc/ - all modified configuration files, including
/etc/portage/ - make.conf, package configuration (package.accept_keywords, package.use, and friends)
/var/lib/portage/ - world, and world sets. basically everything portage needs to rebuild the system
/var/db/repos/ - notably local and/or self-owned package repos
/home/<username>/ - no explanation needed
/root/ - no explanation needed
This should contain enough to pretty much rebuild a Gentoo system from a clean downloadable Stage onwards without (too much) intervention.
cyberax
3 days ago
I typically use restores from backups for my personal computer as opportunities to clean things up.
intrasight
3 days ago
Reverting to a snapshot is a better approach than using backups. My host machine is clean, as all my work is done in VMs.
b3lvedere
2 days ago
If bare metal recovery takes shorter time than reinstalling, then you should make bare metal backups. When bare metal updates, so should your backups.
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
avinoth
3 days ago
Is it possible to do that through Veracity? I don't want to use a non-Veracity based solution when Veracity already supports it.
zdc1
2 days ago
I feel that this made more sense in 1997. Especially in the context of Windows Server and whatever people ran on it.
ocdtrekkie
2 days ago
If you think this is historical, you have a lot to learn about production servers in the wild! =)
Just as a not-insignificant portion of the world still relies on a handful of ancient AS/400 systems, a pretty big portion of society depends on the good running of hand-configured Windows Server.
kbenson
2 days ago
You've missed the point. It doesn't matter that you can rebuild the system exactly, the example in question had little to do with how accurate the restores were, it had to do with the time to reinstall.
I too can entirely rebuild many systems simple from kickstarting a new one and applying an ansible playbook. Do I want to? No, that's too slow, and I might be doing this operation for tens or hundreds of systems.
We have multiple levels of backups. We use veeam for VM and physical backups of the entire system to a disk store that's also shipped off-site to an immutable store, but we also run an older system based on rsync which rsyncs the system to directory on a server (sans some data directories) that uses snapshots to keep space usage minimal for the purpose.
In many cases, if we want to restore we can literally just rsync data back to the host, whether one or a few files or most of the system.
I can restore many types of failures within seconds or single digit minutes at most, and the time required is mostly unattended time. Time to restore is often an important and overlooked metric, until you've been bitten by it and realized that an hour or more of work from a person to restore something doesn't scale well in some scenarios, and those are often the most important ones.
__MatrixMan__
2 days ago
But it's not actually slower. When NixOS loads you get a list of all previous states to chose from. Booting to any one of them is just as fast as booting to the most recent one.
Restoring some backup via copying bits around is much slower because you have to wait for those bits to copy, whereas booting to previous configs is just rejiggering references to bits which are already in place and need no manipulation.
Immutability buys you quite a lot here, although some up-front discipline is necessary to take advantage of it, which might be a dealbreaker for some.
kbenson
2 days ago
That's useful if the system is still up and you can trust the state information, but when you have a disk failure, or there's enough system state corruption that you don't want to rely on that state tracking, you're then going to need something else.
Also, are you pinning your package versions in your config? For every single case of that, is the original package still available? Were they coming from remote sources which may not keep prior versions always available?
Did you build anything from scratch? Did you supply a source or binary archive to install from? Is that source still available, of the exact same version?
There are numerous things you need to worry about when rebuilding from what the state should be based on a description of that state (which may include arbitrary actions), as opposed to restoring from a recorded state that includes everything needed in the recorded data.
The more you can record the better. Should you also record nix configuration state and use that to restore when it makes sense? Sure! If it's simple and easy and doesn't use an inordinate amount of space, why wouldn't you? Just don't confuse nice to have with sufficient based on the specific requirements and goals your backup system needs to meet.
We have entire system images in an immutable remote store, but that's not what we go to first, it's what we go to when our quicker and easier systems are insufficient. If we were running NixOS, we'd probably keep everything we're doing now, and then also just add the NixOS state as a separate thing that's backed up and saved on every change (and in fact, along these lines, we also run etckeeper on every host and track all changes to /etc with daily commits if something is changes but also an automatic commit at the beginning and end of every ansible run we do that applies our host configuration).
More is better. If I want to see configuration changes, I can usually just look in etckeeper on the box, if I want to restore a tree to the exact state it was at a specific point that a backup was made through rsync, I can do that easily enough (or just navigate the directory tree on the backup server, if I just want to see what it was, or diff between different states at different times). If I need to restore the whole system image from a known good backup, I can use Veeam and the local repository for that. If the local repository is dead, I can use the replicated off-site repository. If an attacker has destroyed both those, I go to the immutable off-site backups, which I'm probably doing to use after rebuilding my backup infra from scratch to restore from. Each of these protects from slightly different scenarios, and offers different trade-offs for security and integrity and ease of use.
__MatrixMan__
2 days ago
Sure, you may lose out of the "faster" part if you have to pull from a remote cache (both inputs and outputs) instead of your local one. And maybe just blindly relying on the community maintained caches isn't right for you, so now you're still in a position maintain a cache and back it up through traditional means. It's not a magic bullet, probably doesn't pay off for a lot of use cases.
But what it buys you is a sort of verifiability that you can't really get any other way. Anyone can build any part of the system from its declared inputs and say "hmm, I got a different hash than you, maybe something is up." If you're restoring from an image which somebody has tampered with, then the image becomes the source of truth, rather than the source code, and it gets a lot more difficult to scrutinize its validity because the pool of available scrutinizers is just the people who care about your particular image--that's likely to be a lot smaller than the set of people who care about whatever sources you're relying on.
kbenson
2 days ago
I don't feel like this is responsive to what I said when taken as a whole. I obviously wasn't just advocating for something because it's faster.
...but, if your goal is really verifiable state, why don't you just use image mode linux with bootc, since it provides much of what you're looking for, except to a higher level. All your arguments for NixOS so far apply to that as well, but there's even more immutability and verifiability.
nekooooo
2 days ago
this is from 1997. much easier to do this in 2026.
drybjed
3 days ago
Tell that to MinIO users, lol.