BorgBackup user here and really happy. It was a set and forget for me and after 7 years, the deduplicated backup is still working flawlessly each week. I recommend pairing it with borgmatic [1], which helps to design away some of the complexities of the underlying borg backup.
[1]: https://github.com/borgmatic-collective/borgmatic
I love vorta, I have it on all my machines. It's great for my (non-technical) husband who doesn't want to know anything about Linux.
Then everything gets backed up to my local server which then syncs out to remote storage. It's great.
Can't wait for Borg 2 to hit stable. The transfer command solves so many problems
My problem is I learn some tool like this, set it, and then indeed forget it. Then I avoid testing my backups because of the work it takes to un-forget it. Because of this, I'm leaning more and more towards rsync or tools that have GUI frontends.
Rather than avoid tools that work well, I would encourage you to adopt solutions that solve your use cases. For instance, if you aren't getting notifications that a backup is running, completing or failing, then all you've set up is a backup job, and not built a BDR process. If you're looking for a tool to solve your entire BDR plan, then you're looking at a commercial solution that bakes in automating restore testing, so on and so forth.
Not considering all the aspects of a BDR process is what leads to this problem. Not the tool.
At a minimum you need backup, regular restore tests, and alerts when backups stop or restore tests fail.
Personally I automate restore testing with cron. I have a script that picks two random files from the filesystem: an old one (which should be in long term storage) and a new one (should be in the most recent backup run, more or less), and tries restoring them both and comparing md5sums to the live file. I like this for two reasons: 1. it's easy to alert when a cronjob fails, and 2. I always have a handy working snippet for restoring from backups when I inevitably forget how to use the tooling.
IMO alerting is the trickiest part of the whole setup. I've never really gotten that down on my own.
> set and forget for me and after 7 years
Please tell me you verify your backups now and then?
Borgmatic runs consistency checks [1] once a month on all repositories and archives and I occasionally retrieve older versions for selected files (archives with --verify-data only once a year or whenever I feel the need - there's 9TB of data in the borg repo, which takes a bit to scan). Note though that borg is not my main backup, it is the fallback "3" in the 3-2-1 principle, where my primary data is a ZFS Raidz2 and my primary backup is an offsite ZFS Raidz2 in pull mode. I added borg because I did not want to rely on a single software (ZFS), although this fear was unstained so far.
[1]: https://borgbackup.readthedocs.io/en/stable/usage/check.html
This always gets repeated, sounds good and makes sense theoretically but in reality there's no good way to do that and it should be the job of a computer to that.
Restoring one file from the backup, works but what if something else is corrupted?
Restoring the system from the image, works but what if some directory is not in the backup and you don't see that while testing?
Does someone know a good Android client?
Currently I'm just using bare rclone to backup to my own remote machines, but obviously this isn't very professional solution. Was thinking to add Backblaze B2 as a remote, but I guess using rclone wouldn't be a state-of-the-art solution here. After all, it isn't really a backup tool, is it? It has some built-in encryption, but it's a bit clunky, and I'd think a proper backup tool should automatically divide data into blocks of suitable size (instead of just creating file-per-file - to make it S3/B2 API-friendly), encode whole directories as tar (if needed to preserve links, for example), do deduplication, and whatever else are best practices I have no idea about, but which backup-proficient people probably invented long time ago.
Does anybody have a recommendation?
I briefly looked at restic and duplicati, but surprisingly none are as simple to use as I'd expect a dedicated backup-tool to be (I don't need, and kindda don't want GUI, I'd like all configuration to be stored in a single config-file I can just back-up to a different location like everything else, and re-create on any new machine). More than that, I've read some scary stories about these tools fucking up their indexes so that data turns out to be non-restorable, which sounds insane, since this is something you must be absolutely sure your backup-tool would never do no matter what, because what's even the point of making backups then.
>I'd like all configuration to be stored in a single config-file I can just back-up to a different location like everything else, and re-create on any new machine
You might want to look into kopia. It accomplishes the same task as restic, but handles configs in a way you might find more appealing. Further reading: https://news.ycombinator.com/item?id=34154052
Don't even bother with duplicati. I've tried to make it work so many times, but it's just a buggy mess that always fails. It's a shame too, because I really like the interface.
I've been using bupstash since trying to do backups on an rpi and finding Borg too slow to be usable. Since then I upgraded to a proper server at home but kept bupstash as I found it to just work better for the most part.
Keep in mind there's not been much progress since the last release two years ago and its still tagged as beta by the author. Tbf I think he has a higher quality standard than in other projects that are not tagged as such.
Useful backup tool comparison: https://github.com/deajan/backup-bench
Whether something is simple or not I'd say depends on the use case. But I found borg to be great. I'd recommend you check it out and go through the quickstart guide in the documentation. It does de-duplication and encryption. It does a lot more but you don't have to use those features if you don't need them. I couple it with borgmatic to implement a backup and disaster recovery procedure that is meant to decrease the risk of data loss. I also use borgbase and they have a good service but using something like B2 with this rclone support would result in a cheaper alternative if you don't need the extra that borgbase provides.
I've been using it for quite a while now both for my personal projects and paid work and have had a good experience with it.
restic + autorestic/resticprofile.
Borg 2 is still beta and Kopia is also there. But it's newer so I am testing it on another redundant backup on the same machine. I have space so why not?
Every once in a while I run integrity check (with data) so I can trust that metadata and data are fine.
I'm very happy with Restic backing up to BackBlaze B2.
I have a "config file", which is really just a shell script to set up the environment (repository location, etc), run the desired backups, and execute the appropriate prune command to implement my desired retention schedule.
I've been using this setup for years with great success. I've never had to do a full restore, but my experience restoring individual files and directories has been fine.
Do you have any links related to the index corruption issue? I've never encountered it, but obviously a sample size of one isn't very useful.
Writing an rclone backend for borg is something I have wanted to do for a long time.
However I found that the backends weren't well abstracted enough in v1 to make that easy.
However for v2 Thomas Waldmann has made a nice abstracted interface and the rclone code ended up being being only <300 lines of Python which only took an afternoon or two to make.
https://github.com/borgbackup/borgstore/blob/master/src/borg...
Thanks a lot for writing the rclone backend!
Oh very interesting. This has been a requested feature for a while especially with the rise in popularity and the decreased cost of object storage.
Borg working with object storage was not supported though some people did use it that way. From my understanding, most would duplicate a repo and upload instead of borg directly writing/manipulating it. This could problematic if the original repo was corrupt as now the corruption would be duplicated. So this will make things much easier and allow for a more streamlined workflow. Having the tool support rclone instead of specific services seems like a wise and more future-proof choice to me.
Does anyone have an up-to-date comparison of Borg vs Restic? Or a compelling reason to switch from Restic to Borg?
I've previously used Borg, but the inability to use anything other than local files or ssh as a backend became a problem for me. I switched to Restic around the time it gained compression support. So for my use-case of backing up various servers to an S3-compatible storage provider, Restic and Borg now seem to be equivalent.
Obviously I don't want to fix what isn't broken, but I'd also like to know what I'm missing out on by using Restic instead of Borg.
I prefer restic simply because I find it easier to understand and use. This means backups actually happen. It also feels less like it is constantly changing. Constant stream of new features isn’t a thing I’ve ever desired in a backup solution.
+1, I'm in a similar situation and be curious too about an up-to-date comparison.
Comparisons might be interesting, but one needs to be aware that they would be a bit apples to oranges:
- unreleased code that is still in heavy development (borg2, especially the new repository code inside borg2).
- released code (restic) that has practically proven "cloud support" since quite a while.
borg2 is using rclone for the cloud backend, so that part is at least quite proven, but the layers above that in borg2 are all quite fresh and not much optimized / debugged yet.
If you're looking for cheap online storage for your backups know this: A Microsoft 365 Single subscription comes with 1 TB of OneDrive space (Family subscriptions with 1 TB per person).
I've been using it with restic + rclone successfully for years. It's not very fast, but works.
I'd recommend having a look at Hetzner's "storage box" products. It's hard to beat 4€ a month for 1TB of SSH accessible storage.
6x1TB for 10€ with 30 day cryptolocker protection comes close.
[deleted]
For personal use, at what point would one recommend using Borg over a regular rsync?
I currently use rsync to backup up a set of directories on a drive to another drive and a remote service (rsync.net). It's been working great, but I'm not sure if my use-case is just simple enough where this is a good solution, or if I'm missing a big benefit of Borg. I do envy Borg's encryption, but the complexity of a new tool tied with the paranoia of me maybe screwing up all my data has had me on edge a bit to make the leap. I don't have a ton of data to backup, say about 5TB at the moment.
For me, the deduping and compression saves a lot of storage. My mail backup (17 backups covering the last 6 months) is originally 837GB, compressed to 312GB and dedupe'd to 19GB. Same with Postgres - 25GB to 7GB to 900MB.
You could probably use rsync's hard linking to save space on the mail backup but I'm not sure you'd get it as small without faffing about.
Usual problem, if you delete/corrupt a file and find out two days later, your daily backup is not going to help you. Having more than one snapshot is very valuable.
http://www.taobackup.com/ etc
Rsync is also very slow with lots of files, and doesn't deal with renamed files (will transfer again).
With rsync, you’re replicating only the last state.
With borg, you can see all backups being made and rollback to any previous snapshot. This is true of a lot of backup solutions btw.
Concretely, if you inadvertently delete a file and this get rsynced, you cannot use the backup to restore that file. With borg you can.
Rclone crypt is not much related to Borg. That’s a tool for copying files from one machine to another, in this case encrypting before copying. That’s rsync, working with cloud.
Borg is a different tool, for backup. It deduplicates, encrypts, snapshots, checksums, compresses, … source directories into a single repository. It doesn’t work with files, rather blocks of data. It includes commands for repository management, like searching data, pruning or merging snapshots, etc. You will then transfer or sync the repository to wherever you want, with a tool such as rsync/SSH or rclone. Rclone is now natively supported, so that you don’t need to store the repository locally and on remote, rather back up directly to remote.
I would also wonder what the difference between this package and Restic is. as far as efficiency and encryption.
How good at deduping is this when encryption is enabled? I was looking at rsync.net and it killed me that they don't support encryption in a sane way.
I've happily been writing borg backups to rsync.net for years. They have support for forcing borg in the ssh session using force-command, and borg has options that can prevent deletion (should the backup ssh key be compromised).
Overall it's a robust solution that isn't too painful to setup.
I second this. I was looking for a solution that prevented a compromised host from deleting its own backups. Forcing the command as you mentioned works for rsync.net, and its snapshots also provide a protection against fat finger errors.
Also BorgBackup user here: I'm running it on a Raspberry Pi to backup important documents to a Hetzner storage box via ssh. The Pi als runs OpenMediaVault to provide a SMB share on my home LAN. So whenever I scan a new document, just put it on the SMB share and from there it's backed up automatically every day.
I've always been doing "two-pass" backups to achieve "3-2-1" goal: first pass is to run BorgBackup to backup devices to my home server. The second pass is to use rclone to transfer the repos on home server to an object storage service (B2).
With rclone support built-in, the setup would be much easier.
I think this is heavily discouraged? Instead you should have multiple separate borg repos to minimize risk of misconfiguration and data corruption.
I'll use Restic or BorgBackup for servers, but ended up going with Kopia for machines that are not always on, like laptops. It has the advantage that it will take something of an opportunistic approach where it will start backing up if it hasn't done so in a while, and seems to be able to restart with aplomb if it gets interrupted (machine shutdown or laptop lid closed).
That and being able to have multiple machines writing to a shared repository at the same time is handy. I have the kids' Windows computers both backing up to the same repo to save a bit of storage. (Now if only Kopia supported VSS on Windows without mucking around with dubious scripts.)
Vorta (GUI for BorgBackup) will also opportunistically backup if it's overdue for a backup.
I used to use Borg but the fact that it can't work with a dumb storage device like SMB meant I eventually moved to Rustic, which is even better:
https://github.com/rustic-rs/rustic
I see they have gotten support for S3 (and other storage providers) via OpenDal. Might need to revisit rustic for my backup needs then! I once started looking at what it would take to build a GUI using Tauri (Rust backend <-> JS/Web frontend), but didn't have time to figure out the APIs.
What I really like about Rustic is that it understands .gitignore natively so you can backup your entire workspace without dragging a lot of dependencies, compiled binaries, and other unnecessary data with you into your backups.
It wasn't recommended a bit ago to use Borg 2.0 because it wasn't baked enough. Has that changed? Are people using Borg 2?
It is still in beta, and has been in this state for a long time. At some point, the developer thought to delay it further and introduce whatever breaking changes are needed in this release.
Note that if you use, say, the 2.11 version, you cannot upgrade to 2.12, you cannot go back to 1.X either. People like me were stuck, it turned out you have to discard the repo. Sometime later they better clarified this point:
>> Borg2 is currently in beta testing and might get major and/or breaking changes between beta releases (and there is no beta to next-beta upgrade code, so you will have to delete and re-create repos).
I have a 2.X repo. It’s working fine and backs up. I have a lot of snapshots in that repo. If someone knows how to transfer them to a 2.X version once it’s out of beta, let me know.
Author calls that out right at the top of the changelog:
> Beta releases are only for testing on NEW repos - do not use for production.
It's a joy that the OSS world has so many active and really good backup tooling projects like Borg, restic and all the fancy wrapper/GUI tools. I use many of them in different environments for customer setups, desktops and my own cloud setup. It's essential to have several different options and each project has its own USP. A big thanks to everyone involved!
As you seem familiar with the landscape, do you care to share what you think the strengths/USP and weaknesses of each option you are familiar with are? Or do you know of a high quality blog post somewhere that does?
The lack of s3-like remotes support was the reason I switched from borg to restic
rclone has dedupe, I think it does what restic can do plus multiple cloud support.
rclone crypt also does encryption.
so far I think rclone has it all for me.
So, should I plan to switch to this rather than keep using Kopia?
I was using it for years on an external drive, but then I got a NAS, and did not want to fuss with community packages to get Borg working.
Kopia works fine, aside from the confusing GUI setup process, but it seems to be the least popular up and coming option.
Now it seems that this can directly target SFTP? I wonder what that means for the future of Kopia.
Kopia also supported rclone for a long time, though: https://kopia.io/docs/reference/command-line/common/reposito... . However, in my experience backing up over sftp with kopia can be very slow. I suspect it's unable to use parallel sessions for them (or pipelining, but rclone API probably doesn't do that).
My reason to go with kopia was that previously you were not able to backup multiple hosts into the same repo without great inefficiencies. I'm not sure if they still have resolved that. Another was its native S3 support which I use with ceph.
A perhaps more superficial personal reason is that at least Go is a statically typed language, even if its type system isn't that great..
I quite like Borg, others worth checking out are Restic and Kopia.
Restic didnt have a UI for a long time, not sure thats changed...
Nope. No configuration file either. But they added compression recently.
autorestic or resticprofile fill the gap well. Backrest does UI.
New to Borg, and backups in general.
Does borg have the ability to split chunks over multiple repository of varying sizes? For example, I might have just 15GB Google Drive Storage, whereas on others I might have 100GB available.
Used BorgBackup 1.x with rclone, backing up in a local repository and then sending it to S3 with Rclone. For the way that borg works with files, most of the historic data lies on old, untouched files in the repository, only a catalog and the new blocks create new and update files. So it was great for using some of the S3 tiers that move files that doesn't change for some period automatically to a cheaper class.
Having both together makes easier to get this kind of use case.
Too late Borg! I specifically chose Restic for the Rclone support. Can't change backup strategy now. It also save me once.
Pcloud lifetime + Restic, all in one repo, to benefit from dedup.
That's really big and really nice. For anyone unfamiliar, rclone is to online storage what ffmpeg is to multimedia files: a Swiss army knife that adapts anything to anything. It supports everything from S3 and Azure storage, to Google Drive and Dropbox, to sftp mounts; encryption and compression layers too.
I previously used Borg and didn’t have any bad experiences, but in the end, just switched my server over to zfs. Simple, straight-forward backup tools with native snapshots, and bit-rot resistance.
How can we use it for large database backups > 500 GB and is anyone doing that on daily basis?
Please note that this is rather recent "bleeding edge" code from master branch.
It is available as 2.0.0beta11, but not suitable for production yet.
"Beta" also means that there won't be repository migration code from beta N to N+1.
Interesting. How robust is it in practice?
I've been using Duplicacy for a long while, and I've been pretty happy with it. But I'd love to switch to a full open-source solution (Duplicacy is proprietary with sources publically available).
I remember reading quite a few years ago about people working to get Borg to work with windows. Has there been any recent progress with that?
can anybody familiar with the tool comment on if the encryption codein borgbackup is audited or is there a tool whose encryption portion of the code is audited to ensure that there are no glaring bugs in the encryption scheme?
It's kind of ridiculous that there is better tooling for Kubernetes to sync files two ways than there is for the Linux desktop. Rclone is a maze of options, which vary based on version/distro. The configurator is a slow readline console script without enough information. The one decent GUI for Rclone has been abandoned, and despite being able to save "tasks", had no ability to just... schedule one every 10 minutes. And yet if you go into most distros and look at packaged Internet apps, or things on Flatpak, you will find 1,000 different open source GUIs for an RSS reader, BitTorrent client, or chat client.
I would love it if there were some kind of "Linux Desktop co-op", with a couple of staff. Users pay membership dues, vote on apps/features, and some devs get paid to develop it, in addition to "resume fame" that can translate over to a higher paying gig. But something tells me the Linux Desktop is so small and nerd-focused that we'd just end up funding more RSS readers and chat clients.