Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
[1] https://github.com/openzfs/zfs/issues/11353
[2] https://www.rsync.net/products/zfsintro.html
> Not a great time price-wise to be building a NAS
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.
AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
But we now have trillionaires, aren't you happy about that? It means you might be the one some day too /s
You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
> You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
ZFS without ECC is no more risky than any other file system / software RAID without ECC.
As no one owes you an explanation, it would take you five seconds to Google this and discover:
1. It's been disproven, with one of the original ZFS developers chiming in.
2. The original source of the rumor was a forum post that somehow became canon.
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
I've been thinking about building my own NAS as well. Mind sharing how much did you pay for those hard drives, and what motherboard did you choose?
https://diskprices.com/ is great for this
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.
> 4x the 14 TB WD Elements Desktop.
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
--
(*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall.
I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.
I found cockpit to be incredibly useful for managing NAS and VPS. Even though any configuration is not an issue with LLMs, having a dashboard with toggles, formatted logs and such is helpful, and I can even run shell commands on phone without ssh. In the NAS case, you can easily see the results of backup/maintenance cronjobs, manage samba (which is a huge pain in the ass to get to work properly on iPhone), and monitor disk usage
Same, using NixOS as a NAS, though it kept growing and now I'm trying to share it with other people. I use BTRFS on MDADM though. It's ended up being an all-in-one home server and router now because I was tired of the noisy power hungry rack I had in my closet.
https://HomeFree.host
Just yesterday I was brainstorming with ChatGPT about this. I have an ancient QNAP plus a slightly less ancient NUC running PiHole, Wireguard and other services. Both need replaced, so why not combine them?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
It also surprised me that the author said "4 Cores, Xeon Server CPU can be had for cheap".
But the specs also said ECC RAM and I don't think the N100 supports that.
I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
Is snooty a autocorrect for Synology or some other product?
That is kind of exactly how zfs works though. The guide isn't complete, sure, but "rebuilding" the array is just replace the disk and run a single zfs command.
ZFS makes this completely trivial except for the "beeps and lights a led" part
To be fair, the "beeps and lights a led" part is very important in case of a dead disk in an array. The only time, I disrupted a service in production for longer than a few hours was when I didn't get the email of a degraded raid 1 array and the second disk died a few weeks later.
smartmontools makes this trivial and can email you which might be better than beeping and lights if you’re not always near your server.
I use that plus smartctl_exporter so I can do metrics and alerting off that.
Also you can use zfs on a proxmox host and use that both for NAS duty and for the VMs.
For some reason people insist on doing truenas on top of proxmox and then introducing a networking layer between everything they do. Noooo…
It's pretty minimal and straightforward written in Haskell. Probably can write one from scratch with Claude now.
One alternative for those who don't want any of the major NAS vendors, just use RHEL10. It's free up to 16 licenses, it's ultra stable, cockpit is a very mature gui for a lot of maintenance tasks.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
The problem with RHEL is that the free version exists at the sufferance of IBM, a company not well known for being motivated by keeping tech enthusiasts happy. I would use Debian, personally. Not as long of a support cycle as RHEL, but still quite stable and no possibility of corporate rug-pulls.
I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
Tangential, but about this:
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
Is this still true with ZFS RAIDZ1?
It's complicated, but https://jro.io/capacity/ actually walks you though how it works, towards the middle of the page.
```
Unlike traditional RAID5 and RAID6 implementations, ZFS supports partial-stripe writes. This has a number of important advantages but also presents some implications for space calculation that we'll need to consider. Supporting partial stripe writes means that in our 7wZ2 vdev example, we can support a write of 12 total sectors even though 12 is not an even multiple of our stripe width (7). 12 is evenly divisible by +1 (3 in this case), so we don't even need any padding. We would have a single full stripe of 7 sectors (2 parity sectors plus 5 data sectors) followed by a partial stripe with 2 parity sectors and 3 data sectors. This will be important because even though we can support partial stripe writes, every stripe (including those partial stripes) need a full set of p parity sectors.
```
> and while this made sense when drives were expensive
I don't have the answer to the latency question, but HDDs have shot up in $/TB over the last couple years too. They are once again kind of expensive.
The RAID5 write hole is not present in RAIDZ1 I believe.
RAIDZ1 will add more stress to the consumer ssd's as to enterprise one due to surprinsingly different hardware. Consumer ssd's life times are calculated on a 8 hour shift (the tbw) and enterprise are calculated on 24/7.
Please - dont - use - consumer ssd's - with zfs raidz1
To this day I still use a ZFS array for my critical backup storage. I've migrated over to using snap raid + mergerfs for my larger Linux iso storage array. Simple enough and I can pull a drive on it's own without any other stuff.
Can anyone recommend a good server for a homelab to use for a storage purpose like this?
I'm sure there's better options now but the HP ProLiant MicroServers (used).
They support ECC ram, 4 caddies, one extra PCIe slot, and to my knowledge you're not cpu limited for a zfs file server usecase.
Keep in mind though, all you need is linux* support, iDRAC, ECC if you're a snob, and drive bays ... and that's basically any free server.
In my extremely opinionated opinion I would only get used enterprise server gear, because a zfs file server will just work unless hardware fails. And a UPS.
*ZFS will be a more natural choice on FreeBSD. It's better documented, and will meet Linux 1:1 in hardware compatibility for this.
I'm still running an old Gen 8 MicroServer. Modern drives can actually saturate the SATA controller, and because it only has a single PCIe slot I can't add both a 10Gb NIC and a storage controller - I went with the 10Gb NIC.
It works well enough though and has lasted me over a decade at this point. 16GB DDR3 ECC, an old 4 core/8 thread Xeon, 4x14TB drives and the Mellanox NIC.
Not strictly a recommendation, but Terramaster is a good brand to look at if you want Synology-shaped hardware which can run TrueNAS or Proxmox or any flavour of Linux you want.
Along with various other devices (including a large Synology which I wouldn’t buy today), I run Proxmox on a small two bay+two nvme Terramaster. I have a bare bones Ubuntu LXC running Samba configured for Apple Time Machine, an VM running Scrypyed, and PBS for Proxmox backups. Nothing on it is critical so I don’t bother with any storage redundancy.
I have the F8 Plus, great little unit. It did need a BIOS update when I first got it to enable Proxmox/other OSes to work properly.
Recently replaced the internal USB boot drive with a small NVMe USB enclosure; using a 90-degree USB connector and using a dremel to sand away an opening for the cable to come out so I could mount the enclosure externally.
It's a horrible idea likely, but I have an ancient old Dell PowerEdge R510. Probably sucks way too much energy, but it does what it does and the price of SSDs have skyrocketed so I'm not touching it.
this is really cool. I've been dealing with an aging Synology nas and this is something I can pick up, evaluate performance and how safe it is to serve as home for my data.
Yeah no. When it comes to backups and data storage, I would rather use a proven reliable system that’s been used and tested by millions of other users, keep these hacky stuff for your hyperland set up.
I've lost more data to "proven reliable systems" than to my homegrown hacked-up stuff (which may be because I trust the latter much less).
> I would rather use a proven reliable system that’s been used and tested by millions of other users
You're describing ZFS.
Not really, zfs is part of it, but not the whole picture, you as a user you need more than just a file system.
Truenas for example add replication, external cloud sync, gui (yes it’s important not to wipe your data with wrong command), HA, custom caching, containers to extend ot with other apps, raid-z expansion with openzfs, and other for monitoring, user management and a lot more.
Other solutions also add more than just a zfs with ssh access.
Stopped reading at "Debian".
Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
ZFS is a permanent second-class citizen on Linux (due to usual open-source politics). This will never resolve.
I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
FreeBSD ZFS support has matured and is outstanding. Quality-wise it has reached parity with Illumos.
If you can afford Solaris then you're probably not building your own NAS from parts of lesser computers.
Run ZFS backed filestore on FreeBSD, have migrated it to/from Debian. At work and home, not petabyte scale but certainly multi hundred terrabyte. Over 15 years, on over 50 hosts/NAS/SAN instances, different hardware.
Run ZFS on Raspberry Pi, on home builds, on Intel, on AMD, on other ARM chipsets.
I think you're over-stating things. Debian is fine for this. I do think FreeBSD is a better platform for myself.
The code bases adhere (modulo ZFS version numbers) to a spec and you can safely migrate the pools between OS. I've done it multiple times both directions.
You can not do this with BTRFS and other Linux things, I consider this feature of (Open)ZFS a killer-context for me: It's OS portable. I wish Mac OSX hadn't walked out of the room when Oracle went legal.
> You can not do this with BTRFS
There is actually a btrfs driver for Windows [0]. I've used it a few times before, and it works surprisingly well. You probably wouldn't want to use it for any serious work, but that's not because it's technically flawed, but more because it isn't extensively tested or commercially-supported.
[0]: https://github.com/maharmstone/btrfs
There is Fuse support in BSD. I don't consider that a good choice for this role.
Sure you can migrate pools.
Yet everyone is (again) lost in the details and missing the big picture, which is Linux is doing its best to rat fuck OpenZFS at every opportunity, the last of which was the elimination of write_cache_pages in 6.18 behind the GPL iron curtain a mere few months ago.
I don't know about you but I don't want to build my file storage atop hacks on top of more hacks. The kernel has made it clear non-GPL code is not welcome. Struggles will continue in perpetuity. There are better options.
I run ZFS on Linux and FreeBSD and FreeBSD is less of a faff. If you don’t need docker on your NAS, I would go FreeBSD as well.
If you run ZFS with an LTS kernel you're pretty much fine. Yes new Linux releases will break existing ZFS releases - but the LTS tree is in support for long enough that this is never an issue.
Unless you need driver support only found in newer kernels. Then you are screwed.
Do I want my hardware to work or do I want to be able to read my files?
I am a BSD user yet I think this is largely irrelevant in the scope of a NAS which is the conversation we have here. Even in the cases bleeding edge hardware support is relevant, the latest Linux LTS kernel has probably wider hardware support than the freebsd-current at any given point in time anyway.
FreeBSD is a great choice, but there is no need to invent silly reasons to justify using it.
> Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
> ZFS is a permanent second-class citizen on Linux
Linux is the primary target of OpenZFS [0] [1], and has been since 2020 [2]. It may not be supported by the Linux kernel developers, but it's supported by the ZFS developers, and that's all that really matters.
> I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
Sure, it's an out-of-tree module, but that doesn't mean that it will randomly break all the time; it just means that you may occasionally need to wait for a new OpenZFS release before upgrading your kernel.
> FreeBSD ZFS support has matured and is outstanding.
Agreed, but Linux and FreeBSD both use the same ZFS [3], so I don't really see how the ZFS in FreeBSD can be better than the one in Linux. The tooling and install procedure is certainly better on FreeBSD, but the actual filesystem code is the same (and is probably slightly more robust on Linux since that's going to be where most of the testing occurs).
[0]: https://github.com/openzfs/zfs#supported-kernels-and-distrib...
[1]: https://github.com/openzfs/zfs/pull/8987
[2]: https://github.com/openzfs/zfs/releases/tag/zfs-2.0.0
[3]: https://docs.freebsd.org/en/books/handbook/zfs/
And yet TrueNAS is moving away from BSD towards … Debian!
AFAIK this is because TrueNAS is trying to be all things to everyone, and they like Docker more than BSD jails.
You can use past tense there.