We’re migrating many of our servers from Linux to the BSDs

59 pointsposted 9 hours ago
by msangi

9 Comments

riiii

7 minutes ago

Finally someone dares to speak up. This little git from Finland with his egotistical named solo project is finally exposed.

Borg3

5 hours ago

I wish he could write up a bit about XFS failure he had. Im using it from many many years and there is no issues at all.

blipvert

4 hours ago

Used it since the late 90s on IRIX, think there were a few issues early on with the endian swap, but no issues for the best part of twenty years for me!

lotharcable

an hour ago

It hard to know the timeline with his data loss, but I am assuming it was a long time ago.

XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over.

For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that if the machine had a crash or sudden power loss the hardware raid would live on long enough to finish its writes. SGI had tight control over the type of hardware you could use and it was usually good quality stuff.

In the land of commodity PC-based servers this isn't often how it worked. Instead you just had regular IDE or SATA hard drives. And those drives lied.

On cheap hardware the firmware would report back it had finished writes when in fact it didn't because it made it seem faster in benchmarks. And consumers/enterprise types looking to save money with Linux mostly bought whatever is the cheapest and fastest looking on benchmarks.

So that if there was a hardware failure or sudden power loss there would could be several megs of writes that were still in flight when the file system thought they were safely written to disk.

That meant there was a distinct chance of dataloss when it came to using Linux and XFS early on.

I experienced problems like that in early 2000s era Linux XFS.

This was always a big benefit to sticking with Ext4. It is kinda dumb luck that Ext4 is as fast as it is when it came to hosting databases, but the real reason to use it is because it had a lot of robust recovery tools. It was designed from the ground up with the assumption that you were using the cheapest crappiest hardware you can buy (Personal PCs).

However modern XFS is a completely different beast. It has been rewritten extensively and improved massively over what was originally ported over from SGI.

It is different enough that a guy's experience with it from 2005 or 2010 isn't really meaningful.

I have zero real technical knowledge on file systems except as a end-user, but from what I understand FreeBSD uses UFS that uses a "WAL" or "write ahead log".. where it records writes it is going to do before it does it. I think this is a simpler but more robust solution then the sort of journalling that XFS or Ext4 uses. The trade off is lower performance.

As far as ZFS vs Btrfs... I really like to avoid Btrfs as much as possible. A number of distros use it by default (OpenSUSE, Fedora, etc), but I just format everything as a single partition as Ext4 or XFS on personal stuff. I use it on my personal file server, but it really simple setup with UPS. I don't use ZFS, but I strongly suspect that btrfs simply failed to rise to its level.

One of the reasons Linux persists despite not having something up to the level of ZFS is that most of ZFS features are redundant to larger enterprise customers.

They typically use expensive SAN or more advanced NAS that has proprietary storage solutions that provide ZFS-like features long before ZFS was a thing. So throwing something as complicated as ZFS on top of that really provides no benefit.

Or they use one of Linux clustered file system solutions, of which there is a wide selection.

hggigg

3 hours ago

I had one a few years back where we ran out of inodes on a Jenkins machine on CentOS 7 and it crashed and couldn’t remount the filesystem. I had to restore a backup which was time consuming on a 4TB volume with crazy amounts of files.

cyberax

32 minutes ago

> As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend.

Yeah. That guy should not be allowed anywhere near the production workloads. "I solve problems", my ass.

codezero

15 minutes ago

The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interruption of service so there was no need to contact or warn the customer.

vfclists

4 hours ago

> “If nothing is working, what am I paying you for? If everything’s working, what am I paying you for?”

Bloke is not acquainted with Keynesian economics.

https://www.youtube.com/watch?v=9OhIdDNtSv0

https://www.youtube.com/watch?v=NO_tTnpof_o

All a man needs is food in his stomach and a place to rest at the end of the day. Everything else is vanity

What proportion of global GDP is dedicated to fulfilling our basic material needs?

It is mostly unnecessary. Inspite of the huge productivity gains made since the seventies, the current generation of young Americans are poorer than their parents and grandparents were at their age.

So what does all the IT optimization bring? Just more wealth for the owners and redundancies for their employees, including Joe Bloggs here.

It is time people in IT got to understand this. In the long term their activities are not going to improve their wealth. They are one of the few professions whose job is to optimize themselves out of a living, unless they own the means of the production their are optimizing, which they don't.

It is their employers that do.

froh

6 hours ago

the title of the talk is "Why (and how) we’re migrating many of our servers from Linux to the BSDs"

and that should be the title of this post too.

I like that the blog post shares the slides, not just the video.