We (small HPC system) just upgraded our OS from RHEL 7 to RHEL 9. Most user apps are dynamically linked, too.

You don't want to believe how many old binaries broke. Lot of ABI upgrades like libpng, ncurses, heck even stuff like readline and libtiff all changed just enough for linker errors to occur.

Ironically all the statically compiled stuff was fine. Some small things like you mention only linking to glibc and X11 was fine too. Funnily enough grabbing some old .so files from the RHEL 7 install and dumping them into LD_LIBRARY_PATH also worked better than expected.

But yeah, now that I'm writing this out, glibc was never the problem in terms of forwards compatibility. Now running stuff compiled on modern Ubuntu or RHEL 10 on the older OS, now that's a whole different story...

The problem of modern libc (newer than ~2004, I have no idea what that 1996 one is doing) isn't that old software stops working. It's that you can't compile software on your up to date desktop and have it run on your "security updates only" server. Or your clients "couple of years out of date" computers.

And that doesn't require using newer functionality.

> The trick? It's not statically linked, but dynamically linked. And it doesn't like with anything other than glibc, X11 ... and bdb.

How would that work given that glibc has gone through a soname change since then? If it's from 1996 are you sure the secret isn't that it uses non-g libc?

can you write up a blog of how this is working? because both as a publisher and a user, broken binaries are much more the norm

Well, you could just look at things from an interoperability and standards viewpoint.

Lots of tech companies and organizations have created artificial barriers to entry.

For example, most people own a computer (their phone) that they cannot control. It will play media under the control of other organizations.

The whole top-to-bottom infrastructure of DRM was put into place by hollywood, and then is used by every other program to control/restrict what people do.

That was the point of the successive questions in the second paragraph...

existential crisis: so hot right now

And the cycle continues

"... except for the problem of too many levels of indirection."

ad infinitum ;-)

I've got it working. Used Mongoose. Unfortunately Actually Portable Executables seem to not play well with WSL, and the suggested fixes didn't work. I'm able to play with it in a VM. Not as portable as I'd hoped, but I'll see how it goes.

IMO one of the best features of AppImage is that it makes it easy to extract without needing external tools. It's usually pretty easy for me to look at an AppImage and write a PKGBUILD to make a native Arch package; the format already encodes what things need to be installed where, so it's only a question of whether the libraries it contains are the same versions of what I can pull in as dependencies (either from the main repos or the AUR). If they are, my job is basically already done, and if they aren't, I can either choose to include them in the package itself assuming I don't have anything conflicting (which is fine for local use even if it's not something that's usually tolerated when publishing a package) or stick with using the AppImage.

> It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods

'Noticeably slower' at what? I've run, e.g. xemu (original xbox emulator) as both manually built from source and via AppImage-based released and i never noticed any difference in performance. Same with other AppImage-based apps i've been using.

Do you refer to launching the app or something like that? TBH i cannot think of any other way an AppImage would be "slower".

Also from my experience, applications released using AppImages has been the most consistent by far at "just working" on my distro.

We fix this issue by distributing ours in a tar file with the executable bit set. Linux novices can just double click on the tar to exact it and double click again on the actual appimage.

Been doing it this way for years now, so it's well battle tested.

AppImage is not what you need. It's just an executable wrapper for the archive. To make the software cross-distro, you need to compile it manually on an old distro with old glibc, make sure all the dependencies are there, and so on.

https://docs.appimage.org/reference/best-practices.html#bina...

There are several automation tools to make AppImages, but they won't magically allow you to compile on the latest Fedora and expect your executable to work on Debian Stable. It's still require quite a lot of manual labor.

Don't forget - AppImage won't work if you package something with glibc, but run on musl/uclibc.

>I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

It won't work: drivers usually require exact (or more-or-less the same) kernel module version. That's why you need to explicitly exclude graphics libraries from being packaged into AppImage. This make it non-runnable on musl if you're trying to run it on glibc.

https://github.com/Zaraka/pkg2appimage/blob/master/excludeli...

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

Typically appimage packaging excludes the .so files that are expected to be provided by the base distro.

Any .so from nvidia is supposed to be one of those things. Because it also depends on the drivers etc.. provided by nvidia.

Also on a side note, a lot of .so files also depends on other files in /usr/share , /etc etc...

I recommend using an AppImage only for the happy path application frameworks they support (eg. Qt, Electron etc...). Otherwise you'd have to manually verify all the libraries you're bundling will work on your user's distros.

Depends on the license and the specific piece of software. Redistribution of commercial software is may be restricted or require explicit approval.

You generally still also have to abide by license obligations for OSS too, e. G., GPL.

To be specific for the exampls, Nvidia has historically been quite restrictive (only on approval) here. Firmware has only recently been opened up a bit and drivers continue to be an issue iirc.

What if the library you use calls dlopen later? That’ll fail.

There is no universal, working way to do it. Only some hacks which work in some special cases.

No you can't. dlopen signature takes a file path, not a memory range. And if you start to save the libraries to the filesystem before opening them, there's no difference to shipping an archive directly and skip the trouble of your own archive code.

Yes, that's true.

But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work.

These are just strange and confusing from the end users' perspective.

> You're supposed to do this recursively

ldd works recursively.

> Not to mention all the files from /usr/share

Well yeah, there obviously cannot be a generic way to enumerate all the files a program might open...

This is the theory, but not the practice.

In decades of using and managing many kinds of computers I have seen only a handful of dynamic libraries for whom security updates have been useful, e.g. OpenSSL.

On the other hands, I have seen countless problems caused by updates of dynamic libraries that have broken various applications, not only on Linux, but even on Windows and even for Microsoft products, such as Visual Studio.

I have also seen a lot of space and time wasted by the necessity of having installed in the same system, by using various hacks, a great number of versions of the same dynamic library, in order to satisfy the conflicting requirements of various applications. I have also seen systems bricked by a faulty update of glibc, if they did not have any statically-linked rescue programs.

On Windows such problems are much less frequent only because a great number of applications bundle with the them, in their own directory, the desired versions of various dynamic libraries, and Windows is happy to load those libraries. On UNIX derivatives, this usually does not work as the dynamic linker searches only standard places for libraries.

Therefore, in my opinion static linking should always be the default, especially for something like the standard C library. Dynamic linking shall be reserved for some very special libraries, where there are strong arguments that this should be beneficial, i.e. that there really exists a need to upgrade the library without upgrading the main executable.

Golang is probably an anomaly. C-based programs are rarely much bigger when statically linked than when dynamically linked. Only using "printf" is typically implemented in such a way that it links a lot into any statically-linked program, so the C standard libraries intended for embedded computers typically have some special lightweight "printf" versions, to avoid this overhead.

I've heard this many times, and while there might be data out there in support of it, I've never seen that, and my anecdotal experience is more complicated.

In the most security-forward roles I've worked in, the vast, vast majority of vulnerabilities identified in static binaries, Docker images, Flatpaks, Snaps, and VM appliance images fell into these categories:

1. The vendor of a given piece of software based their container image on an outdated version of e.g. Debian, and the vulnerabilities were coming from that, not the software I cared about. This seems like it supports your point, but consider: the overwhelming majority of these required a distro upgrade, rather than a point dependency upgrade of e.g. libcurl or whatnot, to patch the vulnerabilities. Countless times, I took a normal long-lived Debian test VM and tried to upgrade it to the patched version and then install whatever piece of software I was running in a docker image, and had the upgrade fail in some way (everything from the less-common "doesn't boot" to the very-common "software I wanted didn't have a distribution on its website for the very latest Debian yet, so I was back to hand-building it with all of the dependencies and accumulated cruft that entails").

2. Vulnerabilities that were unpatched or barely patched upstream (as in: a patch had merged but hadn't been baked into released artifacts yet--this applied equally to vulns in things I used directly, and vulns in their underlying OSes).

3. Massive quantities of vulnerabilities reported in "static" languages' standard libraries. Golang is particularly bad here, both because they habitually over-weight the severity of their CVEs and because most of the stdlib is packaged with each Golang binary (at least as far as SBOM scanners are concerned).

That puts me somewhat between a rock and a hard place. A dynamic-link-everything world with e.g. a "libgolang" versioned separately from apps would address the 3rd item in that list, but would make the 1st item worse. "Updates are far quicker and easier" is something of a fantasy in the realm of mainstream Linux distros (or copies of the userlands of those distros packaged into container images); it's certainly easier to mechanically perform an update of dependency components of a distro, but whether or not it actually works is another question.

And I'm not coming at this from a pro-container-all-the-things background. I was a Linux sysadmin long before all this stuff got popular, and it used to be a little easier to do patch cycles and point updates before container/immutable-image-of-userland systems established the convention of depending on extremely specific characteristics of a specific revision of a distro. But it was never truly easy, and isn't easy today.

Would be nice if there was a binary format where you could easily swap out static objects for updated ones

Steel-manning the idea, perhaps they would ship object files (.o/.a) and the apt-get equivalent would link the system? I believe this arrangement was common in the days before dynamic linking. You don't have to redownload everything, but you do have to relink everything.

Libraries already break their ABI so often that continuously rebuilding/relinking everything is inevitable.

Then you update those dependencies. Not very difficult with a package manager. And most dependencies aren't used by a ton of programs in a single system anyway. It is not a big deal in practice.

Honestly, that doesn't sound too bad if you have decent bandwidth.

Most open source software tooling Were designed to be dynamically linked. It is non-standard to statically link things together, which causes various random issues.

I'm guessing the pain of fighting the various build systems that insist on dynamic linking, sometimes against the user's explicit wishes.

Linking against an older glibc means setting up an older distribution and accepting all the outdated toolchains and libraries that come with it. Need to upgrade? Get ready to compile everything from source and possibly bootstrap a toolchain. I wouldn't call this trivial.

The fact that you need to use a container/chroot on Linux in the first place makes the process non trivial, when all you have to do on Windows is click a button or two.

> examples ... of the interface to the operating system being exposed differently than a library

Linux syscalls, MS-DOS 'software interrupts'...

But that's not the issue, operating system interfaces can be exposed via DLLs, those DLLs interfaces just must be guaranteed to be stable (like on Windows).

Tbh, I'm not sure why I can't simply tell the gcc linker some random old glibc version number from the late 1990s and the gcc linker checks whether I'm using any functions that haven't been available in that old version (and in that case errors out). That would be the most frictionless solution, and surely it can't be too hard to annotate glibc functions with a version number in the gcc system headers when that function first appeared.

But if I get to Bring My Own Dependencies, then I know the exact versions of all my dependencies. That makes testing and development faster because I don’t have to expend effort testing across many different possible platforms. And if development is just generally easier, then maybe it’s easier to react expediently to security notices and release updates as necessary.. .

I just remember Jonathan Blow mentioning this in one of his streams.

I believe you're thinking of the x86 Hotpatching hook[1], which doesn't exist on x86-64[2] (in the same form, it uses a x86-64 safe one).

[1] https://devblogs.microsoft.com/oldnewthing/20110921-00/?p=95...

[2] https://devblogs.microsoft.com/oldnewthing/20221109-00/?p=10...

Yeah while APE is a technically impressive trick, these issues far outweigh the minor convenience of having a single binary.

For most cases, a single Windows exe that targets the oldest version you want to support plus a single Glibc binary that dynamically links against the oldest version you want to support and so on is still the best option.

>> Linux

> if you configure binfmt_misc

I don't think that's a requirement, it'll just fall back to the shell script bootstrap without it.

It's his personal website lol.

In my experience it seems to be an issue caused by optimizations in legacy code that relied on dlopen to implement a plugin system, or help with startup, since you could lazy load said plugins on demand and start faster.

If you forego the requirement of a runtime plugin system, is there anything realistically preventing greenfield projects from just being fully statically linked, assuming their dependencies dont rely on dlopen ?

Ack. I went down that rabbit hole to "just" build a static Python: https://beza1e1.tuxen.de/python_bazel.html

The thing with static linking is that it enables aggressive dead code elimination (e.g. DLL are a hard optimization barrier).

Even with multiple processes sharing the same DLL I would be surprised if the alternative of those processes only containing the code they actually need would increase RAM usage dramatically, especially since most processes that run in the background on a typical Linux system wouldn't event even need to go through glibc but could talk directly to the syscall interface.

DLLs are fine as operating system interface as long as they are stable (e.g. Windows does it right, glibc doesn't). But apart from operating system interfaces and plugins, overusing dynamic linking just doesn't make a lot of sense (like on most Linux systems with their package managers).

I've been static linking my executables for years. The downside, that you might end up with an outdated library, is no match for the upsite: just take the binary and run it. As long as you're the only user of the system and the code is your own you're going to be just fine.

I don't think dynamic libraries fail at "utilizing" any available RAM.

you can patch musl, to initialize the brk state as -1 and then it won't use it.

It however shifts a lot of the complexity of building the application to the distro maintainer, or a software maintainer has to prioritize for which distribution they choose to build and maintain a package, because supporting them all is a nightmare and an ever shifting moving target. And it's not just a distribution problem, it's even a distribution version/release problem.

Look at the hoops you sometimes have to jump through or hacks you have to apply to make something work on Nix, just because there is no standardization or build processes assume library locations etc. And if you then raise an issue with the software maintainer - the response is often "but we don't support Nix". And if they're not Nix/Nixos users, can you blame them?

If you've ever had to compile a modern/recent software package for an old distro (I've had to do this for old RH distro's on servers which due to regulations could not be upgraded) - you're in a world of pain. And both distro and software maintainers will say "not my problem, we don't support this" - and I fully understand their stance on that, because it is far from straight forward, and only serves a limited audience.

There is however also the long tail of open source software that isn't packaged for your favorite distribution.

These days Linux binaries usually work fine, even older ones, and when they don't the reason is that they often don't get the same attention as their Windows counterparts.

Arch?...