nyanpasu64
3 days ago
I've done some digging in Linux power management a while ago, while debugging a (not-fully-fixed) Linux AMDGPU dGPU crash on low memory (https://gitlab.freedesktop.org/drm/amd/-/issues/2362). Along the way, I discovered that you can hibernate both through /sys/power/disk, and the userland snapshot/hibernate/suspend interface (https://docs.kernel.org/power/userland-swsusp.html, snapshot_ioctl()). IIRC these two mechanisms go along quite different codepaths internally.
The specific crash bug I encountered was because Linux calls pm_restrict_gfp_mask() to prevent swapping to disk, before dpm_prepare() (the first opportunity for a GPU driver to backup VRAM to system RAM before the PCIe GPU is shut down and VRAM is lost). So if you don't have enough free system RAM to hold all VRAM, the sleep is aborted midway through (waking the system) or produces a failed memory allocation later during sleep or resume (often resulting in undefined system state, halted network or USB controllers, or worst yet a halted NVMe controller resulting in the system running around like a headless chicken unable to load data from disk or even log data to the journal). I'm wondering if this was a deliberate decision or an unforeseen interaction between suspend-time GFP masks and GPU drivers.
It seems Nvidia can't reliably backup VRAM either without being informed by systemd prior to the kernel initiating suspend (https://download.nvidia.com/XFree86/Linux-x86_64/560.35.03/R...).
nubinetwork
3 days ago
In a way, I'm not surprised... when I reboot my systemd-based servers, it almost seems like (based on speed) that it didn't do anything to the running services and filesystems, and just tells the kernel to immediately reboot.
zokier
2 days ago
Why guess when the shutdown(/reboot) process is explicitly documented:
> Shortly before executing the actual system power-off/halt/reboot/kexec systemd-shutdown will run all executables in /usr/lib/systemd/system-shutdown/ and pass one arguments to them: either "poweroff", "halt", "reboot", or "kexec", depending on the chosen action. All executables in this directory are executed in parallel, and execution of the action is not continued before all executables finished. Note that these executables are run after all services have been shut down, and after most mounts have been unmounted (the root file system as well as /run/ and various API file systems are still around though).
https://www.freedesktop.org/software/systemd/man/devel/syste...
bbarnett
2 days ago
Yup. And unlike sysvinit Debian systems I run (hundreds under bookworm), systemd init systems (I run thousands) require all sorts of workarounds for this sort of behaviour. I get VMs not rebooting due to NFS umount failures, VMs not logging shutdown info because rsyslogd is terminated too early, literally endless issues.
Killing services without a proper TERM and wait prior to -9 is only one of the wonderful shortcomings I find with systemd.
zokier
2 days ago
TERM, wait, KILL is exactly what systemd does by default, and it's again configurable (and documented):
> If no ExecStop= commands are specified, the service gets the SIGTERM immediately. This default behavior can be changed by the TimeoutStopFailureMode= option. Second, it configures the time to wait for the service itself to stop. If it doesn't terminate in the specified time, it will be forcibly terminated by SIGKILL (see KillMode= in systemd.kill(5)).
https://www.freedesktop.org/software/systemd/man/latest/syst...
https://www.freedesktop.org/software/systemd/man/latest/syst...