System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I

17 pointsposted 4 days ago
by matt_d

6 Comments

quotemstr

2 hours ago

Linux is unusual in OS kernels in that direct system calls from arbitrary userspace code are supported and ABI-stable. This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition.

If, instead, as on OpenBSD, the kernel enforced the rule that all system calls had to go through libc (or perhaps a big ntdll.dll-like VDSO), then the whole problem the linked article tries in vain to solve would disappear. If you wanted to hook a system call, you'd just change the libc/VDSO dispatch. No need to rewrite any instructions.

If I were Linus, I'd make a new rule: starting today, all new system calls must go through VDSO. No exceptions. SYSCALL from anywhere else? SIGKILL.

This way, you can just LD_PRELOAD in front of the VDSO and system call interception in userspace Just Works.

freestanding

a few seconds ago

thats why OpenBSD is unconvinient for development - because it binds to libc bloatware

yjftsjthsd-h

an hour ago

> This model has always been a terrible idea. It robs the system of an ability to intercept system calls in userspace before doing an expensive privilege-mode transition.

This model has always been a trade-off. It has downsides, but it also has upsides, including an immense boost in flexibility; decoupling from any particular userspace is useful.

> This way, you can just LD_PRELOAD in front of the VDSO and system call interception in userspace Just Works.

Can you LD_PRELOAD in front of the vDSO? I was under the (possibly mistaken) impression that the kernel injects it directly.

throwaway7356

32 minutes ago

> all system calls had to go through libc (or perhaps a big ntdll.dll-like

Which makes containers crap on Windows and *BSD as they have to run the currect libc or equivalent. Thus you need to build a different container per OS version which sucks compared to Linux.