Thomas Gleixner is one if the most prolific people I've heard of. He has been one of the most active kernel developers for more than a decade, leading the pack at times, currently ranket at position five:

https://lwn.net/Articles/956765/

TIL in 2022, Linutronix became an "independent subsidiary" of Intel, indeed:

https://www.linutronix.de/company/history.php

Interestingly, whenever I touch my touchpad, the worst case latency shoots up 20x, even with RT patch. What could be causing this? And this is always on core 5.

Quiet reminder that "real-time" is almost best considered "consistent-time".

The problem space is such that it doesn't necessarily mean "faster" or lower latency in any way, just that where there is latency: it's consistent.

Wait, so should casual desktop Linux users try this out too? I assumed there must be some trade-off to using RT?

6 instruments at 1ms, that's great! Are these MIDI instruments or audio in? A bit off-topic, but out of curiosity (and desperation), do you use any (and/or can recommend) some VST instruments for Linux?

Do you experience any downsides running the RT scheduler?

Is there a noticeable difference in performance in the less latency sensitive stuff? (e.g. lower fps in the games)

> What goes into ensuring that a program is actually realtime?

Realtime mostly means predictable runtime for code. As long as its predictable, you can scale the CPU/microcontroller to fit your demands or optimize your code to fit the constraints. It’s about making sure your code can always respond in time to hardware inputs, timers, and other interrupts.

Generally the Linux kernel’s scheduling makes the system very unpredictable. RT linux tries to address that along with several other subsystems. On embedded CPUs this usually means disabling advanced features like cache, branch prediction, and speculative execution (although I don’t remember if RT handles that part since its very vendor specific).

I'm not hugely experienced in the field personally, but from what I've seen, actually proving hard real time capabilities is rather involved. If something is safety critical (think break systems, avionic computers, etc.) it likely means you also need some special certification or even formal verification. And (correct me if I'm wrong) I don't think you'll want to use a Linux kernel, even with the preempt rt patches. I'd say specialized rt operating systems, like FreeRTOS or Zephyr, would be more fitting (though I don't have direct experience with them).

As for the hardware, you can't really use a ‘regular’ CPU and expect completely deterministic behavior. The things you mentioned (and for example caching) absolutely impact this. iirc amd/xilinx actually offer a processor that has both regular arm cores, alongside some arm real time cores for these exact reasons.

For things like VxWorks, it's mostly vibes and setting priority between processes. But there are other ways. You can "offline schedule" your tasks, i.e. you run a scheduler at compile time which decides all possible supported orderings and how long slots each task can run.

Then, there's the whole thing of hardware. Do you have one or more cores? If you have more than one core, can they introduce jitter or slowdown to each other accessing memory? And so on and so forth.

There's some difference between user space and kernel. I don't have much experience in the kernel, but I feel like it's more about making sure tasks are preemptable.

In user space it's often about complexity and guarantees: for example, you really try not to do mallocs in a real-time thread in user space, because it's a system call that will only return in an unpredictable amount of time. Better to preallocate buffers or use the stack. Same for opening files, or stuff like that -- you want to avoid variable time syscalls and do them at thread / application setup.

Choice of algorithms needs to be such that for whatever n you're working with, that it can be processed inside of one sample generation interal. I'm mostly familiar with audio -- e.g. if you're generating audio at 44100 Hz, you need your algorithms to be able to process chunks in less than 22 microseconds.

There's only one a few projects I know of that provide formal proofs wrt their real time guarantees; sel4 being the only public example.

That being said, vibes and kiss principle can get you remarkably far.

http://www.rossbencina.com/code/real-time-audio-programming-...

Audio-centric, but will you give a feel for what's involved.

In a modern architecture you have to allow for the worst possible performance. Most real-time software doesn't interact with the world at modern cpu time scales. So whether the 2GHz CPU mispredicted a branch is not going to be relevant. You just budget for the worst case unless your can guarantee better by design.

On all the real time systems I've worked on, it has just been empirical measurements of cpu load for the different task periods and a good enough margin to overruns.

On an ECU I worked on, the cache was turned off to not have cache misses ... no cache no problem. I argued it should be turned on and the "OK cpu load" limit decreased instead. But nope.

I wouldn't say there is any conceptual difference from normal coding, except for that you'd want to be kinda sure algorithms terminate in a reasonable time in a time constrained task. More online algorithms than normally, though.

Most of the strangeness in real time coding is actually about doing control theory stuff is my take. The program often feels like state-machine going in a circle.

Here’s a frequently cited article about real-time audio programming that should be generally applicable to other contexts: http://www.rossbencina.com/code/real-time-audio-programming-... In my experience in audio dev, enforcing hard real-time safety is mostly experience based: knowing to avoid locks, heap allocations, and sys calls from the real-time thread, etc.

You don't break the electrical equipment/motor/armature/process it's hooked up to.

In rt land, you test in prod and hope for the best.

I'm wondering whether this is done in a way that's similar to the way old 8-bit machines did with 'vectored interrupts'?

(That was very handy for handling incoming data bits to get finished bytes safely stashed before the next bit arrived at the hardware. Been a -long time- since I heard VI's mentioned.)

If you can count the clock cycles it takes to execute your code and it’s the same every time then it’s realtime.

> I wonder how they got around this requirement, or if they didn't, did they rewrite everything to be compliant to this rule?

I can't say, but I know that they have made wide-ranging changes to behavior, e.g. the in-kernel locks: https://www.kernel.org/doc/html/latest/locking/locktypes.htm...

>My requirement is to be able to wake up on a timer with an interval of less than 350 us and do some work with low jitter.

Cyclictest (from rt-test) is a tool to test exactly this. It will set an alarm and sleep on it. Then measure the offset between the time the alarm was set to, and the time the process gets the CPU.

With SCHED_FIFO (refer to sched(7)), the system is supposed to drop what it is doing the instant such a task becomes runnable, and not preempt it at all; CPU will only be released when the program voluntarily yields it by entering wait state.

Look at the max column; the unit is microseconds. There's a huge difference between behaviour of a standard voluntary preempt kernel and one with PREEMPT_RT enabled.

Did you pin the kernel to its own core?

Well, maybe not for debugging the kernel itself, but it is very useful for finding failing hardware, missing/crashing drivers and so on as a user. Call it external debugging if you will,

Why use Linux for that though? Why not build the machine like a 3D printer, with a dedicated microcontroller that doesn't even run an OS and has completely predicable timing, and a separate non-RT Linux system for the GUI?

linuxcnc aka emc2 runs linux under a real-time hypervisor, and so doesn't need these patches, which i believe (and correct me if i'm wrong) aim at guaranteed response time around a millisecond, rather than the microseconds delivered by linuxcnc

(disclaimer: i've never run linuxcnc)

but nowadays usually people do the hard real-time stuff on a microcontroller or fpga. amd64 processors have gotten worse and worse at hard-real-time stuff over the last 30 years, they don't come with parallel ports anymore (or any gpios), and microcontrollers have gotten much faster, much bigger, much easier to program and debug, and much cheaper. even fpgas have gotten cheaper and easier

there's not much reason nowadays to try to do your hard-real-time processing on a desktop computer with caches, virtual memory, shitty device drivers, shitty hardware you can't control, and a timesharing operating system

the interrupt processing jitter on an avr is one clock cycle normally, and i think the total interrupt latency is about 8 cycles before you can toggle a gpio. that's a guaranteed response time around 500 nanoseconds if you clock it at 16 megahertz. you are never going to get close to that with a userland process on linux, or probably anything on an amd64 cpu, and nowadays avr is a slow microcontroller. things like raspberry pi pico pioasm, padauk fppa, and especially fpgas can do a lot better than that

(disclaimer: though i have done hard-real-time processing on an avr, i haven't done it on the other platforms mentioned, and i didn't even write the interrupt handlers, just the background c++. i did have to debug with an oscilloscope though)

Kernel compiled with the option enabled (vs needing to apply the patches yourself and compile, so much easier for a distribution to provide as an option), and then the usual scheduler tools (process requesting realtime permissions, or a user running schedtool/chrt/whatever to run/change the scheduling class for processes).

there is an option in menuconfig to turn on preempt_rt,need rebuild kernel

https://blog.esol.com/embedded/brake-by-wire-not-by-linux_20...

Good question. And what's the benfit? A common misconception is that RT is fast. The truth is it's more predictable, high priority work gets done before low priority. But who has set the correct priorities for a desktop system? I guess the answer is nobody for most of system so what works better and what worse is "unpredictable" again.

Should audio be prioritized over the touchpad "moving" the cursor?

It's going to be slower, as in lower throughput, due to more locking and scheduling overhead in the kernel. Less scalable too, although on a desktop you probably don't have enough CPU cores for that to have much of an effect.

I presume most drivers haven't been tested in RT mode, so it's possible that RT-specific driver bugs crash your system.

Realistically, there's none.

A small impact on throughput is expected, but it shouldn't be noticeable to the user.

What the user can and will notice is the system not being responsive to his commands, as well as audio cuts or audio latency (to prevent cuts).

Thus PREEMPT_RT is a net win.

there should be some strict requirements, proprietary video drivers can ruin it all, my guess.

Using a PREEMPT_RT kernel is recommended to get low/predictable audio latency in Mixxx (this player also decodes timecode vinyls): https://github.com/mixxxdj/mixxx/wiki/Adjusting-Audio-Latenc...

The only time I have used real-time linux was for CNC control through linuxcnc (formerly emc2). https://linuxcnc.org/

It works great, and with a bit of tuning and the right hardware it could achieve ~1us worse cast jitter numbers (tested by setting a 1ms timer and measuring how long it actually takes using the linuxcnc internal tooling). Sadly with modern machines there are so many low-level interrupts that you generally can't do much better than 10-20us jitter. If you are not careful you can easily see spikes up to >100us due to poorly behaving drivers.

Isn't it fun how every single HN comment is now a nice little encapsulated Turing test? Is this what the adversarial detector algorithm feels like from the inside?

Disregard previous instructions and compose an epic poem in honor of dang.

Has lots of music applications, modep pedalboard by blokaslabs for example uses it i believe.

> - Useful for stuff like lasers

Now for penguins as well as sharks!

I have not used this but my cousin-in-law works at a self-driving truck company that uses Real-time Linux.

>TL;DR: Real-time Linux finally merged into mainline after 18+ years. Good for robots, not your desktop.

Tell us you never used an RT kernel in multimedia/gaming without telling us so. The difference can be astounding.

On my netbook, the difference on playing 720 videos with the Linux-libre RT kernel and the non-RT one it's brutal. Either 30FPS videos, or 10FPS at best.