The heart of real-time support for Linux has finally made it into the official kernel. As before, real-time support cannot be activated there, because there are still some medium-sized and all kinds of small construction sites. Developers of the project, which has been dragged out due to lack of money, discussed these things at the Linux Plumbers Conference 2021, which is currently taking place. There it was also discussed how they want to maintain the whole thing in the future.
The heart has landed
Real-time support for the Linux kernel reached an extremely important milestone on the penultimate day of August: Linus Torvalds has the majority of what is required for proper real-time support Locking infrastructure integrated into the main Linux development branch. This was previously the heart of the “RT Kernel” (also known as “RT Patch” or “RT Branch”), in which the developers of the PREEMPT_RT project have long been collecting, maintaining and improving the source code patches that help Linux achieve real-time properties.
The recorded changes to locking techniques such as spinlocks, mutexes and Rwlocks are ultimately the core of it all, with the PREEMPT_RT in autumn 2004 still started under a different name Has. The now integrated adjustments added up to a whopping seventy patches. They allow a kernel compiled with real-time support to interrupt almost all of the tasks it has performed at any time without any major delay in order to temporarily devote itself to more important things.
This is crucial for real-time support: Linux can quickly turn to programs that always have to complete a certain task in a pre-defined time – even if something unimportant demands a lot from the system and creates adverse conditions. A marking on the process determines which programs the kernel should prefer. For many PCs and servers in particular, this is of no interest, because the greater responsiveness means that other tasks are interrupted more frequently and run more slowly.
Second milestone just a few days later
With the PREEMPT_RT core, Linux can interrupt itself in significantly more places, but that alone is not enough: the kernel has to execute some code areas in one go to ensure the integrity of the data currently being processed in multiprocessor systems.
There were numerous areas of this type leading to longer waiting times in the SLUB Memory Allocator, which provides small areas of the main memory internally in the kernel. A series of over 30 patches eliminated the cause of this, which Torvalds integrated into the main Linux development branch a few days after the locking changes.
The adjustments will be included in Linux 5.15, which should be released on November 1st or 8th. Its code already contains basics for activating realtime support that are already integrated in Linux 5.3. However, the option cannot be activated when compiling a kernel.
A big and all sorts of small construction sites
This is due to other uninterruptible parts of the Linux code, which are known to lead to undesirably long waiting times that would interfere with real-time use. One of them is the print infrastructure, via which the kernel accepts and outputs log messages. The PREEMPT_RT developers have considerably improved the situation there in the last year and have been able to integrate another important change in Linux 5.15; at the same time, however, further work is still pending. Details can be found in the slides and the video recording one LectureThe developer behind the conversions on Tuesday on the Linux Plumbers Conference 2021 has held.
In addition to the print work, there are still many small construction sites. This can be seen when looking at the preliminary version of the realtime-compatible Linux 5.15-rt, for which the developers now “only” apply 175 patches, change about 215 files, inserting 1350 lines and removing 2650 – that’s around a hundred patches fewer than with Linux 5.14-rt (260 files changed, 3700 lines inserted, 7450 lines removed). However, some of the patches are only relevant for certain areas of application or processor architectures.
Time frame remains uncertain
On Tuesday, the developers discussed some of these and other construction sites as part of a Status overview of the RT patches discussed. There, two more medium-sized construction sites were addressed with “cpu_chill ()” and NOHZ behavior. But between the lines I could hear that even the developers don’t dare to predict how long it will be.
Ultimately, it remains uncertain whether months or years will pass before all major changes move into the main development branch of Linux – although it was already said in 2018 and 2019 that it would soon be there, but that it would ultimately turn out differently. Sometimes problems were simply underestimated. A big problem was and is always a lack of money: Although many companies use the RT kernel, relatively few actively participate in the development or contribute money. Central PREEMPT_RT developers therefore had to postpone work on the real-time patches several times and devote themselves to other tasks in order to earn their bread and butter.
Care today and tomorrow
In the status discussion and in the lecture Maintaining PREEMPT_RT: now and then (Foils, recording) it was partly a question of how the maintenance of real-time support should proceed when everything essential is contained in the official kernel. Because inexperienced changes at any point by other developers could massively disrupt the real-time suitability. The financing of the developer resources required for this should therefore also influence the development speed and the quality of the real-time capabilities of Linux in the future.