A pioneer of embedded operating systems (OSs), Jim Ready is not only credited with the creation of one of the first commercially available real-time operating systems (RTOSs), the Virtual Real-Time Executive, under his guidance MontaVista helped pave the way for the use of Linux in embedded devices in the early 2000s. Now an independent consultant, Ready reflects on how early work in embedded Linux that prompted modern mobile OSs like Android also branched into more deeply embedded applications through the advent of capabilities such as the Realtime Preemption (RT-Preempt) patch, and how that evolution could ultimately impact the software hierarchy in the Internet of Things (IoT).

How do you describe “real time” today in the OS sense?

READY: There are two ways to think about this. One is, in academic circles you can find what people have defined as real-time computing, and I don’t think that’s changed any: The classic phrase is, “The right results, on time,” and that’s always whatever your requirements are. It has nothing to do with time necessarily; it has to do with meeting the requirements of the time-related aspects of your application, which could be very soft or very stringent.

The second is some of the things that have changed how people look at real time. If you look at your smartphone, it’s doing all sorts of stuff that is obviously real time, from the radio to picture processing. There is a range of ways that’s solved: Number one is you just run what you need in hardware and achieve whatever requirements you have; or it can be very dedicated software on that hardware, so classic hard-real-time software running in the bowels of the hardware; or, even with the software that moves up into the application area and into the main processor, because the underlying power of an ARMv8-A in an iPhone allows for incredible processing in a very, very short period of time.

The system design aspect of this has settled down. In other words, people know how to do this now and there are no giant arguments about what Linux can or can’t do or what dedicated software can or can’t do or what an RTOS can or can’t do. It’s settled into, “Choose the right solution for the right aspect of the problem,” and people are reasonably sophisticated about that now. 15 years ago we might have had some of these arguments where people were going berserk about Linux not being able to do real time, but that’s all gone. Whatever deficiencies there were have been fixed, solved, and are good enough; people have 15 more years of experience designing these systems; and the hardware has gotten 15 years better. It’s a non-issue that’s been solved effectively, to my mind.

What was done to make the Linux kernel real time?

READY: All of the classic things you did when you built an embedded system, Linux had not been geared that way because Linus [Torvalds] was working on his desktop. Linux at the very beginning had something called Big Kernel Lock (BKL), which meant that relatively large portions of the kernel were made non-preemptive just to make things simpler, and once you got into those you had to execute to completion.

The whole game of adding real-time capabilities to Linux was to reduce the size of that BKL, which then would increase responsiveness to be able to do other things. That was what the RT-Preempt patch work focused on, and the point was basically to get the kernel out of the way. Once you did that you had the same instructions everybody else had in terms of handling interrupts, hardware latencies, and everything else, so it removed an artificial restriction and still protected the kernel, just in a much more fine-grained way. So there could be mountains of code that execute when they execute, but not at the expense of whatever needs to be done in real time. This was done through a kind of partition.

[Figure 1 | Depicted in this histogram is a comparison of the latency performance of the default preempted Raspbian Linux kernel (red) versus the Raspbian kernel with the RT-Preempt real-time patch. While the minimum and average latencies for each were similar (14 µs minimum and 28 µs maximum for the default preemptive configuration of the Raspbian kernel versus 12 µs minimum and 27 µs average for one with the RT-Preempt patch), the most important measurement, maximum latency, shows a huge discrepancy (1277 µs for default versus 77 µs for RT-Preempt). Data and figure courtesy Emlid Ltd.]

At the Linux Foundation there is a working group that continues to maintain and mainstream the RT-Preempt patch. It’s lead by Thomas Gleixner, and he was one of the original guys who worked on Linux to make it more real time. So there’s a long, long continuum of really good embedded guys who have put their efforts into continuing to improve the capabilities of Linux.

Given work on the RT-Preempt patch and other deterministic enhancements, how does real time Linux now compare to an RTOS?

READY: Whether you’re running an RTOS or you’re running Linux, the processor doesn’t go any faster – the guys who write the RTOS effectively use the same instructions as the guys who write Linux. It’s really how you organize everything, and for all intents and purposes, 10 or more years ago we showed that the latencies you could achieve with Linux were on the same order of what you could achieve on the same processor with an RTOS. It wasn’t like 100 to one or even 10 to one – by careful design, you could utilize the underlying hardware and get the results you needed.

At MontaVista we had something called the Bare Metal Engine that produced an environment within Linux that allowed access to the bare instruction sets that the hardware provided so you were at the same level of latencies that you would be at with an RTOS. Say you had a multicore network processor like a Cavium Octeon with dedicated hardware engines in it doing packet processing. If you wanted to have an environment that was Linux but you didn’t want to have any of the latencies of Linux get in the way, the Bare Metal Engine allowed partitioning and the use of the internal capabilities of Linux such as processor affinity to tailor an environment so that the application could have full access and control over the underlying packet processing hardware, but still in a Linux context. It was a real Linux program, just with a very refined, very high-performance, very dedicated environment. If you wanted to put an RTOS down, you could, but you didn’t necessarily have to. It could be a pure application, for example, that was driving these hardware accelerators.

We were doing this in 2009, 2010, and 2011, so a relatively long time ago. We were taking capabilities that found their way into the Linux kernel like the RT-Preempt patch and exploiting them, and doing very low latency, very high-performance environments that were as capable as if there was no OS running on the chip. It’s not like you were chopping out big chunks of stuff at all. There was no removal of code whatsoever. It was just functions and capabilities of Linux itself, and if you use them in a particular way and configure them in a particular way, you can achieve this. It wasn’t a trick where there was one line of Linux left and you snuck in an RTOS or you had an RTOS and you ran Linux as all of the tasks. It’s Linux. We weren’t gutting it. We weren’t ripping it apart. We were using features that were there and adding some in the classic open source way, but it was absolutely in the mainstream of Linux.

If you put together a full RTOS you need 1 MB or 2 MB, and it’s not that hard to get Linux configured to run in that range. Now, it still wouldn’t make any sense to use Linux where you need a five-line executive or a little tiny kernel. But the whole point of early work on the RT-Preempt patch was to make Linux capable enough to be able to use it where it makes sense to use it. What it took out of the equation was “I’d really like to use Linux here but it’s not quite real time enough” so that it could be symmetric with your requirements.

What implications does embedded Linux have on the future of OSs given ongoing convergence in the IoT?

READY: If you meet your requirements, mainstream is the place to be. Fundamentally that’s why I got interested in Linux for embedded systems in the first place many years ago, because it was just so expensive to be on the outside. If you had an RTOS you had to go find a TCP stack, etc. Once I saw Linux came out, I said, “Hey, this solves a huge number of problems if we can make it work for embedded systems.” Of course, we did, and the rest is history. The more mainstream you are, the less work you’ve got to do.

The way that’s happening in the IoT is through the consortiums that have been formed. And it’s not even so much standards based as it is implementations, so these consortiums are cutting right to the chase and providing open source versions of middleware and protocol stacks through reference implementations that give people a place to start. The vast majority of the software and intelligence will be shifted off the devices into standard computing environments designed for IoT, so you’ll want to be in the normal world as much as possible. People are taking a system approach to this, secondly there are all of these consortia, and it’s very hard for a proprietary technology to carve out any kind of control. I think that’s impossible now.

In the actual physical systems there is a hierarchy of underlying hardware capability and then there will follow a hierarchy of software. There is a role for RTOS in there, and at some point where you leave off with RTOS, the next stop is Linux, which can then carry you quite a ways since that’s what’s also on the cloud side of things. It will span both sides, as it will obviously have a home even in the hardware hierarchy that’s supplying the cloud.

If you look at it economically, the pressure is probably going to be on the smallest number of people to adapt so the vast majority of folks can continue doing what they’re used to doing, just for efficiency reasons. You’re not going to retrain a gazillion IT guys.