For many applications, employing common programming techniques—such as memory locking—and savvy scheduling under the POSIX (Portable Operating System Interface for Unix) API (application program interface) can make the real-time performance of standard Linux acceptable. Applications requiring better responsiveness must turn to other techniques to enhance real-time performance under Linux. One of these is a microkernel-style architecture for high-speed performance with guaranteed task scheduling.
Linux architecture: Most Unix operating systems, including Linux, have a modular architecture consisting of both core and extendible elements. Linux implements its core kernel as a single piece of compiled code. It contains services critical to its basic ability to function within the specific requirements of the platform and the application.
When the monolithic core includes support for kernel-loadable modules, Linux can extend the kernel's services after compilation and installation. Loading and unloading these modules gives on-demand, run-time support to device drivers, file systems, communications protocols, and more.
This modular system provides two primary advantages over a fully monolithic approach. First, the kernel can remain small and streamlined, loading additional services as needed. Second, major portions of Linux functionality can be isolated, aiding both enhancement and debugging.
The microkernel approach used by the Real-Time Application Interface (RTAI) inserts a separate hardware abstraction layer (HAL), which intercepts and distributes the hardware interrupts of the system. Interrupts meant for scheduled real-time tasks are passed directly to those tasks, while interrupts not required by real-time tasks are captured and then emulated to Linux. This technique completely eliminates the operating system's ability to directly or indirectly affect time-critical tasks via interrupt control. For example, when Linux turns off interrupts, it shuts off the emulation of those interrupts to itself. It doesn't turn off the physical hardware, so all real-time tasks function as expected.
The HAL, implemented by about 70 lines of code on x86 microprocessor platforms and by about five lines on Power PC (PPC) architectures, provides the framework on which both the real-time tasks and the real-time services—like schedulers, semaphores, FIFOs (first-in, first-out), shared memory, POSIX, etc.—are mounted as kernel-loadable modules. Figure 1 depicts a typical system configuration, with Linux kernel, device drivers, real-time services, and real-time tasks.
Compared with alternative solutions, the microkernel approach offers three significant advantages to the real-time problem. First, this architecture provides a true hard real-time solution, offering complete pre-emption and determinism under all conditions. In addition, because the real-time tasks interface directly with the hardware and don't have to transfer signals through the standard Linux kernel, they offer the highest level of performance available. In fact, standard Pentium II class hardware has the ability to run tasks in excess of 30 kHz.
Finally, the standard Linux kernel only requires a small change to support RTAI, so the real-time services can keep pace relatively easily with, and separately from, the standard Linux source tree. This allows them to independently evolve without relying on each other, providing both improved maintenance and reliability.
The primary drawback to the microkernel approach is the increased complexity of application development. Applications implemented in the kernel, such as device drivers and real-time tasks, don't have direct access to Linux system and C library calls. Consequently, the full system solution must usually be separated into real-time components, implemented in the kernel under a special API, and supporting nonreal-time components—those employed in user space under the standard Linux API. But these limitations aren't as restrictive as they might seem.
Real-time application development: Under RTAI, a real-time system solution can be broken down into five fundamental development steps (Fig. 2):
- Separate the system into time-critical and nontime-critical elements, as illustrated by the discrete real-time and user-space regions.
- Provide control and data communication between real-time and support elements (sections A and B), including real-time and user-space FIFO initialization, and FIFO write/read.
- Establish a command handler to control the real-time element from the support element. The command handler spans the real-time and user spaces (represented by sections C and D).
- Develop the time-critical elements as hard real-time tasks (section C). This step includes the definition of the acquisition function, the initialization and scheduling of this task, and task cleanup.
- Set up nontime-critical (support) elements as standard Linux tasks (represented by section D). Implementation details of this step are application-specific and often include calculation, display, and disk-write steps.
Separating the system: The concept of system separation can be illustrated by a standard Linux device driver, which typically is written in two parts—the bottom and the top halves.
The bottom half usually reads and writes data directly between the device and an input/output (I/O) buffer. One interrupt asserts bottom-half processing, while all others are disabled. The interrupt must be dealt with as quickly as possible to minimize the blocked time of all other system activity.
Note that the bottom-half blocking provides significant degradation to the real-time performance of standard Linux and to real-time solutions that rely on pre-emption improvements to, or replacement of, the existing Linux scheduler. Under the microkernel approach, however, this bottom-half activity is simply pre-empted until the real-time task has completed.
The device driver's top half manages the device's associated I/O buffers, which are read and written by the bottom half, providing a higher-level interface to other processes. But unlike the bottom half, the top doesn't block because it runs in a process context.
The device driver provides a good example of separating a system's functionality according to how timely the system around it requires service. This is because there's a logical separation between elements that must interface directly to the hardware, and elements that support the data sent to, and received from, this hardware.
Like device drivers, real-time tasks can be logically separated into time-critical elements, or those requiring immediate service, and elements that support the time-critical tasks. This partitioned approach eases development, debug, maintenance, and testing throughout the development cycle.
Consider an industrial application that accepts data from an acquisition card at a 10-kHz rate, performs a calculation, displays the resulting data, and then saves it to disk. In this application, the hard real-time element is the precise control of the exact instant at which the data sample is taken. The calculation, display, and storage of that data can be thought of as soft real-time support tasks, because they must be performed within a "reasonable" time period.
In this example, the data-acquisition task would be implemented as a real-time task, and the remaining tasks could easily be implemented as standard Linux tasks. If enhanced soft real-time performance is required, the developer can use POSIX scheduling and memory locking without affecting the timing of the data acquisition itself.
RTAI's microkernel approach guarantees that the data-acquisition task will take place on schedule, even while the previously acquired and calculated result is written to disk. In contrast, other real-time solutions can quickly fall victim to Linux system activity, such as disk access.
Communicating between real-time and support elements: A system that's separated into real-time and support elements must have communication mechanisms for the elements to operate cooperatively. Two features ease this task. First, RTAI's Interprocess Communication (IPC) is quite similar in syntax and implementation to standard Linux System-V IPC. Among the communication mechanisms are FIFOs, shared memory, semaphores, mutexes (which are mutual exclusion mechanisms), mailboxes, and message queues.
Also, much of the RTAI/Linux symmetry described above has been extended to the IPC domain. For example, RTAI permits using the same FIFO or shared memory calls in both the real-time and user-space applications. This feature eliminates the need to employ one syntax for the real-time task and another for the support task.
Because the system requires both data (e.g. I/O data) and command/control signals, independent but similar IPC mechanisms must be provided for each. Code Listing 1 is a snippet showing the user-space FIFO open, and the real-time FIFO write, including useful error traps and print statements.
The command handler provides a method for the user-space application code to control the real-time task via the Command FIFO created in Code Listing 1. Refer to Code Listing 2 for how to get 'cmd' data, and either activate the data-acquisition thread if an 'a' is received, or terminate it if a 't' is received.
Developing the real-time task: Once you have partitioned your system and provided communications and control mechanisms, it won't require expertise in the black arts to develop real-time tasks under RTAI. Those experienced with real-time systems programming, or Linux kernel development, will find application development under RTAI to be rather straightforward. Even those lacking such experience will find that these concepts aren't much more difficult than learning how to develop a real-time system in the first place.
To develop real-time solutions, RTAI uses the standard Linux GNU (not Unix) tool chain—just like employing these tools to develop standard applications, the Linux kernel, and device drivers. Those accustomed to using an integrated development environment (IDE) will find that a standard one, such as the CodeWarrior development tool from Metrowerks of Austin, Texas, works just fine for real-time Linux projects.
Manual system debug and analysis is supported through RTAI's /proc interface, which is the standard Linux mechanism for reporting the status of various system elements. Manual debug is also supported with the rt_printk call, which effectively functions identically to Linux print_k but is real-time safe. Similarly, the rt_print_to_screen can be used if you don't need to log messages but only want to display them onscreen instead. To further aid system analysis and debugging, Lineo has produced a run-time debugger, modified gdbstubs (the standard Linux kernel debugging stubs) for step-and-trace kernel module debugging, and funded Opersys Inc. of Montreal, Canada, to provide real-time support to the Linux Trace Toolkit. Stubs are routines that don't actually do anything other than act as placeholders for routines yet to be developed.
There are three ways to accomplish application development under RTAI: with a separate RTAI API, with industry-standard POSIX threads (pthreads) and POSIX queues (pqueues), or by using a combination of the two when services supported by the RTAI API have no pthreads or pqueues equivalent.
A separate real-time API is necessary because Linux runs completely unaffected by, and oblivious to, the presence of RTAI. So, there must be a mechanism to distinguish RTAI calls from standard Linux calls, especially when each supports similar functionality. In addition, RTAI supports a number of services that have no equivalent in standard Linux. Therefore, new calls are required to define this functionality.
RTAI API supports several new real-time operating-system (RTOS) services, including schedulers, System-V-like IPCs, POSIX pthreads and pqueues, and dynamic memory allocation. In general, the standard kernel API can't be implemented directly in a real-time task because if that kernel call blocks, a system lockup usually results. System calls can be made using the system request mechanism, though, which ensures that the standard Linux kernel functions are called at a "safe" time.
Although the RTAI API is different from the traditional Linux API, developing systems with RTAI isn't as onerous as it might seem. The RTAI package provides API symmetry with standard Linux. In other words, it permits using many of the same calls and syntax in both real-time and standard applications. Also, those who have previous RTOS experience will feel quite comfortable with RTAI's support for familiar features and services, such as periodic and one-shot schedulers, semaphores, FIFOs, message queues, shared memory, mailboxes, and mutexes.
RTAI also supports programming with fully portable calls of the industry-standard POSIX API. In fact, RTAI provides nearly the same POSIX pthreads support as standard Linux, and its support of pqueues functionality is unmatched by Linux.
Code Listing 3 illustrates an example data-acquisition real-time thread. This thread waits for the 'a' activation command from the user-space program, via the command handler. It then loops, first reading data from the data-acquisition card, next transmitting the data to the user-space program through the data FIFO, until a terminate command is received.
Once the real-time thread has been defined, it has only to be created and initialized for its execution. At that time, the periodic task schedule is set, as shown in Code Listing 4.
Memory protection for real-time tasks: RTAI is similar to incumbent proprietary RTOS solutions because neither these nor RTAI have native memory protection for real-time tasks. However, RTAI differentiates itself by also supporting hard, fully pre-emptive, real-time tasks from memory-protected space.
RTAI's LinuX-Real-Time (LXRT) module enforces hard real-time, fully pre-emptive scheduling of real-time tasks residing in Linux user space. This capability comes with an increase in maximum jitter, from about 15 µs (worst case) for hard real-time tasks in the kernel to around 40 µs (worst case) for hard real-time tasks in user space.
Because these hard real-time user-space tasks are written with the same RTAI API as the kernel tasks, they can be moved from user space to kernel space quite easily. By this portability, the developer can write a single application and then deploy it to obtain either the best performance or the benefits of memory protection.
System support elements: A real-time system generally consists of a small segment of time-critical code surrounded by a much larger collection of support code. Although nearly everything in a system is time-critical to some degree, let's assume the acceptance of nontime-critical elements provided that they're performed within a "reasonable" time period.
The system support tasks are implemented as standard processes, so they have full and unencumbered access to all Linux system and C library calls, device drivers (network, display, disk, etc.), services (memory protection, network protocols, file management, etc.), and features. Code Listing 5 shows the application code for sending the command to start data acquisition to the RT task. No example code is given for the calculation, display, and data-logging elements because they're implemented as standard Linux tasks.
The microkernel architectures of RTAI and Linux yield a solution that provides guaranteed, fully pre-emptible hard real-time tasks with microsecond response times, while allowing full access to everything standard that Linux has to offer. Linux is a customizable, excellent performer, and it offers readily available source code. The combination of Linux and RTAI results in a solution that's very competitive with incumbent embedded RTOSs.
In the future, RTAI will improve the programmer's accessibility to Linux services and system calls. In fact, RTAI's evolving "Unix-server" feature now allows limited comingling of the Linux and RTAI APIs from within the same code space. The ultimate goal is to offer complete and transparent integration of these same APIs.
The latest news, documentation, code examples, and downloads on RTAI can be found at the RTAI homepage Web site: www.aero.polimi.it/projects/rtai/.
David Beal is the product marketing manager for Embedix Linux and µClinux at Lineo Inc. (www.lineo.com), Lindon, Utah. He earned his BS degree in electrical engineering technology from the State University of New York at Utica, where he graduated with high honors. He was instrumental in the success of Zentropix Inc., a pioneer in the development of real-time Linux tools, until its merger with Lineo in early 2000. He has written extensively on RTAI for Linux and continues to support the RTAI documentation project. His e-mail is [email protected].
Steve Papacharalambous is the software engineering director at Lineo. He graduated with honors from England's University of London, where he received his BS in electronics engineering. He also has the distinction of being a Chartered Engineer. He has worked in various areas of real-time computing, including defense and industrial control, and has contributed extensively to RTAI for Linux. He now serves as the chief architect for Lineo's hard real-time legacy RTOS API solutions. His e-mail address is [email protected].