Embedded Linux Starting To Make Sense

The Linux hype is in full swing in embedded-systems projects. A small but rapidly growing number of startup companies are offering Linux code, tools, and services for embedded development. Web sites such as www.Linuxdevices.com are springing up to provide resources on tools and development techniques. In addition, vendors such as Motorola and Force Computers have announced that hardware products which use different Linux builds are being manufactured.

Because of its advantages, Linux makes a lot of sense for embedded systems. These include the availability of source code, no licensing fees, and readily available free and paid support at Linux vendors and on newsgroups. The ability to customize an operating-system build for a specific application, download and use free yet high-quality programming tools, and doing it your own way without starting from scratch appeals to both project managers and individual developers.

Chances are high that your engineering team is anxious to get a Linux project under its belt. Linux carries such a cachet that even a long-time employee would be interested in learning the hot technology. Team members seeking to broaden their skills base for new projects or better employment prospects will jump at the chance to work with Linux.

Linux is known to offer high reliability. Many discussion groups and Usenet forums include comments on how Linux servers run for months, or even years, without reboot. Its open-source development model, with contributions from programmers around the world, means that the best technical solutions and fixes are likely to find their way into source-code distributions. From any standpoint, the result is a sturdy technical solution offering the best efforts of thousands of programmers.

Still, that's not all that's important for embedded developers. They need solutions that work—tools that are powerful, easy to use, well supported, and in step with the technology. The operating system must fit into the specified footprint, and with the performance and response time needed for the project. Powerful development tools have to be available to customize the operating system and produce a wide variety of applications.

Unfortunately, Linux has a mixed track record on many of these issues. Free or inexpensive Linux development tools tend to be highly capable, but not as easy to use as tools for other embedded operating systems. A Linux operating-system build can be almost infinitely customized. But there are few documents or guidelines for doing so. While Linux has the potential to offer advantages over the long run, the learning curve of a first Linux project may be longer than that with standard commercial tools.

Embedded-system vendors are working quickly to overcome these and other deficiencies. Further, engineers are using Linux in actual designs. The result is an explosion of technical innovation and engineering skill in fielding a new generation of powerful embedded devices.

Many embedded developers already use the GNU development tools from the Free Software Foundation. Starting with founder Richard Stallman's GNU C, the GNU library now has dozens of programming tools, including over a dozen programming languages, as well as debuggers, code frameworks, libraries, editors, Web development tools, utilities, and much more.

Plus, many of these tools are of higher quality than those available commercially. Many real-time operating-system vendors support development with GNU programming tools, because of their quality and wide availability for different microprocessors. In 1998, the last year for which figures are available, Cygnus Solutions estimated that approximately half of the embedded development community used at least some of the GNU tools.

What does GNU have to do with Linux? Many Linux developers have always used the GNU tools when working with the operating system, because of their wide availability with source code. Many of the core components of Linux were actually developed and distributed under the GNU General Public License (GPL), a free licensing policy developed by Stallman.

The GPL may deter some embedded developers from working with GNU programming tools. Software distributed under the GPL typically comes with source code. Developers can modify that source code, but they can't then redistribute it as a proprietary product. Developers can even charge for distributing it, but they still have to give users the same rights that they received.

Yet, the GNU license doesn't bother most developers using the programming tools rather than application source code. Even some vendors have made it work to their advantage. Cygnus Solutions, which was acquired early this year by Linux distributor Red Hat Software, developed and distributed GNU tools at a minimal cost. Cygnus made money from consulting and other products.

Cygnus/Red Hat has supplemented the GNU tools with its own proprietary tools, such as the Source Navigator debugger, and Code Fusion integrated development environment (IDE). These tools allow users to target Linux on Intel processors (Fig. 1). Unlike GNU tools, these aren't free, but can incorporate appropriate GNU tools into the environment. In addition to enabling Linux development, they also are hosted on Linux and Windows.

Many GNU programming tools—including language compilers, debuggers, and various utilities—are included in a large number of commercial Linux distributions, including Red Hat, Debian, and Caldera (see the opening figure). Free tools, usually with source code, also are available from the Free Software Foundation, research universities, and several of the Linux portals, such as www.Linuxcare.com. Finally, more commercial tools like those from Cygnus/Red Hat are emerging. This brings about a wider range of quality development software than is currently available for virtually any embedded operating system.

The issue of real time and Linux has been a curse, as well as a blessing. It's a curse because the operating system isn't inherently real time, and getting there is technically problematic. But, Linux is a blessing from the perspective of it forcing users to confront just what part of a project requires a real-time response, how much real time is needed, and how to best deliver on that requirement. In many cases, real-time responses are only needed under certain circumstances, or in specific classes of responses. Embedded-system designers are figuring out the best way to make a less-than-full-time operating system function as effectively as a real-time operating system.

What does Linux offer out of the box? The POSIX 1003.13 standard defines a Multiuser Real-Time System Profile. It allows for real-time processes to be locked into memory to prevent the process from being paged out to the hard disk. And, a special scheduler ensures that these processes are always executed in a predictable order.

Linux meets this standard by providing POSIX-compliant memory locking, special schedule system calls, and POSIX RT signals. While those features improve deterministic performance, the resulting tasks cannot be defined as hard real-time tasks because the soft real-time process still can be blocked by kernel activity.

There are two general approaches to turning Linux into a more deterministic real-time system. The first is exemplified by the RT-Linux project out of New Mexico Tech or the Real-Time Application Interface (RTAI) project from the University of Milan. Users of this approach take a generic Linux operating system and run a nonLinux real-time kernel underneath it. The Linux operating system is run as the lowest-priority task of a small real-time operating system.

Jim Ready's Monta Vista Software pioneered the other approach, with its Hard Hat Linux distribution, a variation of Red Hat. In this approach users tune drivers and other system components to focus on how to get better and more predictable responses out of existing operating-system components. By improving driver throughput, interrupt latencies, and scheduler response times, it's possible to make the operating system deterministic enough for a much larger range of uses. In the spirit of the Linux movement, Monta Vista freely distributes Hard Hat, and offers consulting services to pay the rent.

Still, each approach has limitations that might cause an embedded designer to think twice before adopting it. The first approach has the advantage of using Linux for application development. But if any applications need real-time services, all of the those advantages are lost. The latter approach enables Linux to be used with a wider range of embedded systems. But, it still lacks hard determinism. The Hard Hat solution may also be more difficult to maintain, because code modifications might affect real-time performance.

Under way are initiatives to improve on the tuning approach, and make better information available to embedded developers. For example, to help developers characterize the responsiveness and determinism for specific hardware targets, MontaVista Software has formed the Standard Linux Real-Time Characterization Project. The goal of this project is to define and publish Linux real-time performance numbers, and bring tools to the embedded Linux community to perform similar tests.

The company also is developing support software for Linux outside of the operating system. Hard Hat Net enables both CompactPCI system controllers and peripheral devices to communicate using standard networking protocols across the CompactPCI backplane at high data rates (Fig. 2). This enables embedded Linux developers to implement a variety of loosely-coupled and distributed pc-board architectures.

In addition to Monta Vista's Hard Hat Linux, at least two other versions of Linux have been adapted for embedded use. One is Red Hat's eCos, the latest version of what began as a Cygnus Solutions' open-source, configurable, application-specific operating system for embedded systems. It provides manufacturers of embedded devices with a special-purpose Linux alternative for bringing intelligent applications to embedded devices like Internet appliances, personal digital assistants (PDAs), and handheld computers. The latest version, eCos 1.3, features a built-in TCP/IP stack that enables embedded devices to communicate with the Internet.

As open source code, eCos is freely available. Developers can download eCos implementations from Red Hat's website for many different processors and board support packages, including ARM, Hitachi SuperH, Intel x86, Matsushita AM3x, MIPS, PowerPC, and SPARC microprocessor families.

A second alternative is Lineo's (formerly a part of Caldera) Embedix. Lineo's distribution of embedded Linux is engineered and reconfigured for the specific requirements of embedded devices. Currently available for PowerPC and x86 microprocessors, it's based on Caldera Systems' OpenLinux. Lineo plans to add support for MIPS, Hitachi, and ARM microprocessors later this year.

Furthermore, Embedix includes an SDK and an embeddable web browser. The SDK contains custom embedded Linux technologies and a comprehensive suite of native development tools for configuring, customizing, debugging, and testing components before combining them into an image for installation on the target platform.

There are hybrid alternatives too. Some traditional real-time operating-system vendors are attempting to ride the Linux wave with their own unique contributions (see "The Hybrid Alternatives," p. 108).

Linux makes a reasonably complete host development system right out of the box. For example, Linux servers from vendors like Caldera Systems provide all required network services. The desktop distribution incorporates TCP/IP, as well as the ability to set up and use network hosts. It also includes a graphical configuration utility called LISA which provides a means of configuring the network environment.

While most Linux desktop distributions make good individual development workstations, it's less well known that utilities included on the distribution CDs also provide full facilities for group development. One utility useful in laying the groundwork for collaboration between developers is the Network File System (NFS). This system lets workgroup members share directories across the network as though they were local. A software team can set up a master NFS file system that can be used as a repository for project code and documentation. There's also SAMBA, the Unix sever message-block facility, which allows shared drives with Windows systems.

Also, SLIP or PPP enables remote users to work as full team contributors, and permits individual team members to work while they are out of the office. Linux client systems can be set up for remote access using the LISA utility to add the remote host and appropriate protocol.

For group communications and information exchange, you might want to start with the Apache web server. With Apache, each team member can set up a personal web server, to post individual documents for viewing and downloading. In addition, it includes web development tools that allow individual users to post and exchange information. A web server on the network server can serve as a central point for group-wide documents and information.

E-mail also is given on Linux. The Linux server provides a POP and IMAP mail server, and the desktop distribution includes a POP client, along with the longtime mainstays Elm and Pine, and the metamail multimedia e-mail package. Sendmail may also be set up on the server to act as the mail server and gateway to the outside world. Sendmail will pretty much work out of the box for SMTP-only networks.

Some of the most interesting communication software available on Linux distributions enables a workgroup to set up a NNTP news server and list server. Setting up your own group-wide news server is an excellent way to establish threaded discussions among team members. When the news server is set up on the Linux server, each client can access it, read it, and post messages.

These components are simply the tip of the iceberg. A visit to the Free Software Foundation site, or Linuxcare, will turn up full-development IDEs, productivity applications, compilers for virtually every language, and source-code control systems, just to name a few. All this is before making a payment for any software.

Of course, there's another price to be paid for this free software. The most obvious price is the limited support options. While one may argue that the uncertain support for Linux software is at least as good as the demonstrably poor support provided by an existing tools vendor, it might be a case of "better the devil you know than the one you don't."

The less obvious price is that you and your team may want to contribute code and expertise to the Linux community, both because of an interest in doing so and because of recognition that contributions help the Linux alternative advance. This may require additional development work, and also cause conflicts with corporate goals to protect the work done in-house. When considering using Linux and participating in the Linux community, make sure your intellectual-property people are aware and approve of such participation.

Linux is still in the very early stages of use, as a development platform and a target operating system. It has some significant advantages, because of the wide availability of Linux source code, the free or inexpensive availability of high-quality development tools, and the rapid rate of advancement of Linux technologies. Embedded engineers will appreciate the ability to customize their own solutions without the need to build the operating system from scratch.

But, building an embedded product by customizing a Linux distribution is still a black art. There's little documentation on how to customize it. Engineers will have to learn the operating system inside and out before doing it effectively. Also, Linux was never built with footprint and scalability in mind. It may be a modular operating system, but that's out of necessity rather than design. Getting it down small enough for a specific embedded application might require more work than starting from scratch.

No one should embark upon an embedded project incorporating Linux lightly. There are some points to consider before dropping existing development plans to grab the coattails of the Linux revolution.

First, have a clear understanding of what you want to accomplish by using Linux in an embedded-systems project. Don't "do Linux" because it happens to be the technology of the month.

In all likelihood, you also are going to use Linux as the development platform, as well as the target operating system. Check out the Linux resources on the web to ensure that the tools you need for your project are readily available. Make certain that you have the language compilers, code libraries, debuggers, device drivers, and development utilities before embarking on the project. If you additionally need cross-compilers for your target, test them first.

Also, research available information and support resources. There's free support on the web, in Usenet forums, and through e-mail list servers. That support, while freely given and often highly accurate and insightful, can't be depended on to provide the answers you need when you need them. Make sure you can line up professional, paid support, with embedded development experience.

Set your expectations appropriately. If this is a first project, factor in time for the team to learn the operating-system internals, and get used to the tools. Don't expect miracles. After all, it's only an operating system.

Linux, as both a host and target platform, will undoubtedly become a major force in embedded development over the next several years. A growing number of developers are seeking access to small and fast database-management systems (see "Building Linux Databases," p. 110). Linux, however, is neither a miracle technology, nor a way to salvage a bad design. The rules that apply to other operating systems are the same for embedded development using Linux.

The open-source nature of Linux is a two-edged sword. It provides an almost unprecedented freedom in adapting the operating system and development tools, at a potentially a lower cost than other alternatives. But, like most freedoms, it should be used in moderation.