Dynamic languages have proved invaluable to improving fast and agile development, most notably for Web development and system administration. Some precursor languages such as Unix shells, Perl, or PHP have paved the way and remain widely used today. More modern, rationally structured languages such as Python and Ruby are now inescapable for people who want high productivity for tasks such as Web 2.0 development.
However, these improvements haven’t yet been widely adapted to software development for embedded devices. Although this adoption delay was once justified due to technical issues, based on our experience in machine-to-machine (M2M) communication systems, the Lua programming language is very well suited to overcoming this historic reticence.
Choosing The Best Suited Language
A programming language consists of three things:
- A core (the language’s grammar and semantics), embodied by its interpreter or compiler
- A set of standard libraries, provided with the interpreter or through a central repository, optionally through some package management software
- A community, which writes and maintains the libraries, deciding the proper way to develop with the language and influencing the evolution of the core
Although some details vary, Python, Ruby, Javascript, and Lua share pretty similar cores. They’re heavily influenced by common ancestors such as Scheme and offer more or less the same structuring mechanisms: scoped variables, runtime type checking, loops, first-class functions, classes, and objects. Each has some distinctive features, but there are much more common traits than differentiating ones, especially when one focuses on the language’s core.
Among the best known dynamic languages, Python shines on two points. First, it has a strong community culture that focuses on enforcing a maintainable coding style. Non-conforming code is dismissed as “unpythonic.” Second, it comes “batteries included,” with an impressive corpus of standard libraries, both quantitatively and qualitatively.
Ruby came to widespread fame thanks to Rails, which reinvented the way we do Web development. It leverages meta-programming concepts that were before considered the realm of academics and hackers and dramatically lightens the development cycle through advanced automated features such as scaffolding, integrated tests, migration management, and object-relational mapping.
Javascript’s main interest is that it runs on every browser and very efficiently thanks to huge investments in virtual-machine development.
Lua runs fast on a shoestring, and it interfaces very well with C. It’s the de facto standard for video game programming, where high-level logic is often left to level designers who aren’t professional developers, yet must integrate seamlessly with highly optimized 3D and physics engines written in C++. Secondary features include coroutines, a very efficient way to control concurrency.
Once it had been decided that we wanted dynamic languages’ productivity for embedded development, Lua appeared to be the best choice for almost every objective metric:
- It allows complex applications to run on very cost-effective systems: a couple hundred kilobytes and a CPU clock with a couple of megahertz, with no full-fledged operating system underneath. It can do without a file system, OS-level threads, and processes.
- The other languages’ strengths aren’t applicable to the embedded cases. While Python and Ruby have great libraries for their typical application domains, M2M communications isn’t one of them. Embedded systems largely remain uncharted territory for dynamic languages, their communities, and the libraries they provide. As for Javascript, running in a Web browser isn’t relevant for automated embedded devices.
- Tight C integration allows developers to reuse existing code. M2M traditionally works with C, if not assembly language. It also lets developers optimize ruthlessly. Instead of tweaking critical Lua code, we can simply translate it into C. Since such critical parts tend to belong to core libraries rather than to the final application, most users will never have to deal with such issues.
- It is easier to find seasoned Python or Javascript developers than Lua developers. However, experience consistently shows that migrating from one dynamic language to another is much faster than learning a new application programming interface (API) or framework. Experienced professionals typically become productive with Lua in a matter of hours.
A Deeper Look Into Lua
Although it firmly belongs to the generic-purpose dynamic languages family, Lua has a couple of defining features worth mentioning. A striking one is its “everything is a table” paradigm. A very powerful table object is used as a universal container, subsuming lists, arrays, hash tables, sets, records, objects, classes, and modules. Its implementation is carefully optimized so each of these roles is fulfilled not only intuitively, but very efficiently.
Unlike Lisp, which used the “everything is a list” paradigm and tended to frighten inexperienced developers with what they saw as shapeless oceans of parentheses, Lua provides adequate syntactical support for table manipulations so it looks like what most developers would describe as a normal programming language.
Beyond atomic types such as numbers or strings, the other Lua-specific type is the _userdata_, which allows developers to manipulate arbitrary C data from Lua in a type-safe and memory-safe way. It obviously remains possible for the underlying C code to crash, but every necessary mechanism is there to provide a crash-proof API to this code from Lua.
Providing a C API that would allow the Lua virtual machine to crash is considered a critical design bug. Most established embedded companies have large repositories of C code, so powerful interfaces allowing them to reuse these assets is vital.
Letting Lua Handle Multitasking
Lua also offers direct access to coroutines, which essentially are first-class call stacks that can be started, stopped, and rescheduled as needed without being attached to any OS-level thread or process. They enable system developers to roll out their own tailored scheduling system, with no prerequisite on the underlying OS, and without bothering the regular users.
Most M2M programs spend most of their time waiting for I/O events. This is typically addressed either through callback mechanisms or by using many system threads. Callbacks can be very efficient, but at the cost of program readability. Lines of code located next to one another generally don’t handle events that occur one after the other. Figuring out and maintaining a callback-driven program’s execution flow is no small achievement. Running lots of threads that spend most of their time sitting on resources and waiting for rare events is more readable, but it bears a cost in computing resources that often cannot be afforded, especially on embedded devices.
Coroutines offer a very elegant solution to this dilemma. They run with an efficiency close to that of callback-driven frameworks (less than a kilobyte overhead per coroutine), but they allow the code to be organized in a sequential, maintainable way. At the core of Sierra Wireless’ ALEOS Application Framework is a Lua collaborative scheduler, used both by core services and by user applications.1 It’s tailor-made for the kinds of applications that make M2M systems tick.
Being collaborative, the scheduler only switches tasks when a task waits for an I/O, which greatly reduces the concurrency issues that typically plague multithreaded systems. Switching only on I/O operations works great in practice, because M2M applications typically spend all their time moving data across I/O channels. Of course, it remains possible for long-running computations to voluntarily release the CPU, but this is almost never needed in practice.
To conclude, the tremendous productivity gains provided by dynamic languages have yet to be enjoyed by the embedded development world, and Lua is for many reasons the best vehicle to deliver these gains.
Reference
- Sierra Wireless’ ALEOS Application Framework is a Lua framework and tooling for M2M applications development, based on the Koneki open-source project.