Yes, You Can Do Digital Audio With Real-Time Java

Java has become the preferred programming language in the traditional information-technology domain. That's because it offers improved developer productivity, greater software reuse, lower software-maintenance costs, more flexible and general software architectures, and higher software reliability.

Now, various approaches to using Java in lower-level, hard, real-time software realms have been proposed. But when Java technologies are applied to very low-level software, such as digital audio-signal processing, certain approaches deliver more of the traditional Java benefits than others.

One such approach is based on a proposed specification for a resourceconstrained and safety-critical Java definition. The driving objective in designing this approach has been the maintenance of Java's portability, maintainability, and scalability benefits. The information flow within the system includes two computers cooperating to facilitate exchanging audio information via network communication channels (Fig. 1).

Audio signals collected at one node are transmitted to the other node and output to the speakers on the remote computer. Audio signals collected at the second node are output on the speakers of the first. Conceptually, the information flow is structured as two independent streams of digital audio data.

This simple audio-processing application can be implemented as a PERC Pico application. The software, currently under development, represents the first implementation of the proposed Real-Time Specification for Java (RTSJ) profile for safetycritical and resource-constrained applications. "Hard real-time profile" refers to this environment.

MAINTENANCE AND SCALABILITY REQUIREMENTS Moore's Law helps explain the rapid growth in the size and complexity of typical embedded applications. Competitive pressures encourage software to expand to fill the capacity of ever-more-capable hardware. Studies of certain consumer electronic devices reveal that the amount of code in new product releases closely tracks Moore's Law, roughly doubling in size every 18 to 36 months.

Twenty years ago, it was common for all software in each new embedded device to be written by one or two engineers in less than a year's calendar time. Modern embedded software development is very different. Given that each new product revision incorporates hundreds of thousands, if not millions, of lines of additional code, the embedded software developer's responsibility is now shifting more toward the challenge of integrating many independently developed software components.

This simple digital audio example represents a prototypical low-level embedded software "product." For most products, the cost of developing the original software is small compared to the cost of maintaining the software throughout the product life cycle. Consider how this application would likely evolve during the product's lifetime:

• The software would need to be ported to different operating systems and different processing platforms. This will change its CPU time and memory requirements.
• The software would be integrated with different mixes of complementary functionality. Perhaps the next-generation product would include video signals, too. Maybe it would support the sharing of digital whiteboards to facilitate remote conferencing or possibly become integrated with e-mail and calendar software. Or, some uses may add a recording facility to capture a session to disk.
• The two-node network topology may need to be generalized to support the conferencing of arbitrarily large groups of participants.
• The interface to analog-to-digital converters (ADCs) and digital signal processing (DSP) may evolve. In some configurations, an operating system provides this service. In others, this application might include the devicedriver interface to the audio subsystem hardware and DMA memory devices. The audio hardware itself is likely to evolve, requiring the software device drivers to evolve.
• The network communication protocols may need to change. In some circumstances, the software will rely on underlying operating-system services to interface to the network. Even the interface to operating-system network services is likely to change as various network communication protocols evolve, yielding new quality-of-service parameters and higher bandwidths. In other cases, this application will need to include the low-level device drivers for the hardware interface and may need to implement the communication protocol stacks. The same fundamental communication capabilities might be implemented on low-cost dedicated serial channels, coax and twisted-pair data links with carrier-sense multipleaccess/collision detection (CSMA/CD) technology, wireless, fiber-optic, and other media that's still to be invented. The communication libraries may incorporate compression, encryption, error detection and correction, and sliding window protocols.

This list identifies some of the many ways in which this software would likely evolve if it were put into commercial service. The list isn't intended to be exhaustive, but rather to illustrate the benefits of retaining the design advantages of Java, even for resource-constrained and hard real-time applications.

REAL-TIME JAVA CAPABILITIES These real-time Java programming technologies derive from the RTSJ. The specification allows for considerable generality to support a wide variety of distinct realtime programming requirements. Since the article focuses on very low-level realtime software, we constrain developers to a subset of the full RTSJ specification.

This profile improves portability, reliability, and efficiency because it forbids the use of features that incur high run-time overheads, expose non-portable implementation dependencies, and introduce software complexity that's more prone to programmer error. Some of the specific differences between the hard real-time profile and the full RTSJ include:

• The full RTSJ uses priority inheritance for synchronization locks and makes support for priority ceiling emulation optional. The hard real-time profile forbids the use of priority inheritance and requires support for priority ceiling emulation.
• The full RTSJ allows various thread scheduling and object synchronization parameters to be modified on-the-fly. The hard real-time profile forbids onthefly adjustments to thread scheduling and synchronization protocols.
• The full RTSJ supports mechanisms to automatically fire asynchronous events whenever a task misses a deadline or overruns its CPU-time budget. Note that the implementation of these services is very non-portable, and precise enforcement imposes a very high run-time overhead. Furthermore, there's no need for this run-time enforcement in a hard real-time application, because compliance with resource budgets and deadlines is enforced statically—prior to execution. As a result, the hard real-time profile doesn't support these mechanisms.
• The full RTSJ supports the mixing of traditional threads (java.lang.Thread), realtime threads that access the garbage-collected heap (RealtimeThread), and real-time threads that don't access the garbage-collected heap ( NoHeapRealtimeThread). This intermingling of different thread types adds considerably to system complexity and size. Such complexity increases the likelihood of realtime programming errors that result from incorrect sharing of information between the different thread types. The hard realtime profile only supports real-time threads that don't access the garbagecollected heap.
• The full RTSJ provides a set of libraries used by application programmers to instantiate dynamic memory scopes and to allocate objects within particular scopes. Use of these library services is especially problematic because programmers may make many subtle errors when developing or integrating components that use nested scopes. To enforce proper scoped-memory usage protocols, the RTSJ performs special run-time checks every time a reference field is fetched and/or overwritten. In the full RTSJ, run-time checks may cause a program component to fail, aborting execution with a run-time exception because of illegal assignments, illegal fetches, scoped-memory protocol errors, out-of-memory errors, or memory fragmentation errors. The hard real-time profile forbids the use of the RTSJ memory scope manipulation libraries. Instead, it requires programmers to describe their scoped-memory usage in terms of programming annotations that can be analyzed and enforced at compile time. The @Scoped and @StaticAnalyzable annotations illustrated in this sample application are examples of these annotations.
• The RTSJ doesn't standardize libraries for interrupt handling or low-level device I/O. The hard real-time profile defines these libraries.

Experimentation with a pre-commercial implementation of the hard real-time profile demonstrates that it runs certain CPU-intensive benchmarks over three times faster than standard Java and full RTSJ implementations. This is because the hard real-time execution environment is much simpler than standard RTSJ, and because it replaces various run-time checks with compile-time verification. This performance is comparable to, and sometimes even superior to, equivalent C and C++ programs.

Even though using the constrained hard real-time profile is more difficult than traditional Java, developing and maintaining code for this platform is far easier than doing equivalent development in C or C++. This is because the hard real-time Java platform is more portable and offers superior object-oriented abstractions. Furthermore, the hard real-time Java platform includes important development tools that ease the development, maintenance, and integration of real-time components (Fig. 2).

Compared to C and C++, Java development improves reliability and maintainability, thanks to the inclusion of a byte-code verifier that enforces strict compliance with type safety. C and C++ programmers have a variety of mechanisms that let them defeat type safety, and intentional or accidental exploitation of these loopholes makes the code more error-prone and less portable.

The constrained real-time environment supplies even more rigorous byte-code verification than traditional Java. In particular, the Hard Real-Time Verifier in Figure 2 ensures that references (pointers) to stack-allocated objects never live longer than the objects themselves. It also ensures that program components flagged with the special @StaticAnalyzable annotation restrict their use of Java to the analyzablesubset. Integrated with the Hard Real-Time translator is the ability to determine upper bounds on the amount of CPU time and stack memory required to execute each component.

All of the temporary memory allocation required for execution of hard real-time components must be satisfied from the executing thread's run-time stack. Execution begins with a single main thread, and that main thread's run-time stack represents all reusable memory. For each additional thread spawned by the main thread, it provides a portion of its runtime stack to serve as the run-time stack of the spawned thread.

DIGITAL AUDIO APPLICATION IMPLEMENTATIONFigure 3 depicts the Digital Audio Application architecture. There are six threads in total, including the main thread and the asynchronous event handler represented by the Orchestrator instance. A BufferPair connects each socket interface to the corresponding DSP interface. The main thread monitors user directives and invokes the terminateActivity() method of the SimpleAudio instance when the user asks to shut down the session. All other threads periodically poll the SimpleAudio instance by invoking its continueActivity() service. This method returns false when it's time to shut down.

In the default configuration, the application samples microphone input at 8 kHz, gathering 8 bits of data with each sample. This configuration, which generates 8 kbytes of digital audio data every second, is quite adequate for simple voice applications. However, it's not appropriate for highfidelity stereo signals. A typical CD recording samples 16 bits for two stereo channels at a sample frequency of 44.1 kHz. The bandwidth requirements for this high-fidelity signal would be 176.4 kbytes/s.

In the default configuration, the socket reader and writer modules assume reliable transport with sufficient bandwidth capacity to reliably deliver all the data collected from the DSPReader modules. We implement a straightforward compression technique, under which a sequence of identical byte values (like those that might occur during a period of silence) are represented by a special escape value, the repeat count, and the repeated value. More sophisticated compression techniques would be much more appropriate.

In a real-time system, jitter characterizes the deviation from the desired ideal execution time for a particular real-time component. A distinct thread represents each of the components of the digital audio application. The SocketWriter thread accepts a stream of raw data from the DSPReader module, compresses this data, and transmits the data to the network socket channel. If the network socket channel is bandwidth-limited and can carry no more than the budgeted 8 kbytes/s, any jitter effects that cause the SocketWriter to delay data transmission will accumulate over time.

For the default configuration, the SocketWriter is expected to transmit one byte of data every 125 µs. If one byte in each second of audio data is delayed by only half a millisecond, after one hour, the accumulated delay will be roughly two seconds. To prevent the accumulation of jitter delays, the architecture includes a supervisor thread that runs at 16 Hz.

In each period, this thread forces the SocketWriter and DSPWriter components to discard any data that's more than 62.5 ms old. Since we're dealing with audio data, it's generally preferable to drop occasional data values rather than allow the timeliness of delivery to drift. The impact of dropping occasional data bytes typically won't be noticed. The source code for the main program (Simple Audio Main Program) of this sample application is available at www.electronicdesign.com, ED Online 12432.

Note the @StaticAnalyzable annotation that appears on line 1, @ StaticAnalyzable(enforce_time_analysis = \\{false\\}, enforce_non_blocking = \\{false\\}) of the source-code listing. This represents part of the method signature. Note that this annotation is parametrized with values for the enforce_time_analysis and enforce_non_blocking attributes. Both are indicated as false. This signifies that the implementation of this method needn't constrain itself to the subset for which the static analyzer can derive a strict upper bound on the amount of CPU time required to execute this method, nor is the static analyzer required to prove that execution of this method never blocks.

Were these attribute definitions not present, the Hard Real-Time Verifier would reject this program as illegal because the static analyzer cannot determine how many times the loop comprising lines 55, while (!orchestrator.destroy()) \\{ through 57, \\} of the source-code listing executes. Furthermore, execution of this main method may block at lines 59, socket_ reader_thread.join(); through 63, orchestrator_thread.join(); and within the await-Termination() method invoked from line 51, sa.awaitTermination();.

Not noted in the @StaticAnalyzable annotation is the value of the enforce_memory_analysis attribute. The default value of this attribute is true, meaning that the implementation of this method must conform to the restrictive guidelines that enable the static analyzer to determine the maximum amount of memory to be allocated when executing this method. Given that this environment's real-time Java conventions organize this memory as part of the run-time stack, the upper bound on temporary memory allocation represents the required size of the main thread's run-time stack.

Annotations contribute to software development and greatly simplify software maintenance. Usually, system architects divide a complex system of capabilities into smaller components to be implemented by different teams of developers. Therefore, the interface definitions that describe connections between various components clarify not only the types of arguments that can be passed between components, but also the constraints on real-time behavior that must be implemented within each component. This has a big payoff during software maintenance as well.

Modifications to the existing software must adhere to all other special real-time constraints represented in the component interface annotations. If software maintainers violate these interface requirements, they get immediate and specific feedback from the byte-code verifier. This ensures that incremental changes to an existing large system of real-time software won't destabilize the existing system.

In analyzing the stack memory required to reliably run this main program, the static analyzer must determine how much memory is required by each object allocated within this method and within the methods-that are invoked from this method. To support modular composition of static analysis results, the byte-code verifier needs every method invoked from the main program to be declared as @Static-Analyzable, with the enforce_time_ analysis attribute set to true. A quick review of the main method's implementation validates that no allocation occurs within unbounded loops. That's one of the considerations the byte-code verifier enforces.

Several new ThreadStack objects are allocated on lines 37, socket_reader_thread = new Thread-Stack(SocketReader.class); through 41 orchestrator_thread = new ThreadStack(Orchestrator.class);. Each allocation represents the stack memory to be used by the threads spawned by the main program. In general, it might be difficult for a static analysis tool to determine the amount of stack memory required to reliably execute these subthreads.

The argument to each ThreadStack constructor is the Class that supplies the code to be executed by the corresponding thread. The static analyzer requires each of the NoHeapRealtimeThread subclasses passed in this context to have a run() method declared with the @ StaticAnalyzable annotation and the enforce_ memory_analysis attribute set to true. If the ThreadStack constructor's argument derives instead from BoundAsyncEventHandler, as in the case of Orchestrator class, the static analyzer requires this class' asyncEventHandler() method to be declared with the @StaticAnalyzable annotation and the enforce_memory_analysis attribute set to true.

All temporary memory needs are satisfied from the run-time stack of the current thread. Note that we allocate two temporary BufferPair instances on lines 23, microphone_stream = new BufferPair(); and 24, speaker_stream = new Buffer-Pair();. References to these objects are then passed to the constructors for the threads that comprise the various functional components of this software application. One of the constraints enforced by the Hard Real-Time Verifier is that references to stack-allocated objects never survive longer than the referenced object itself. This also is enforced through an annotation mechanism. Consider the constructor for the SocketReader class:

   @ScopedPure
@StaticAnalyzable(enforce_time_analysis = \\{false\\}, enforce_non_blocking = \\{false\\})
SocketReader(SimpleAudio sa, Buffer-Pair buffers, String socket_name) throws
FileNotFoundException

The @ScopedPure annotation denotes that each of the incoming reference parameters to this constructor may refer to objects that reside on the run-time stack of a more outer-nested scope. The byte-code verifier ensures that the contents of these parameters are never copied into variables that aren't likewise distinguished as having the @Scoped designation.

Furthermore, it prohibits the copying of values from inner-nested scoped variables to outer-nested scoped variables. One exception is under special circumstances, where it can demonstrate that the referenced object resides in a scope that's at the same or more outer-nested level than the variable to be assigned. If this constructor's parameters hadn't been designated with the @Scoped annotation, the byte-code verifier would not have allowed the main program to pass references to its stack-allocated BufferPair and SimpleAudio objects.

One of the RTSJ-supported real-time programming abstractions demonstrated in this application is the PeriodicTimer class. Note that this application instantiates a PeriodicTimer object at line 49, drumbeat = new PeriodicTimer(start_time, period, orchestrator); and assigns the result to the local drumbeat variable. One of the arguments is a reference to the orchestrator object, which itself is an instance of BoundAsyncEventHandler. This drumbeat periodic timer is configured to trigger execution of orchestrator's handleAsyncEvent() method 16 times per second, which is once every 62.5 ms.

Real-time developers who use the C or C++ languages can implement many of the same constructs supported by these real-time Java technologies. However, C and C++ programmers must be trusted to avoid creating dangling pointers and memory leaks. They also lack standard tool support to automate the analysis of execution times and stack sizes.

Additionally, there's no integrity checking to ensure that method implementations satisfy the documented real-time interface requirements and that method invocations pass arguments that likewise satisfy documented interface requirements. Finally, during the maintenance of existing software systems, the C and C++ programmer has no tool support to guarantee that modifications to the existing software are compatible with the compositional requirements assumed during the development of the original software.

Traditional Java delivers many productivity and cost benefits. Disciplined use of real-time Java technologies offers many of those same benefits. Compared with the use of C and C++, typical Java developers are about twice as productive during development of new code and five to 10 times as productive during maintenance of existing software. As embedded realtime software grows in size and complexity, the factors that motivate a switch to more modern software engineering technologies, such as those that are enabled by real-time Java, grow in importance .