Download this article in PDF format.
A traditional debugger allows you to inspect the system state once the system is halted, i.e., after an error has been detected, but doesn’t reveal the events leading to the error. To see this information, you need tracing.
Tracing means that the software’s behavior is recorded during run-time, allowing for later analysis of the trace. A hardware-generated trace (in the processor) gives you all details regarding the control-flow and doesn’t impact the execution of the traced system. However, it requires a special trace debugger unit and trace-enabled hardware in general. Software-generated tracing works on any hardware platform, and tracing can be active continuously during long testing sessions. Software-based tracing can even be deployed in production systems in many cases.
The software trace will use CPU cycles and RAM of the traced system, though, but this is often a reasonable tradeoff given the value of the resulting traces. Unless you have very strict timing requirement, down to the microsecond level, software tracing is simple, straightforward, and a perfectly viable solution. Let’s explore some of the myths surrounding this approach.
1. Tracing requires an advanced trace debugger.
This isn’t necessarily true; there are different kinds of tracing. Tools like the Linux Trace Tool (LTTng) and Percepio’s Tracealyzer rely on software-based tracing for systems using an RTOS, allowing for the capture of all relevant events in the RTOS kernel and any extra events you add in your application code. This kind of tracing doesn’t require any special hardware and can be used on essentially any system. The data is generated by software and stored in a local RAM buffer, or streamed to a PC using any communication port with sufficient performance.
2. Tracing is complicated to set up.
Software-based tracing is generally quite easy to set up—it’s typically done over a cup of coffee. You need to include the recorder library in your build, configure it to match your hardware, and enable the trace option in your RTOS. Then you’re ready to start tracing.
3. Tracing is the last resort when debugging.
Some think of tracing as a fire extinguisher: When you can't find the bug in other ways, you try tracing. However, a good tracing tool is better viewed as insurance. Sure, adding trace support to your system consumes some of your precious clock cycles and bytes, but that small premium buys you a tool that will help you understand, debug, and optimize your software system. You may find irregularities long before the underlying bugs manifest themselves in customer operation.
Event tracing is a valuable complement to traditional debugging tools, helping you to “zoom in” on the cause of complicated issues, so that you know where to focus your further debugging efforts. No more frustrating debug sessions that are done blindly.
4. Traces are overwhelming and hard to understand.
Trace logs can certainly be difficult to interpret, especially without proper visualization. It’s also a fact that most of the data consists of repetitive patterns, many of which are irrelevant to the issue at hand. Finding the needle in this proverbial haystack can be an arduous task.
More advanced trace tools have dozens of graphical views that will let you view the system from different perspectives and at various levels of abstraction, allowing you to “drill down” on anomalies and understand the cause. This makes it much easier to overview your traces and draw conclusions from them.
5. RTOS tracing is just for kernel hackers.
Totally not true. Application-focused developers can benefit a lot from RTOS-level tracing, since you can log custom events and data (e.g., inputs, outputs, state transitions) to facilitate analysis of algorithms and state machines in your application. This kind of tracing is very fast and therefore can also be used in time-critical code, unlike traditional "printf" calls.
Tracealyzer can visualize run-time interactions between application tasks in the form of dependency graphs (who is talking to who) and filtered timelines that gives a better picture of the application design, a complement to analyzing the source code. In addition, Tracealyzer can profile your application performance with respect to processor time and memory usage per task and interrupt handler—not just static averages, but visual graphs over time.
6. Trace tools are expensive.
On one hand, that is true. Good trace tools cost good money, especially compared to those developer tools you can download for free. On the other hand, being able to find bugs earlier in the development process can save you a lot of money. A bug that slips into production code or a delayed product release can cost magnitudes more than the license fee of a premium tracing tool.
7. Tracing is only for hard real-time, safety-critical systems.
Not true. Less-critical embedded systems are often larger and have more code and more complex run-time behavior compared to safety-critical real-time systems. Even though the consequences of a failure are less severe, their behavior can be much more difficult to predict from the source code alone due to multithreading, preemptions, and shared resources. This calls for trace analysis to help solve elusive bugs and performance issues.
8. Tracing is mainly used for analyzing the code coverage of test cases.
Tracing is indeed used for analyzing the code coverage, but for that you use hardware-based tracing on the instruction level. Software-based tracing is a quite different thing, focusing on relevant software events only and with any relevant data included. This is a more flexible analysis method that can be used in a wide range of situations, including debugging, validation, and performance optimization.
9. Software-based tracing is intrusive and slows down the system.
Once again (see Myth #3 above), there’s some truth to this. Tracing does add a bit of code to your application, typically a few hundred clock cycles per event. But since you record relatively few events (typically a few thousand per second), the total effect is often next to negligible, usually a few percent extra CPU load. But software tracing may not suit every system; if the cycle times or timing requirements are measured in microseconds, software tracing may be too intrusive.
10. Tracing requires special trace support in the processor and on the board.
This is definitely true for hardware-based tracing, but software-based tracing does not require anything particular from the target system. Most systems can allow for tracing in “snapshot mode”, i.e., when storing the data to an internal RAM buffer. This does however limit the trace length to the available RAM; a few seconds at most, if not using external DRAM. For longer traces, the target needs a decently fast connection to the host PC such as a debug port, TCP/IP network or USB connection.
11. Tracing to a RAM buffer requires lots of RAM.
Not true. Many trace tools encode events in a very compact format. For example, Percepio's trace recorder uses only 4 to 8 bytes per event, and it supports ring-buffer mode where older data is overwritten when the buffer becomes full. It can also filter trace data at the source, before going into the buffer. This makes tracing useful even on MCUs that only has a few kilobytes of RAM available. Such a small trace buffer won't buy you very long traces, but you can catch the latest events leading up to the current state.
