This article series is in the Improving Software Code Quality topic within our Series Library
What you'll learn:
- Commercially available vs. free dev tools.
- What is "schedule slipping"?
- What types of tools are best when it's time to debug.
The presumption is that using commercial tools will make the development process faster and therefore less expensive. Is this accurate? Let’s look at key factors for a “fully loaded” embedded development engineer.
Including hidden costs like vacation time, benefits, paid holidays, and such, an engineer costs a company in the U.S. approximately $200,000 per year, or about $4000.00 per week for pure engineering. Though this clearly varies from company to company, the calculations and variables remain constant. The goal in choosing between commercially available or free development tools is to determine whether you might save enough engineering time to justify the cost of the tools.
Less-Definable Costs
In addition to the cost of an engineer’s time, it's important to anticipate and take into account less easily quantifiable expenses. These may include costs associated with slipping schedules, tool support, debugging time, and the possibility of introducing mistakes when engineers are burning the midnight oil to try to meet deadlines.
Schedule Slipping
This is extremely hard to quantify partly because it includes related issues such as what happens when closing in on the ship deadline and the project isn’t done? In this case, companies often use two approaches.
One approach is to slip the delivery deadline. This has an apparent direct cost of lost immediate revenues. However, it also includes the harder-to-quantify cost of going to a market late and giving competitors the opportunity to beat you. The cost of giving up first-to-market advantage can be unimaginable and sometimes unrecoverable if you planned on being the market initiator and getting ahead of competitors.
The second approach is based on today’s common practice of providing on-the-fly updates to product software after it has shipped. In many technology products shipped today, the first thing a customer will do upon unboxing their new toy is to run a software update. This loads the newest features and bug fixes that were implemented between the time the product shipped from the factory and the time the customer sets it up. (This also is true for products that have been shipping for some time.)
In such a scenario, a product team may decide to keep the ship date immovable, and ship with a reduced set of features. This assumes that it’s more important to enter the market and add the other features later through software updates by the customer. It will work as long as the minimal feature that’s complete and tested at ship time is compelling enough to drive the product.
Of course, it’s better to avoid both of these scenarios.
And naturally, none of this addresses the real, but hard-to-quantify, costs of wear on an engineering team whose work hours increase as the delivery deadline approaches and work still needs to be completed. What is the cost of late nights and weekend work that’s sometimes needed to meet difficult delivery schedules?
Saving Time in Product Development
Several phases and tasks in product development aren’t affected by the development tools chosen. For example, the process of specifying the product, its features, and their importance are all time-consuming and important.
However, for the purposes of this discussion, we’ll assume that this is the same regardless of which tools are used. And, of course, we’ll assume that this process is done perfectly with minimal changes incorporated once the initial product definition is completed. (Did you see me smile when I wrote that?)
The next phase where dev time may vary is during development of the initial product code. Many companies assume that the configuration tools needed to initialize peripherals and chip resources, as well as third-party stacks such as communications protocols, USB port drivers, file systems, and more, are available for all toolchains under consideration.
However, two main categories may fluctuate considerably from toolchain to toolchain:
Tool Support
When an engineer encounters a problem during development, finding the issue can be a challenge and hard to assess. Is the problem caused by the design, a fault in the tools being used, or some other factor? If the tools are of unknown provenance, the immediate reaction may be to blame the tools and look for a workaround for the issue.
While some “free” tools come from silicon vendors with their own support staff to help their customers, the primary support mechanism often is a public bulletin board or online chat facilities where users can trade war stories and help each other out. In all fairness, many silicon vendors also cruise these sites and add helpful recommendations.
However, going this route to determine the cause of an issue can be difficult, time-consuming, and an unbounded process. And, if the problems are caused by the tools, determining whether it’s something that’s been seen by others and whether there’s a simple workaround, can be very difficult.
This is further complicated by the fact that response times can be unpredictable at best. And when a response is given, determining the quality of the response, and judging the capabilities of the person providing the answer, can be problematic.
Unlike free tools, a good commercial tool vendor will provide a live support capability that can log customer questions and track them to completion. This enables the customer to spend time focusing on their design and not fighting with tools issues.
Ideally, such a support entity can help with problems that aren’t in fact caused by a fault in the tools, but rather a misunderstanding by the engineer. The sooner the issue is understood and resolved, the sooner the engineer can get back to development work. This can save hours and even days when compared to bulletin-board based support of tools.
Some commercial tool companies have support organization that offers access by email, phone, and live video chat sessions to help customers quickly resolve their problems and keep moving forward.
Debugging
It’s been said that the best way to shorten debugging time is to not inject errors into your code in the first place. While it’s an excellent goal, it’s often hard to achieve in practice.
One way in which tools can assist with this goal is to offer add-ons, whether they’re included or optional, that provide automated code checking using techniques such as static code analysis. This process may be done by live teammates in code review sessions or automatically by commercial code-quality tools. Using both methods can be helpful in getting to a code base with fewer errors that need to be uncovered and fixed.
Assuming the process isn’t perfect, which it never has been in my experience, the next step for the engineer is to debug the code, uncover errors or bugs, and fix them before the product ships.
Based on personal experiences as well as customers’ feedback, a software engineer may spend an estimated 50% of their development time debugging the code. Anything that can be done to reduce this would be quite valuable both in saving direct costs for engineering time, as well as in keeping a project on schedule.
The process of debugging code is hard to define precisely. Its very nature is based on errors injected into software that’s unknown and can have undefined effects on the behavior of the system.
The start point is generally one where the system is turned on and it doesn’t do what the engineer intended. Determining exactly why that’s the case, and how to fix it, is the objective of debugging a system.
To shorten debugging time, the more a toolchain does in helping the engineer find and fix bugs, the greater its effectiveness. In other words, if a debugger package helps the engineer visualize what’s happening in the software and system, and if the tools are intuitive to use, the engineer can more quickly zero in on problems in the code or the system and find solutions.
This is one area where “free” tools can fall far short.
Commercially available debugging tools provide robust debugging capabilities that enable productivity of the engineering team that uses them. For instance, such tools typically have features that enable debugging tasks to be accomplished faster and reduce the bottleneck time to completion.
Furthermore, these tools allow the debugging process to be visual so that the engineers can better see and trace back where the bugs occur and what needs to be fixed. Being able to expose what’s happening in the system helps simplify the debugging tasks.
Usually, a debug window allows an engineer to view multiple sources of accurately timestamped debug data in the same time scale so that interaction of different events can be viewed. For instance, interrupt activity can be viewed at the same time as power consumption or the target system, along with noninvasive sensing of critical data values changing and other important events.
On this front, a central capability provided by the Arm CoreSight architecture is a unit called the ITM or Instrumentation Trace Macro-cell. This unit enables a programmer to write to a debug register, which will cause the debug logic to create a timestamped message that’s sent to the debug probe without any overhead on the CPU.
Therefore, with the overhead of writing to a debug register, similar to what would be needed to toggle a GPIO pin, a timestamped message can be sent to the debugger to be displayed on the commercial debug program. This capability can be used to measure the execution time of critical software routines or for complex functions like displaying task switch events in an RTOS-based system.
To debug more effectively, commercial vendors have appropriate hardware debug probes to connect to the target and intuitively display this information in many ways to help the engineer determine what’s happening.
This kind of visibility can be invaluable to engineers to see what’s working in their systems. And it’s even more valuable when it uncovers behaviors that aren’t expected. Just seeing that an expected interrupt isn’t firing can save many days of work trying to figure out why the system doesn’t respond as expected.
Summary
Controlling development time for a product is essential to managing costs and hitting delivery targets. By using commercial tools and services designed to help engineers quickly and efficiently create the products, engineers are able to meet their productivity and delivery goals.
Using commercial tools with an upfront cost, as compared to “free” tools offered to minimize barriers to using specific chips in product designs, can be an effective way to keep on schedule and minimize the overall cost of development of a product. This perhaps counterintuitive conclusion is borne out by the experience of many development teams over the years.
Read more articles in the Improving Software Code Quality topic within our Series Library