Faulty Code Will Lead To An Era Of Firmware-Related Litigation

It seems like embedded software is turning up everywhere—even in courtrooms. Have you heard about the Drager Alcotest 7110 breath analyzer? After a two-year fight, seven defendants in New Jersey DUI cases won the right to have experts review the source code for the Alcotest 7110 firmware. Each side produced expert reports evaluating the quality of the firmware source code.

Although the experts disagreed about the code quality, they still found several flaws in the process. For example, the firmware first averages two blood alcohol readings. Then it averages the third reading with that average. Then it averages in the fourth reading, and so on.

The formula gives the last reading a weight of 0.5 in the final value and the next to last value a weight of 0.25, which is an unusual way to compute an average. In addition, static analysis with lint produced more than 19,000 warnings about the code, or about three for every five lines. So, what might you infer about the reliability of a defendant’s blood-alcohol reading if you could look over the shoulder of the test system’s creators?

Human health and wellbeing are both enriched by and increasingly dependent upon proper functioning of the firmware inside electronic systems. And consumers are beginning to notice the presence of software because of the need for frequent reboots and upgrades. There is no end in sight, as several billion more such devices are produced each year.

Lawsuits are on the rise too. In recent years, I have been called into U.S. District Court as an expert witness in several dozen lawsuits involving embedded software. Popular claims range from copyright theft and patent and trade secret infringement to traditional product liability with a firmware twist. Unfortunately, the quality and reliability of our collective firmware leaves the door open for more cases caused by faulty firmware.

SPOT THE BUGS

At the most recent Embedded Systems Conference in Boston, I gave a talk titled “This Code Stinks! The Worst Embedded Code Ever” that used lousy code from real products as a teaching tool. A number of engineers from a broad swath of companies gathered the example code over several years. For instance:

y = (x + 305) / 146097 * 400 + (x + 305) % 146097 / 36524 * 100 + (x + 305) % 146097 % 36524 / 1461 * 4 + (x + 305) % 146097 % 36524 % 1461 / 365;

I don’t know if the above snippet contains any bugs, as most of the other examples did, which is also a problem. Where are we supposed to begin an analysis of this code? What is this code supposed to do when it works? What range of input values is appropriate to test? Is this code responsible for gracefully handling out-of-range inputs?

In the original listing, there were no comments on this line to help. I eventually learned that this code computes the year, accounting for extra days in leap years, when given the number of days since a known reference date. But we still don’t know if it works in all cases, despite its presence in an FDA-regulated medical device. The Microsoft Zune Bug was buried in a much better formatted snippet of code that performed a very similar calculation.

Before you think these examples are exceptions to the norm because we were on the prowl for bad code, consider a couple of well-publicized failures.

First, recall the Patriot missile failure in Dhahran, Saudi Arabia, during the first Gulf War. Twenty-eight U.S. soldiers were killed when a Patriot missile battery improperly tracked a Scud missile and failed to shoot it down near a military base. A GAO report examined the events leading to the failure and concluded the problem was partly in the requirements given to the code’s designer: the government didn’t say the system would need to “operate continuously for long periods of time.”

“At the time of the incident, the battery had been operating continuously for over 100 hours,” the report said.

Or, consider that GPS maker Garmin recently announced a “free, mandatory GPS software update to correct a software issue that has been discovered to cause select GPS devices to repeatedly attempt to update GPS firmware, and then either shut down or no longer acquire GPS satellite signals.” This sounds like a bug in the bootstrap loader, or bootloader. Garmin said many GPS units were affected.

LIABILITY BEFORE RELIABILITY

I foresee an era of widespread firmware-related litigation. Once lawyers understand the poor state of embedded software, they will descend on these cases like locusts. If we are lucky, this will lead to increased firmware reliability. For that to happen, our peers must improve the groundwork in embedded software process and architecture.