Applying Exhaustive Static Analysis to Automotive Software Testing

This article is part of the TechXchange: Developing High-Quality Software.

What you’ll learn:

How exhaustive static analysis overcomes the limitations of traditional tests and static-analysis tools.
How exhaustive static analysis identifies a buffer overflow by using code samples.
How hardware awareness improves the accuracy of software testing.

Automotive software developers hold the belief that verification, validation, and compliance activities can never approach 100% code and inputs coverage. This conviction is fueled by the rising complexity in vehicle systems, with distributed architectures, connectivity, virtualization, and other needs pushing the envelope of testing capacity and compliance to standards like ISO 26262 and ISO 21434.

Recent advances in academic research and computing power have proven that 100% code and inputs coverage is possible, and its benefits are now available to safety- and security-critical automotive development teams.

Exhaustive Static Analysis Guarantees Code Correctness

To bridge the gap between “sufficient” code coverage and 100% coverage, exhaustive static-analysis tools use mathematical models and techniques to verify properties and behaviors of software against precisely defined specifications. Known in research circles as formal methods, these techniques prove that code is free from issues like bugs and security vulnerabilities, and they overcome the key limitations of traditional tests and static-analysis tools:

Coverage: As software complexity grows, it becomes increasingly harder for test teams and traditional tools to cover the code base in levels sufficient for compliance activities, including functions, statements, paths, decisions, and conditions.
Scale: The more units under test—features, components, libraries, and functions—the more time and resources are required to test them.
Speed: Shorter release cycles are often at risk due to the inability of traditional test development and execution practices to keep pace with software complexity, forcing a tradeoff between test scope and time.
Proof of compliance: Identifying and eliminating undefined behaviors and security risks, as expected by the ISO 26262 and ISO 21434 standards, is difficult to achieve due to growing code complexity and the number of certification components.

Exhaustive static-analysis tools are designed and proven to integrate the power of formal methods into the existing test and compliance processes of automotive teams. These tools hold several advantages over traditional testing and static-analysis methods:

Up to 100% application coverage that includes all possible functions, statements, paths, decisions, and conditions.
Up to 100% input coverage, including all possible values within the scope of the unit under test.
A mathematical guarantee of the absence of undefined behaviors (errors and vulnerabilities) in code, resulting in zero issues at deployment time.
Zero false negatives, ensuring all issues are found.
Low to zero false positives such that developers aren’t overwhelmed by analysis findings and don’t waste time chasing non-issues.
Provide proof that supports ISO 26262 and ISO 21434 certification.

Figure 1 illustrates how exhaustive static analysis improves test coverage, showing how traditional “best effort” test design executes only one code branch per run versus all branches in parallel per run. The use of formal methods enables the running of tests for all possible inputs and leads to 100% code and inputs coverage, with dramatically reduced test time.

1. Code paths are compared between traditional testing methods (left) and exhaustive static analysis (right). Visited segments are in orange; unvisited segments are in black.

How Exhaustive Static Analysis Identifies a Buffer Overflow

Memory buffer problems have long plagued embedded software developers, and the buffer overflow exploit remains a top automotive vulnerability. This exploit occurs when the bounds of allocated memory aren’t checked during read and write operations. It leads to the application “overflowing” the capacity of the buffer, impacting areas not intended for data extraction or modification.

The following code sample illustrates a C function that increments cell values in an array:

void increment_array(int array[], int len) 
{ 
    while (len >= 0) 
    { 
        (*array)++; // Increment the value of the array cell 
        array++;    // Move to next array cell 
        len--;  // Decrement counter 
    } 
} 
int main(int argc, char *argv[]) 
{ 
    int data[4] = {1, 3, 5, 7}; 
    char name[] = "foobars"; 
    increment_array(data, 4); // Increment array 
}

A traditional test would validate the function’s requirement to increment the cell values in the output array and report a pass or fail based on the result. The test designer may not consider checking whether the loop end condition—and therefore the array index—caused an out-of-bounds memory access due to unexpected or undesired side effects in the system. This does occur in this code sample due to an improper condition specified in the while loop.

A traditional test against requirements would miss the buffer overflow condition and report that the array is {2, 4, 6, 8} after calling the function. Thus, it would always pass, as shown in the following console output from an example test run:

gcc -I. increment.c -o ub && ./ub 
Run test_increment_array() 
increment_array({1, 3, 5, 7}) = {2, 4, 6, 8} 
--> PASSED

Unless the test designer considered the possibility of an out-of-bounds array access, the test would never report the buffer overflow.

This subtle flaw could cause memory corruption, leading to a potential bug, crash, or application vulnerability in vehicle software. While traditional tests rely on the designer to remember and implement all possibilities, exhaustive static-analysis tools provide 100% coverage out of the box (Fig. 2).

2. This is a screenshot of a buffer overflow finding within the TrustInSoft exhaustive static-analysis tool.

Here, the tool detects the application writing 16 bytes after the beginning of the array. This memory corruption is revealed by a more verbose test case where the out-of-bounds write condition affects the value of the variable name, even though it’s not involved in the tested function, as shown in the following console output:

gcc -DVERBOSE -I. increment.c -o ub && ./ub 
Run test_increment_array() 
Address(data) = 0x7ffe8eb68f50 = 140731292749648 
Address(name) = 0x7ffe8eb68f60 = 140731292749664 
Before increment array = {1, 3, 5, 7} - name = foobars 
After  increment array = {2, 4, 6, 8} - name = goobars 
increment_array({1, 3, 5, 7}) = {2, 4, 6, 8} 
--> PASSED

While this is a simple example for illustrative purposes, applications with complex control and data paths can benefit from the use of exhaustive static analysis.

How Hardware Awareness Improves Testing Accuracy

Exhaustive static-analysis tools that take the actual vehicle computing hardware into account help to improve the accuracy and efficiency of software testing. Differences in compiler implementations, hardware architectures, and memory alignment between platforms can lead to drastically different code behaviors. For example:

On 64-bit targets, long is typically 64 bits and int is typically 32 bits.
On 32-bit targets, both long and int are typically 32 bits.

These implementation characteristics influence test design, as shown in this code sample:

long double_that(int i) 
{ 
   return (long)i * 2; 
} 
 
double_that(0x7FFFFFF0);

Tests or static-analysis methods that are unaware of the underlying implementation would not know whether the last statement causes an integer overflow (32-bit target) or is safe behavior (64-bit target). This code would decrease the efficiency of test designs that assume a 64-bit data range when the target is actually 32-bit, and negatively impact accuracy when assuming a 64-bit target and potentially missing the integer overflow when it’s actually 32-bit.

Hardware-aware exhaustive static analysis offers the perfect balance between 100% coverage and the minimum number of test cases necessary to achieve it. It also offers these benefits:

Tests can be run without requiring a physical target connected to the host.
Target tests can be run earlier in the development lifecycle before the physical hardware is available.
Test capacity can increase while costs decrease as physical hardware isn’t required for every developer.

Figure 3 summarizes the benefits of exhaustive static analysis throughout the automotive software V-cycle.

3. Exhaustive static analysis offers multiple benefits across the automotive software V-cycle.

Conclusion

The road to 100% code coverage starts with exhaustive static analysis. Automotive software development teams that prioritize exhaustive static analysis derive the greatest value from their testing and compliance investments. Achieving up to 100% code coverage with a mathematical guarantee of the absence of undefined behaviors supports standards like ISO 26262 and ISO 21434, and positions manufacturers to deliver safer, more secure code.

Achieving a "zero issue" guarantee is a long-term objective that can only be met with formal methods-based testing practices. Being proactive with exhaustive static analysis now significantly reduces the likelihood of software bugs and vulnerabilities in the field, further enhancing the overall reliability and security of automotive software systems.

Read more articles in the TechXchange: Developing High-Quality Software.