Paul Anderson, the vice president of engineering, GrammarTech, took set some time aside from his schedule to discuss CodeSonar and CodeSurfer with us.
ED: Can you give a little overview and history of CodeSonar and CodeSurfer?
Our main product is CodeSonar, an automated source-code analysis tool that performs a whole-program analysis and finds defects in C/C++ code. CodeSonar has been available for three years and uses program-analysis infrastructure from CodeSurfer, a tool that GrammaTech has been selling for nine years. CodeSurfer is based on research conducted at the University of Wisconsin.
ED: What kinds of problems can CodeSonar and CodeSurfer find?
CodeSonar finds flaws where the fundamental rules of the language are broken. Examples include buffer overruns, leaks, race conditions, null pointer dereferences, and uninitialized variables. It also finds inconsistencies in code that which while not necessarily bugs per se, indicate that a programmer may have misunderstood something. These correlate well with real bugs. Another class of flaws is those where an API is being misused. Finally, users can add their own checks, even sophisticated ones.
GrammaTech has focused on depth rather than breadth. CodeSonar does a superb job of finding the core problems that plague developers. For example, many different effects can contribute to the cause of a buffer overrun: integer arithmetic, pointer arithmetic, static allocation, dynamic allocation, aliasing, interprocedural effects, and others. CodeSonar understands the complex effects, which enables it to find more types of buffer overruns while maintaining a low level of false positives.
CodeSonar targets programming errors that adversely impact reliability, but some of these errors can also create security vulnerabilities. Buffer overruns are a good example. For this reason, some of our customers use our tool for security reviews.
ED: What were some of the lessons gained from developing and using CodeSonar and CodeSurfer?
Some flaws are easy to find with high precision and can be explained with easy-to-understand warning reports. However, the nastiest flaws are often very subtle and arise from interprocedural effects. While a tool that targets hard-to-find flaws it will have less precision (i.e., more false positives) and generate reports that require more thought from the user, the payoff is much greater because so many serious defects have complicated roots. It is easy to create a tool that finds simple flaws, and some users like such tools because the generated bug reports are trivial to interpret; there is actually a lot of pressure on vendors to dumb down the analysis. However, GrammaTech has resisted this temptation. Our customers develop avionics, medical, and other critical applications. They want their static-analysis tool to identify the serious flaws, even if it means they will spend a little longer reviewing warning reports.
ED: Can you comment on the current state of affairs with respect to static analysis tools?
Today, there are a large number of static-analysis tools. At one end of the spectrum are lint-like tools, which are useful but generate a lot of false positives; they are best at enforcing coding standards. At the other end are sophisticated model checkers that are very powerful but operate on an abstract specification of a program; users need to have expertise in model checking to use them. There is a sweet spot that has proven to be popular with users that balances false positives, false negatives, and performance. The companies in this spot are Coverity, GrammaTech, and Klocwork. From what we have heard, Coverity and Klocwork have biased their tools heavily toward eliminating false positives, but at the cost of missing many bugs. GrammaTech reports more complex bugs, but can have a slightly higher level of false positives. Many GrammaTech customers write mission-critical code, such medical device or avionics software, and they like how CodeSonar identifies more bugs.
ED: Has the increase of applications requiring higher levels of safety and reliability affected the importance of static analysis tools?
It is definitely a driving force. Regulatory agencies are pushing for the use of these tools by developers in response for the need for extra safety and reliability. For example, the FDA has an explicit policy to speed the adoption of these tools. Many of our customers write or review safety-critical code. NASA, the FDA, GE Aviation, Lockheed Martin, Cardinal Health, Zoll Medical, and other similar organizations use CodeSonar.
ED: How have improvements in IDEs and system performance affected the adoption of static analysis tools?
IDEs are great and improve programmer productivity. CodeSonar integrates seamlessly with IDEs, including Eclipse, so developers can analyze code without making changes to their development environment or build system. However, we have found that displaying the warning reports in an IDE is not ideal. One reason is that it is difficult to summarize a complex bug clearly within an IDE. A bigger reason is that results are reviewed by a wider audience than developers. Managers want to look at warnings, categorize results, create reports, see trends, and identify problem spots. They don’t want an IDE and they don’t want to have to install software. CodeSonar has a Web-based GUI (graphical user interface) that reduces setup and makes it easy for teams to manage results. In addition, the interface is uniform across build environments, which is handy because many of our customers are using multiple toolchains. Improvements in system performance have been instrumental in making this class of advanced static-analysis tools feasible.