Chapter 12: Strategies for Testing LINT

Overview

Don't blame the Compiler.

—David A. Spuler: C++ and C Debugging, Testing, and Code Reliability

IN THE PREVIOUS CHAPTERS WE have encountered here and there hints for testing individual functions. Without meaningful tests to ensure the quality of our package, all of our work would be for naught, for on what else are we to base our confidence in the reliability of our functions? Therefore, we are now going to give our full attention to this important topic, and to this end we ask two questions that every software developer should ask:

1. How can we be certain that our software functions behave according to their specifications, which in our case means first of all that they are mathematically correct?

2. How can we achieve stability and reliability in the functioning of our software?

Although these two questions are closely related, they are actually concerned with two different problem areas. A function can be mathematically incorrect, for example if the underlying algorithm has been incorrectly implemented, yet it can reliably and stably reproduce this error and consistently give the same false output for a given input. On the other hand, functions that apparently return correct results can be plagued by other sorts of errors, for example an overflow of the length of a vector or the use of incorrectly initialized variables, leading to undefined behavior that remains undetected due to favorable (or should we rather say unfavorable?) test conditions.

We thus must be concerned with both of these aspects and institute development and test methods that can provide us sufficient trust in both the correctness and reliability of our programs. There are numerous publications that discuss the significance and consequences of these wide-ranging requirements for the entire software development process and delve deeply into the issues of software quality. Considered attention to this topic has found expression not least in the international trend to institute the ISO 9000 standard in software production. In this regard one no longer speaks merely of "testing" or "quality assurance," but instead one hears talk of "quality management" or "total quality management," which in part are simply the result of effective marketing, but which nonetheless cast the issue in the proper light, namely, to consider the process of software creation in its multifaceted entirety and thereby improve it. The frequently employed expression "software engineering" cannot blind us to the fact that this process, as it relates to predictability and precision, as a rule can scarcely compete with the classical discipline of engineering.

The comparison may be characterized aptly by the following joke: A mechanical engineer, an electrical engineer, and a software engineer have decided to take an automobile trip together. They seat themselves in the car, but it refuses to start. The mechanical engineer says at once, "The problem is with the motor. The injection nozzle is clogged." "Nonsense," retorts the electrical engineer. "The electronics are to blame. The ignition system has certainly failed." Whereupon the software engineer makes the following suggestion: "Let's all get out of the car and climb back in. Perhaps then it will start."

Without pursuing the further conversations and adventures of the three intrepid engineers, let us proceed to consider some of the options that were implemented in the creation and testing of the FLINT/C package. Above all, the following references were consulted, which do not exhaust the reader with abstract considerations and guidelines but get down to concrete assistance in solving concrete problems, without in the process losing sight of the big picture.^[1] Each of these books contains numerous references to further important literature on this topic:

• [Dene] is a standard work that deals with the entire process of software development. The book contains many methodological pointers based on the practical experience of the author as well as many clear and useful examples. The theme of testing is attacked again and again in connection with the various phases of programming and system integration, where the conceptual and methodological fundamentals are discussed together with the practical point of view, all in conjunction with a thoroughly worked out example project.

• [Harb] contains a complete description of the programming language C and the C standard library, and it gives many valuable pointers and comments on the prescriptions of the ISO standard. This is an indispensable reference work to be consulted at every turn.

• [Hatt] goes into great detail on the creation of security-critical software systems in C. Typical experience and sources of error are demonstrated by means of concrete examples and statistics—and C certainly offers many opportunities for error. There is also comprehensive methodological advice, which if heeded would lead to increased trust in software products.

• [Lind] is an excellent and humorous book, which reveals a deep understanding of the C programming language. Moreover, the author knows how to transmit this understanding to the reader. Many of the topics considered could be supplied the subtitle, "Did you know that ... ?" and only a very few readers could honestly—hand on heart—reply in the affirmative.

• [Magu] deals with the design of subsystems and is therefore of particular interest to us. Here are discussed the interpretation of interfaces and the principles of dealing with functions with input parameters. The differences between risky and defensive programming are elucidated as well. The effective use of assertions (see page 151) as testing aids and for the avoidance of undefined program states is a further strong point of this book.

• [Murp] contains a host of testing tools that can be put to use in testing programs with little effort and that yield immediate useful results. Among its other features the book offers libraries on an accompanying diskette for the implementation of assertions, testing the processing of dynamic memory objects, and reporting the degree of coverage of tests, which were also used for testing the FLINT/C functions.

• [Spul] offers a broad view of methods and tools for testing programs in the C and C++ languages and gives numerous pointers for their effective use. The book contains an extensive overview of programming errors typical in C and C++ and discusses techniques for recognizing and eliminating them.