Application Note: Structured Testing - The Path to Thorough Code Coverage

Basis path testing, also known as Structured Testing, is the primary code-based testing strategy recommended by McCabe Software and supported by McCabe IQ. McCabe IQ also supports various other code-based testing strategies including line, branch, and boolean (MC/DC). The fundamental idea behind basis path testing is that decision outcomes within a software function should be tested independently. The intuitive appeal of this idea is typically adequate to generate interest, but is certainly not a justification for implementation. After all, implementing a new testing procedure requires effort, and developers want to understand the benefits before making a commitment. This means that the information about the benefits of the technology needs to be communicated to those who will actually be using the technology. This article provides an overview of the benefits of basis path testing.

Testing is Proportional to Complexity

A major benefit of basis path testing is that the number of tests required is equal to the cyclomatic complexity metric. Since complexity is correlated with errors, this means that testing effort is concentrated on error-prone software. Additionally, since the minimum required number of tests is known in advance, the testing process can be planned and monitored in greater detail than with most other testing strategies. Statement coverage, branch coverage, and even obscure testing strategies such as variable definition/usage association coverage do not have this capability. In the case of arbitrarily complex and error-prone code, it might be possible to satisfy those criteria with one or two tests, or it might take thousands.

Testing Detects Interaction Errors

Unlike other common testing strategies, basis path testing significantly reduces the probability of errors being hidden due to interactions between decision outcomes during testing.

The most common code-based testing strategies are line coverage (sometimes referred to as line of code coverage or code coverage), statement coverage, and branch coverage. Line coverage, in which the number of executable lines that were encountered during testing is compared to the total number of executable lines, can be dismissed as a thorough test strategy, because it measures the code format rather than the code. In many programming languages, we could format a program as a single line and satisfy line coverage with one test. Statement and branch testing are stronger, but have the weakness that interactions between decision outcomes can mask errors during testing. By requiring decision outcomes to be exercised independently during testing, basis path testing can expose additional errors.

As a simple example, consider the following software function with two sequential decisions:

func()

(

if (condition1)

a = a + 1;

if (condition2)

a = a -1;

)

Assume that the above function is intended to leave the value of variable “a” unchanged under all circumstances, and is thus incorrect. The branch testing criterion can be satisfied with two tests that do not detect error. First let both decision outcomes be FALSE, in which case the value of “a” is not affected. Then let both decision outcomes be TRUE, in which case the value of “a” is first incremented and then decremented, ending up with its original value. The statement testing criterion can be satisfied with just the second of those tests. Thus, neither statement nor branch testing is adequate to detect the error in our example. Basis path testing, on the other hand, detects the error. Since three independent tests are required, at least one will set one decision outcome to TRUE and the other to FALSE, exposing the error by leaving “a” incremented or decremented during testing.

Integration Errors Based on Interaction

The integration-level Structured Testing strategy, based on design complexity and supported by McCabe IQ, requires independent testing of each decision outcome that affects the module-calling sequence and shares many of the benefits of basis testing. Call-pair coverage, a common integration testing measure based on exercising all caller/callee pairs, has the same weaknesses as branch coverage. Consider the slight modification of the above example function so that the increment and decrement operations on variable “a” are performed by function calls rather than in-line operators. Then, branch coverage yields call-pair coverage, which therefore does not detect the error. Since all decision outcomes affect the calling sequence, integration-level Structured Testing is, in this case, equivalent to basis path testing and therefore detects the error.

Conclusions

While detailed analysis of the underlying mathematics yields even further insight, consideration of the properties and simple example described above provides a good understanding of the benefits of basis path testing beyond the underlying “test decisions independently” description. The key properties of basis path testing, which are not shared by other common code-based testing strategies, are that:

• Testing is proportional to complexity
• Testing effort is concentrated on the most error-prone software
• Testing progress can be monitored with precision
• Errors based on interactions between decision outcomes are more likely to be detected

Call 800-638-6316 or click here to see McCabe IQ’s basis path coverage technology in action.