Test or spec? Test and spec? Test from spec!

September 2004

Which came first? As an intermediate result towards a more general research goal, it has recently been demonstrated that the egg precedes the omelet. A full proof falls beyond the scope of this modest EiffelWorld column, but here is the idea: you can construct the omelet from the egg, but not the egg from the omelet. (The chicken is covered by a separate lemma.) The reader will already have jumped mentally to an important special case; for omelet read test, and for egg read specification, particularly in the form of Eiffel contracts.

We are being told from some quarters that you can't specify anyway, and that development should be "test-driven". Eiffel programmers know from daily practice and from observation of the libraries that the first proposition is false: precise specifications in the form of contracts, even if they cover only part of the functionality, radically change the software development process. In the new scheme, tests play a critical role (see another little article of a few years back, "Qualityfirst" at:

http://www.inf.ethz.ch/~meyer/publications/computer/quality_first.pdf

which I believe already suggested a few of the good ideas of agile methods), but tests are not a replacement for specification: they follow from these specifications. It is indeed remarkable to see how the presence of contracts can drive the testing process. If a slogan is needed and two can do, I will venture "Contract-driven testing" and "Test-obsessed development". That works very well -- you should test all the time, with the intent of finding bugs -- but it's not a reason to drop specification and design. Specification and design are what propels both the testing process and the test cases themselves.

Going from specifications to test is one-way: you lose the abstractions. A specification describes the general properties of a computation, for all possible inputs; a test addresses one particular result for one particular input. From the general you can deduce the particular, but not the other way around: even a billion tests don't reveal the insight -- the abstraction -- of a specification. Omelets beget no eggs.

For an Eiffel programmer, testing consists of turning on contract monitoring, exercising components, and waiting for a contract to fail. Since any test oracle can be expressed by a postcondition, but no collection of test oracles can replace a specification, we lose nothing, but we gain a great advantage over naïve (contract-less) test-driven development: retaining the fundamental abstractions that underlie our programs and their individual modules.

In some later installment of this column focused more on research, I'll discuss how it is becoming possible to automate the component testing process completely, test case generation included. Suffice it for the moment to note that while we should be grateful to our extreme friends for helping to rehabilitate the much maligned process of software testing, we -- Eiffel programmers -- know something they don't seem to have realized yet: you can test from specs, but you can't spec from tests.

-- Bertrand Meyer