Does the use of formal engineering methods help in the design of tests?

Main -> FAQ -> EM-QAP-1


 * Theme: Exploiting Models (EM)
 * Role: QAP

Answer
Yes, there are at least four ways the design of test can be impacted by the use of formal methods:
 * 1) some kind of tests can become unnecessary if a formal method chain results in a proven implementation;
 * 2) test cases can be automatically derived from formal models with assurance of some type of coverage;
 * 3) in a manual design process, deriving tests from formal models can be more systematic than from informal documents;
 * 4) the use of formal methods results in more precise requirements, thus indirectly helping one to focus the test case elaboration on the targeted requirements.

1. Removing the need for specific kinds of tests
When formal methods are in use, some tests might be dropped, or partially dropped. For instance, if one validates that some communication protocol is deadlock-free through formal methods, one will not focus the testing of the implementation of this protocol on a search for deadlock. Rather, one can validate that the implementation complies with the verified model, and test properties of the protocol that could not be verified.

As a concrete case, Siemens transportation does not perform any unit testing of the software code that is developed through the B formal method, as the B models are proven correct by construction, and the code is generated from these B models through a certified code generation process relying on automated code generation tools. Rather, they focus the testing on the integration phase, to validate e.g. the requirements and the assumption on the domain, as discussed in the FAQ EM-PQAM-3.

Of course, switching from a test-based validation to a formal method-based validation must be done in accordance with the targeted norms or standards. This topic is discussed in a dedicated FAQ ExFac-HM-1. See also EM-PQAM-3 about the level of assurance delivered by formal methods, compared to testing.

2. Model-based testing
One can derive test cases from formal models. This is known as "model-based testing". Model-based testing automates the detailed design of the test cases and the generation of the traceability matrix. More precisely, instead of manually writing several test cases, the test designer writes an abstract model of the system under test, and then the model-based testing tool generates a set of test cases from that model. the tests are generated so as to enforce some completeness property on the model.

For state machine, one can ensure the following test completeness properties:
 * Covering each state of the model
 * Covering each transition of the model
 * Covering each transition pair of the system (provided they can all be taken at some point)

Model-based testing has two main advantages:
 * First, the design time of test cases is reduced.
 * Second, one can generate a variety of test suites from the same model by using different test selection criteria.

Static models can also be used, such as descriptions of data structures. For instance, one can generate a bunch of test XML files from a sufficiently precise description of their expected structure and typing.

Existing tools for model-based test generation include. Formal models from which test cases are generated can be any transition system including state machines, possibly incorporating some executable software code, or formal models such as B or EventB. In the DEPLOY project, SAP showed model-based testing was feasible based on Event-B and ProB tools. It is presented in details in our success story Benefits of Model-Based Test Automation.

Manual design from formal models
Once high quality specifications or models are available, for instance as the result of a formal development process, one can identify test cases in an easier way from these models even on a manual basis. For instance, identifying a test case can be done more easily if one aims at following a given path in an explicitly modelled state machine. This is more-less the process that is automated by model-based testing tools.

Besides this, team working is made easier because engineers can easily communicate based on precise models, and collaborate to develop e.g. test suites.

The better the requirements, the easier to elaborate tests
We showed in our success story Requirements Quality Improvements through Requirements Modelling that using formal methods forces one to precisely specify the required properties. As a consequence, it tend to forces one to deliver higher quality requirements documents. Such document would enable one to more efficiently direct the test cases to validate that the developed system correctly implements these requirements because the requirements are potentially more complete, clear and less ambiguous.