How much do QA Practitioners need to understand about the formal engineering methods being used?

and is this any different from other systems/software engineering techniques?*

Main -> FAQ -> TSP-QAP-1


 * Theme: Training Scope and Pace (TSP)
 * Role: QA Practitioner (QAP)
 * Alternate Similar Question: Is it mandatory for QA Practitioners to understand formal engineering methods used by the development team?

Answer
In general, unlike engineers and analysts, QA practitioners do not need to develop models or to elaborate a formal verification (build a proof, run a checker, etc). Rather, they check that the formal methods are used in a sound way, as detailed in the FAQ CIF-HM-2.

In the context of formal methods for verification, a successful formal verification shows that some claim is true on some model. As such, QA practitioners need to deeply understand:


 * The claims that are proven by the formal method: QA practitioners must to ensure that the proven claim is accurate with respect to the application, and that all the claims that must be enforced on the application are proven through a formal approach, or through some other validation approach.
 * The consequences of the abstraction made in the model: Models introduce abstraction; one must ensure that the established truth remain valid in the real world although they might be done on an abstract model. This is extensively discussed in the FAQ G-EA-1.
 * The level of assurance that is delivered by the provers: Some formal tools only perform a limited form of verification by searching for small-size counterexample. Consider for instance the Alloy tool [Jack01]. This tool searches for a counterexample for some given claim within a given perimeter, whose purpose is to restrict the level of expressiveness of the claim from first order to propositional logics. If one such bounded verification tool does not find a counterexample, it is not a proof that there is no counterexample at all out of the artificial verification perimeter.
 * The compliance of the formal method with the targeted standard: Some sectors require very high assurance, and might require some redundancy on the verification, or to rely only on certified formal verification tools. Some provers of the Deploy project are certified for some CENELEC level of assurance.
 * If the model is easy to prove: Some formal verification technology reach higher automation if the model is developed according to some rules (avoid some type of constructs, avoid symmetries, etc.). QA might also be required verify the easiness of the model before the model is actually proven, to spare the time of engineers.

Sometime, these checks have been synthesized into a set of guidelines. QA then amounts to checking that a formal model complies with the established guidelines. At Siemens transportation, the above tasks are under the responsibility of the safety engineers. QA are in charge of verifying that the developed models match some reference good practice. This verification is performed before proofs are made. Some of these good practices are related to the three aforementioned bullets while some others are aimed at reaching a high level of auto-proving.

Refences
[Jack01] Daniel Jackson, Software Abstractions Logic, Language, and Analysis, MIT Press, April 2006