What impact does the use of formal engineering methods have on the identification of issues at each phase of development cycle?

Alternate Similar Question (stated in a partial way): Is there evidence that the use of formal engineering methods helps identify issues and errors earlier in the development cycle?

Main -> FAQ -> QI-PQAM-1


 * Theme: Quality Improvement (QI)
 * Role: PQAM

Answer
Formal methods can apply at different stages of the development lifecycle to automate various types of verification that would otherwise be performed at later stages of product development lifecycle or not performed at all. As a result, it is generally admitted that formal engineering methods help to identify issues earlier in product development lifecycle unlike non-formal developments where issues are increasingly identified as the development lifecycle advances to reach a maximum at testing time.

Therefore, by identifying errors earlier, formal methods forces one to resolve them earlier, for example, by removing ambiguities and incompleteness in the various work products produced at each stage of the development lifecycle. In addition to removing errors, formal methods also help stakeholders to gain additional knowledge about the product or system being developed. For example, from a formal design, one may learn about timing information in a complex system or from a formal analysis of source code, one may identify performance bottle neck or segments of dead code.

Below, various industrial case studies were reviewed to determine the impact of formal method usage on the identification of issues at various phases of the development lifecycle.

Requirements Phase
The use of formal methods at requirements analysis stage can be very beneficial to the whole development process as issues and error identify at this point are much cheaper to correct.

Requirements documents are usually insufficient for specifying formal models. In a non formal development approach, these insufficiencies can persist much longer in the development lifecycle. On the other hand, formal modelling forces requirement analysts to be very accurate and detailed, in turn, they ask very precise questions to customers to complete or correct problematic requirements. Indeed, given the wealth of information needed to develop formal models, requirement analysts learn to ask question that are often completely overlooked with non-formal development techniques. After a few years of experience with formal modelling, analysts become much better at asking the right question to obtain clear and complete requirement from the customer and they are also much faster at identifying requirements defects even before initiating a formal analysis.

Below is a short presentation on works performed during the DEPLOY project and other endeavours to show that the various facts mentioned above are verified at Bosch, Siemens Transport, XMOS, and the US Navy. ''At Bosch, the substantial refactoring of a requirements document using the Problem Frames method helped to correct a number of problems such as inconsistency, incompleteness, and ambiguity. This is described in more details in this FAQ in the success story: Requirements Quality Improvements through Requirements Modelling.''

''Since Siemens Transport started using B to formally design their software components, they have become much more thorough during requirement development and elicitation. In particular, they report that they now ask questions that were not even considered before due to incompleteness in requirements such as lack of accurate ground topology of the metro system to reason accurately about braking distance to reach a halt.''

Similar observation have also been made outside of Deploy:

''In, , , it is explained that SCR (Software Cost Reduction) and its toolset were used to  identify several serious issues during requirement analysis. SCR is devoted to the proper identification of requirements throug simulation and validation of state machine by end users. SCR is furthermore able to perform simple checks on the formal models such as determinism and forms of logical completeness. This method is still in use nowadays and has matured for more than 15 years.''

''In, two case studies explored the impact of two formal techniques for specifying and checking requirements. In one case study, SRI's PVS specification language was used and Stanford's Murphy finite-state verificatio system was used in the other. Among various benefits of using these two formal requirement analysis, the authors compares the issues identified when using formal requirement analysis and when only performing non-formal reviews. They discovered that the use of formal requirement analysis help to identified 7 major issues while non-formal approach only discovered 1 error; concerning minor issues, 23 were discovered using formal method and 3 only through non formal checks.''

''In, The NewCoRe project run over a two year period in the early 90’s. A specification of 7,500 lines of (non-commented) SDL code was written and about 150 correctness properties were formally specified and verified for the SDL model. As a result, a total of  112 serious design errors were detected in the design requirements.''

Design Phase
Using formal methods at the requirement level aims to ensure that the functionality of the product are well understood and described, and that all angles of that functionality have been analyzed and specified in details in the requirements document. Although requirements document must also state important non-functional requirements, as well as describe the constraint of the environment, requirement analysis cannot really guarantee the non-functional behaviour because detailed this requires more information about the architecture and technical details of the proposed solution. By definition, architectures and technical details of a proposed solution are specified at the design stage.

The most important observation from the field concerning issues identified at design time comes from Siemens Transport. They currently use the B method to design their software component for metro and train systems. In their design, all safety conditions of their formal models are proved to be satisfied. Thus, a system implementation that matches the design will never encounter safety problem (within the scope of the considered requirements). Clearly, when they encounter a problem with a proof, this forces them the redesign their formal models until all safety conditions are guaranteed. As explain in a Productivity Improvement of Data Consistency in Transportation Models, certain proofs can take one to two months to be verified by humans. Thanks to Pro-B, a model checker and animator build for B and Event-B models, Siemens Transport can now check their model and associated proofs in matter of seconds. We can therefore argue that in the Siemens case, formal modelling not only helps to build safe transport systems but also do so at a much imporved productivity rate.

A second observation is not directly link to issues identified in formal design but rather on the benefit of formal design modelling on requirements document. First we note that in most cases, industry projects start from an informal requirements document. Consequently, the initial attempt to construct a formal design is hampered by weaknesses and errors in the informal requirements document. This forces the design analyst to further question customers to improve the accuracy of the requirements document. Some examples of this situation that happened to Deploy partners, e.g. Space System Finland

''At SSF, they attempted to model formally two different systems using Event-B, namely, the BepiColombo SIXS/MIXS on-board software and attitude and orbit control systems. The modelling exercise of BepiColombo SIXS/MIXS on-board software requirements in Event-B helped identify ambiguity, incompleteness and redundancy that are commonly discovered later in the development cycle. When modelling the architecture of an attitude and orbit control systems helped to identify issues that may have remain undetected otherwise. Although the associated observations were also identified with less formal techniques such as algorithm inspections, the fact that Event-B models also helped identified this issues and not others raised the confidence of design analysts about the proposed system architecture and algorithms. '' Another important observation is with regards to the impact of formal design on issues at later stages of the development cycle. In particular, SSF noticed that using the Event-B formalism for modelling attitude and orbit control systems helped to create simpler more modular system architecture easier to understand for system developers. Consequently, they believe that developers will introduce fewer errors at the implementation stage.

Implementation & Debugging Phase
Siemens has implemented a complete formal chain from design to code by using a code generator (recognized for SIL 4 certification). B specification proven correct are fed to the code generator that generates the full code. No edition to this code is performed and given that correctness of the code generator, no issues are identified during the testing phase, which could therefore be dropped.

Source code analysis has proven their effectiveness in industry, notably because they directly analyse the source code, which is an artefact of all software development. The Polyspace tool is a well-known example that performs static analysis of C, C++ or ADA code through abstract interpretation, thus relying on automated algorithm. It has been extensively deployed in Space and avionics sectors for instance.

Testing Phase
At Siemens (transport systems), code is generated from B specifications proven correct. Hence, they needn't perform any unit and system testing as no issue would be identified (except for error in the code generator or complier). However, given the habits of most customers to participate to acceptance test session to sign the final acceptance record, Siemens performs acceptance test to validate the overall functioning of the software. In other words, given the very thorough design specification induced by the use of the B formal method, all requirement issues are identified during the design phase. Acceptance testing is only limited to performing a demonostration of the system to the customer.