Are the tools associated to a particular formalism backed by responsive, robust and enduring organisations?

(that follow proven software development processes and that can provide quality user support)

Main -> FAQ -> TOOL-HM-1


 * Theme: Known Strengths and Weaknesses of Tools and Tool Providers (TOOL)
 * Role: HM

Answer
Nowadays, using formal methods without a supporting tool is not conceivable anymore. The industrial usage of formal methods requires high quality tools supporting different tasks (e.g. modelling, verification, code generation...). Important qualities are:
 * Reliability: the correct operation over time. In addition to a well-defined development process and quality control, this may also require some qualification or even certification activities (see the related FAQ)
 * Long term availability and support: capacity to stay available over time, industrial systems might require several years/tens of year of support between the development and the decommissioning of a system.
 * Scalability: the ability to cope with large and complex
 * Usability: the ease of use to perform the various tasks in terms of ease to navigate in the interface, to perform modelling, validation and verification activities, to work in team, etc.
 * Adaptability: the ability to integrate into existing industrial tool chains. This requires the existence of well-documented data format, ideally, availability of APIs, availability of specific integration plugin for popular tool platforms, availability of binaries on specific OS's,...
 * Security: the absence of vulnerabilities or even malicious code possibly disclosing company information.
 * Development process maturity: generally the development of formal tools largely follows a traditional development process. The way software is developed impacts its quality in terms of how the features are specified, designed, coded, tested.

Building tools with such qualities requires a strong organisation able to comply with a well defined development process and to provide quality support to end users. In addition, formal tool address a niche market and they require a dedicated support. Obviously, developing or maintaining a formal tool requires advanced skills that are seldom available to industrials working in the field. Typically formal methods tools have the following kind of evolution:
 * they are initially developed as results of some research projects. In this phase, they are generally made freely available, possibly as Open Source, but as-is and without any strong support but some community support.
 * after a successful series of validation, they can then evolve towards industrial use. In this case, they can become supported by a specific private company (generally a spin-off of the initial R&D project). This company will provide industrial level support. At this stage the product can still remain Open Source or partly Open Source under some mixed licence.
 * later evolution can be the acquisition of the product (and initial company) by a larger company which will incorporate it into a larger tool portfolio. This gives better assurance of long term support and improved integration within the tools (of that company) but generally also means a less transparent development process (closed source).

From the above, two main support models emerge: Open Source vs. Closed Source. In the rest of this FAQ we explore both models (including some variants), based on the previously identified quality dimensions. In what follows, we will use the acronym FLOSS for Free/Open Source Software.

Reliability
Concerns have been raised about whether FLOSS can achieve high reliability levels. While there are benefits to the open source software movement, caution is advised to expect high reliability from such software. One reason mentioned is that many individuals involved with using and testing such software will have only fleeting commitments.

However there is a large number of very successful FLOSS tools supporting formal methods, among the best known: Community Z tools (CZT), ZETA, ProofPower, Overture, ACL2, Coq, E, Otter/MACE, PTTP, Isabelle, HOL4, HOL Light, Gandalf, Maude Sufficient Completeness Checker, KeY, RODIN, Hybrid Logics Model Checker, Spin, NuSMV 2, BLAST, Java PathFinder, SATABS, DiVinE. A larger list is available from. A large number of these tools are recognized as very strong and innovative. They have also been used in industrial applications. Spin and the Boyer-Moore Theorem Prover (which is at the roots of ACL2) have received the ACM’s prestigious Software System Award, which recognizes a "software system that has had a lasting influence". Both have been used for extremely important applications, from checking space probe algorithms through microprocessor design. PVS is also widely considered to be one of the best tools of its kind.

Reliability is related to the potential of massive (worldwide) peer review and also the fact that developers know that their code will be subject to such a review. FLOSS also tends to have better defined interfaces and careful designs, since their developers must usually work worldwide and cannot depend on walking over to someone else to understand a design.

Long term availability
Commercial software can disappear due to the company going out of business. Some ways to protect against this risk are to:
 * place its trust into a large company and widely used tools, but this can restrict the access to a smaller set of tools.
 * rely on open standards. However, in a niche field like formal methods it is not likely to find many competitors providing alternative implementations to the same language.
 * have access to the source code: one can ask for an escrow as guarantee. Alternately the company might already publish its code as FLOSS and only focus on the service. However, in both cases this does not solve the maintenance and evolution problem after bankrupt even if the sources are available.

FLOSS can provide a community potentially surviving a very long time. However this depends on some characteristics of the community such as its size and who are the key contributors. For example if the key contributor is restricted to a single organisation, we have the same risk as for a commercial company in case this organisation stops supporting the project. The size of the community and of its core contributors does not necessary need to be huge as long as there is a viable business model. An example of SME producing FLOSS is AdaCore which makes profit by selling services to its customers. A more recent example is AtelierB maintained by ClearSy which decided to go Open Source and also adopt a service model. As formal tools are highly specialised, such services are expected to be highly specialised and highly profitable.

From the adoption on the company side, an increasing amount of corporate involvement in Open Source projects and their creation of new "open projects" was recently reported. This opens new questions about what motivates corporations to seek community support, whether they are successful and how they (expect to) pay back the community. This requires further investigations but could indicate that corporations are becoming more aware of the potential of FLOSS, even for high reliability.

Scalability
The ability to scale up depends on different factors. It can be related to limitations of:
 * the modelling formalism itself (e.g. lack of modularity)
 * the underlying formal technology (model checker, theorem prover...)
 * implementations problems such as some bottleneck in a processing chain, as resource leaks
 * usability: difficulty to navigate into large projects, to manage large pieces of models,...

The first limitation should discourage the use of that formalism while the others should be assessed in more details. A natural first step is to have access to feedback and reviews which can help rule out inadequate tools. A second step is to try challenge tools on a realistic case study from their own domain as the way the model are build can also impact on the ability to scale up. This is the approach taken in DEPLOY and reported (see DEPLOY Success Stories)

Open Source tools might have higher risk of not scaling up, especially if they are still at the R&D stage. However there are also highly scalable Open Source tools (e.g. apache webserver in the area of server infrastructure, nuSMV as model-checker in the area of formal tools).

Usability
Commercial tools generally have a better usability because this a special attention is devoted to it while FLOSS tools tend to focus more on the core functionality and efficiency, with sometimes only a command line interface.

Adaptability
FLOSS tools tend to be easier to adapt and integrate. The main reason is not because the source code is available (working at that level might not be a good idea as it requires advanced skills and will not be easy to evolve over time). It is rather due to the facts that FLOSS development will favour:
 * the use of open standards (like XML, XMI,...)
 * the definition of clear interfaces (API, services...)

Of course commercial software can also adhere to similar open principles. They can also provide specific integration capabilities, for example as part of a larger company toolset.

Related to the availability on specific OSes, it is generally not a problem. See our specific FAQ on this

Security
An common belief and barrier to the use of FLOSS is related to the sensitivity to malicious code introduction. However statistics from security monitoring organisation shows that the biggest software with credential backdoors are closed sources. It is also reported that the time to remove a backdoor is a matter of week for Open Source (thanks to the “many eyes effect”) while it can last for years in Closed Source. In any case, formal method tools are often used in house and not run by remote users external to a company. Thus, security of formal method tools is clearly not of foremost concern.

Development Process
A company developing can provide guarantee about its process by undergoing certification, especially if it produced closed source software. A generic software certification can adhere to the ISO-12207 (SPICES) or CMMI. However as the formal methods are often related to safety critical more specific standards can apply (often including the former as a basis) - see our FAQ on the position of standards regarding formal methods.

Related to Open Source development, there is more transparency on the process, often through a publicly accessible forge hosting a number of development support services such as code versionning, request and bug trackers, design documentation and test suites. There are a few methodologies available to assess FLOSS and the specific way such software is being developed. To give an illustration of the area assessed, the following figure reproduces a past QUALOSS assessment (January 2009) of the Rodin toolset.