Integrating model-based testing and analysis tools via test case exchange

Our first publication in the MBAT project was presented at TASE 2012 and in a seminar given at UNU-IIST:

Bernhard K. Aichernig, Florian Lorber, and Stefan Tiran. Integrating model-based testing and analysis tools via test case exchange. In TASE 2012, 6th IEEE International Symposium on Theoretical Aspects of Software Engineering, July 4–6, Beijing, China, pages 119-126. IEEE Computer Society, 2012. (PDF)

In this paper we show an integration of two tools via test cases. For example test cases are generated from a model in Tool 1 and then model checked in another Tool 2. This is useful when a modeling tool does not support model checking as it is the case with most UML tools.

Abstract: Europe’s industry in embedded system design is currently aiming for a better integration of tools that support their development, validation and verification processes. The idea is to combine model-driven development with model-based testing and model-based analysis. The interoperability of tools shall be achieved with the help of meta-models that facilitate the mapping between different modelling notations. However, the syntactic and semantic integration of tools is a complex and costly task. A common problem is that different tools support different subsets of a language. Furthermore, semantic differences are a major obstacle to sound integration efforts.

 In this paper we advocate an alternative, more pragmatic approach. We propose the exchange of test cases generated from the models instead of exchanging the models themselves. The advantage is that test cases have a much simpler syntax and semantics, and hence, the mapping between different tools is easier to implement and to maintain. With a formal testing approach with adequate testing criteria a set of test cases can be viewed as partial models that can be formally analysed. We demonstrate an integration of our test case generator Ulysses with the CADP toolbox by means of test case exchange. We generate test cases in Ulysses and verify properties in CADP. We also generate test cases in CADP and perform a mutation analysis in Ulysses.

ICTSS 2012, Aalborg, Denmark

Photo: dinilu

I have been invited to serve on the Program Committee of ICTSS 2012, the 23rd IFIP Int. Conference on Testing Software and Systems, November 19-21, 2012 Aalborg, Denmark.

Please, consider submitting your testing work until June 11, 2012 (abstracts), June 18, 2012 (full papers).

Brian Nielsen, Aalborg University, DK and Carsten Weise, IVU Traffic Technologies, DE are the program chairs. For details see the conference homepage.

Last year, we presented a paper at ICTSS in Paris:

Christian Schwarzl, Bernhard K. Aichernig, and Franz Wotawa. Compositional random testing using extended symbolic transition systems. In Burkhart Wolff and Fatiha Zaïdi, editors, Testing Software and Systems - 23rd IFIP WG 6.1 International Conference, ICTSS 2011, Paris, France, November 7-10, 2011. Proceedings, volume 7019 of Lecture Notes in Computer Science, pages 179–194. Springer-Verlag, 2011. (PDF)

Towards Symbolic Model-Based Mutation Testing: Combining Reachability and Refinement Checking

We present our newest paper on model-based mutation testing accepted at MBT 2012.

Model-based mutation testing uses altered test models to derive test cases that are able to reveal whether a modelled fault has been implemented. This requires conformance checking between the original and the mutated model. This paper presents an approach for symbolic conformance checking of action systems, which are well-suited to specify reactive systems.

We also consider non-determinism in our models. Hence, we do not check for equivalence, but for refinement. We encode the transition relation as well as the conformance relation as a constraint satisfaction problem and use a constraint solver in our reachability and refinement checking algorithms.

Explicit conformance checking techniques often face state space explosion. First experimental evaluations show that our approach has potential to outperform explicit conformance checkers.

This research is part of the TRUFAL project.

Download PDF

Protocol conformance testing a SIP registrar

Photo: Tiptel

In recent years we have successfully applied model-based mutation testing to several implementations of communication protocols. In our work reported in

Bernhard K. Aichernig, Bernhard Peischl, Martin Weiglhofer, and Franz Wotawa. Protocol conformance testing a SIP registrar: an industrial application of formal methods. In Mike Hinchey and Tiziana Margaria, editors, Fifth IEEE International Conference on Software Engineering and Formal Methods (SEFM 2007), 10-14 September 2007, London, England, UK, pages 215–226. IEEE Computer Society, 2007. (PDF) (doi:10.1109/SEFM.2007.31)

we tested parts of the Session Initiation Protocol (SIP) used in internet telephony in order to register and connect communication partners.

Our case study targeted a commercial and an open-source (OpenSER) SIP registrar. Furthermore, we compared the mutation approach to a standard model-based test case generation technique. Our mutation technique generated 124 test cases, compared to 6000 test cases generated by the standard approach.

The 6000 tests discovered nine different faults in the commercial SIP registrar and four in the open-source registrar OpenSER.

Interesting was the fact that the comparatively small set of 124 mutation tests uncovered one additional fault in the commercial implementation that was not detected by the 6000 tests of the classical approach.

This is a further indication that mutation testing could play an important role in a mature testing process.

Another important contribution of this paper was to show how our mutation technique could be scaled to larger models. We sliced the models with the help of markers around the injected faults.