Towards Symbolic Model-Based Mutation Testing: Combining Reachability and Refinement Checking

We present our newest paper on model-based mutation testing accepted at MBT 2012.

Model-based mutation testing uses altered test models to derive test cases that are able to reveal whether a modelled fault has been implemented. This requires conformance checking between the original and the mutated model. This paper presents an approach for symbolic conformance checking of action systems, which are well-suited to specify reactive systems.

We also consider non-determinism in our models. Hence, we do not check for equivalence, but for refinement. We encode the transition relation as well as the conformance relation as a constraint satisfaction problem and use a constraint solver in our reachability and refinement checking algorithms.

Explicit conformance checking techniques often face state space explosion. First experimental evaluations show that our approach has potential to outperform explicit conformance checkers.

This research is part of the TRUFAL project.

Download PDF

Protocol conformance testing a SIP registrar

Photo: Tiptel

In recent years we have successfully applied model-based mutation testing to several implementations of communication protocols. In our work reported in

Bernhard K. Aichernig, Bernhard Peischl, Martin Weiglhofer, and Franz Wotawa. Protocol conformance testing a SIP registrar: an industrial application of formal methods. In Mike Hinchey and Tiziana Margaria, editors, Fifth IEEE International Conference on Software Engineering and Formal Methods (SEFM 2007), 10-14 September 2007, London, England, UK, pages 215–226. IEEE Computer Society, 2007. (PDF) (doi:10.1109/SEFM.2007.31)

we tested parts of the Session Initiation Protocol (SIP) used in internet telephony in order to register and connect communication partners.

Our case study targeted a commercial and an open-source (OpenSER) SIP registrar. Furthermore, we compared the mutation approach to a standard model-based test case generation technique. Our mutation technique generated 124 test cases, compared to 6000 test cases generated by the standard approach.

The 6000 tests discovered nine different faults in the commercial SIP registrar and four in the open-source registrar OpenSER.

Interesting was the fact that the comparatively small set of 124 mutation tests uncovered one additional fault in the commercial implementation that was not detected by the 6000 tests of the classical approach.

This is a further indication that mutation testing could play an important role in a mature testing process.

Another important contribution of this paper was to show how our mutation technique could be scaled to larger models. We sliced the models with the help of markers around the injected faults.

Industry Day at QSIC 2012, Xi'an, China

Photo: bachmont

The 12th International Conference on Quality Software (QSIC 2012) will be held in Xi'an, China, 27 to 28 of August, 2012. The QSIC series of conferences is an established event bringing together people in the area of software and systems quality.

I was happy to learn that Zhou Chaochen (Chinese Academy of Sciences), my former boss at UNU-IIST in Macao, is the General Chair of the event. Zhou Chaochen is the inventor of the Duration Calculus. I joined the UNU faculty in Macao from 2002 to 2006.

For the first time, QSIC is planning an industry day with a dedicated program committee build from practitioners from industry and applied research. I was invited to join this program committee. Wolfgang Grieskamp (Google) and Hongyu Zhang (Tsinghua University) are the Industry Chairs of QSIC 2012.

Paper Submission Due: 23rd April 2012

Further details can be found on the website of QSIC 2012.

P.S: QSIC is an established conference series. I had my first QSIC paper in 2005 and my latest paper at QSIC 2010:

• Bernhard K. Aichernig and Percy Antonio Pari Salas. Test case generation by OCL mutation and constraint solving. In Kai-Yuan Cai and Atsushi Ohnishi, editors, QSIC 2OO5, Fifth International Conference on Quality Software, Melbourne, Australia, September 19-21, 2005, pages 64–71. IEEE Computer Society Press, 2005. (PDF) (doi:10.1109/QSIC.2005.63)
• Harald Brandl, Martin Weiglhofer, and Bernhard K. Aichernig. Automated conformance verification of hybrid systems. In Proceedings of QSIC 2010: the 10th International Conference on Quality Software, Zhangjiajie, China, July 14-15, 2010. IEEE Computer Society, 2010. Copyright by IEEE. (PDF).

A new EU project started: MBAT

On November 1st a new EU project started: MBAT - Combined Model-based Analysis & Testing of Embedded Systems.

MBAT is a three-years ARTEMIS joint undertaking project with 41 partners. I am the key researcher and project manager for TU Graz. MBAT combines the latest model-based testing technologies with static analysis techniques leading to a new model-based validation & verification method in the transportation domain (aerospace, automotive, railways).

My team at TU Graz brings in our expertise and tools in model-based mutation testing from the MOGENTES and TRUFAL projects. One aim is to combine model-based mutation testing with structural test case generation approaches. We will also investigate the relation of fault-models to the step-wise refinement in model-driven development.

In model-based mutation testing we inject faults in models of the system under test and automatically generate test sequences that will detect these faults in an implementation. Hence, our tests prevent that the faulty model has been implemented. For details see, e.g., our paper "Efficient mutation killers in action" that can be found in the list of our conference papers. This work describes the case-study of testing a car-alarm system.

In MBAT we will also investigate the current testing practice of our industrial partners and develop a process model where the different testing techniques of MBAT best fit. This will facilitate the adoption of the tools and techniques in industry.

We closely collaborate with the following Austrian project partners: AIT, AVL, and VIF.