Software Testing, Verification and Reliability

Cover image for Vol. 24 Issue 6

Early View (Online Version of Record published before inclusion in an issue)

Edited By: Jeff Offutt and Robert M. Hierons

Impact Factor: 1.2

ISI Journal Citation Reports © Ranking: 2013: 37/105 (Computer Science Software Engineering)

Online ISSN: 1099-1689

Associated Title(s): Journal of Software: Evolution and Process, Software Process: Improvement and Practice, Software: Practice and Experience


  1. 1 - 21
  1. Special Issue Papers

    1. Automated test case generation for FBD programs implementing reactor protection system software

      Eunkyoung Jee, Donghwan Shin, Sungdeok Cha, Jang-Soo Lee and Doo-Hwan Bae

      Article first published online: 5 SEP 2014 | DOI: 10.1002/stvr.1548

      Thumbnail image of graphical abstract

      FBDTesteris an effective tool for automated test data generation for function block diagram programs implementing safety-critical system software.

    2. Advances in noise-based testing of concurrent software

      J. Fiedor, V. Hrubá, B. Křena, Z. Letko, S. Ur and T. Vojnar

      Article first published online: 5 SEP 2014 | DOI: 10.1002/stvr.1546

      Thumbnail image of graphical abstract

      The paper presents multiple recent achievements from the area of noise-based testing of concurrent software, which is based on influencing the scheduling in order to witness many different thread interleavings. Multiple noise heuristics (defining where, when and how to inject noise) are introduced, including several previously unpublished ones, and experimentally evaluated using concurrency coverage metrics. Further, recommendations for setting up noise-based testing are given together with a novel use of the genetic algorithm for finding suitable parameters of noise-based tests.

  2. Research Articles

    1. Formal firewall conformance testing: an application of test and proof techniques

      Achim D. Brucker, Lukas Brügger and Burkhart Wolff

      Article first published online: 27 AUG 2014 | DOI: 10.1002/stvr.1544

      Thumbnail image of graphical abstract

      A HOL-TESTGEN session showing the jEdit-based Isabelle Interface. The upper-left sub-window allows one to interactively step through a test theory comprising test specifications, while the lower-left sub-window shows the corresponding system state of the spot marked in blue in the upper window.

  3. Special Issue Papers

    1. Model-based testing for concurrent systems with labelled event structures

      Hernán Ponce de León, Stefan Haar and Delphine Longuet

      Article first published online: 20 AUG 2014 | DOI: 10.1002/stvr.1543

      Thumbnail image of graphical abstract

      We propose a theoretical testing framework and test generation algorithm for concurrent systems specified with true concurrency models introducing the notions of strong and weak concurrency: strongly concurrent events must be concurrent in the implementation, while weakly concurrent ones may eventually be ordered. We propose an extension of the ioco conformance relation for labelled event structures, named co-ioco, allowing to deal with strong and weak concurrency. We extend the notions of test cases and test execution, and give a test generation algorithm building a complete test suite.

    2. MuRanker: a mutant ranking tool

      Akbar Siami Namin, Xiaozhen Xue, Omar Rosas and Pankaj Sharma

      Article first published online: 11 AUG 2014 | DOI: 10.1002/stvr.1542

      Thumbnail image of graphical abstract

      MuRanker introduces and further implements the idea of ranking mutants. The tool ranks mutants according to their complexity in being detected by a test pool. The tool captures the distance between a mutant and the original program in deciding about ranking mutants. The empirical evaluation demonstrates the validity of the hypothesis stating that detecting medium and hard-to-kill mutants first will automatically detect easy-to-kill mutants, and thus, it reduces the number of test cases needed to achieve the mutation adequacy criterion.

    3. Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation

      Shin Hong, Matt Staats, Jaemin Ahn, Moonzoo Kim and Gregg Rothermel

      Article first published online: 23 JUN 2014 | DOI: 10.1002/stvr.1539

      Thumbnail image of graphical abstract

      We explore the impact of concurrency coverage metrics on testing effectiveness and examine the relationship between coverage, fault detection, and test size. We study eight existing coverage metrics and six new metrics formed by combining complementary metrics. Our results indicate that the metrics are moderate to strong predictors of testing effectiveness and effective at providing test generation targets. Nevertheless, metric effectiveness varies across programs, even for the combinations of metrics. This result highlights the need for improving concurrency coverage metrics.

    4. Test generation with Satisfiability Modulo Theories solvers in model-based testing

      Jérôme Cantenot, Fabrice Ambert and Fabrice Bouquet

      Article first published online: 29 MAY 2014 | DOI: 10.1002/stvr.1537

      Thumbnail image of graphical abstract

      The paper presents a framework for generating test from ULM/OCL models using Satisfiability Modulo Theories transformation. The metamodels associated with the two models and the transformation rules are presented. An implementation and experimentations have been realised. Copyright © 2014 John Wiley & Sons, Ltd.

    5. Reducing mutation costs through uncovered mutants

      Pedro Reales Mateo and Macario Polo Usaola

      Article first published online: 15 MAY 2014 | DOI: 10.1002/stvr.1534

      Thumbnail image of graphical abstract

      Mutant Schema Improved with extra Code (MUSIC) annotates the statements covered by test cases in the original program. Later, when mutants are executed, the execution engine avoids going through uncoverable mutants. Our empirical results prove that music is an efficient cost reduction technique. Copyright © 2014 John Wiley & Sons, Ltd.

    6. A random testing approach using pushdown automata

      Aloïs Dreyfus, Pierre-Cyrille Héam, Olga Kouchnarenko and Catherine Masson

      Article first published online: 13 MAY 2014 | DOI: 10.1002/stvr.1526

      Thumbnail image of graphical abstract

      Because finite automata are, in general, strong abstractions of systems, many test cases, which are automata traces generated uniformly at random, may be unconcretizable. This paper proposes a method extending the aforementioned testing approach to pushdown systems providing finer abstractions. Using combinatorial techniques guarantee the uniformity of generated traces. In addition, to improve the quality of the test suites, the combination of coverage criteria with random testing is investigated. The method is illustrated within both structural and model-based testing contexts. Copyright © 2014 John Wiley & Sons, Ltd.

  4. Research Articles

    1. Specification guidelines to avoid the state space explosion problem

      Jan Friso Groote, Tim W.D.M. Kouters and Ammar Osaiweran

      Article first published online: 12 MAY 2014 | DOI: 10.1002/stvr.1536

      Thumbnail image of graphical abstract

      This paper provides seven guidelines that help in preventing the transition systems generated from behavioural process models to become extremely large. This is required to allow the models to be formally analysed. Experience teaches that this is essentially the only way to get such models correct.

  5. Special Issue Papers

    1. Quality metrics for mutation testing with applications to WS-BPEL compositions

      Antonia Estero-Botaro, Francisco Palomo-Lozano, Inmaculada Medina-Bulo, Juan José Domínguez-Jiménez and Antonio García-Domínguez

      Article first published online: 9 MAY 2014 | DOI: 10.1002/stvr.1528

      Thumbnail image of graphical abstract

      Mutation testing is a successful testing technique. However, it can be very costly. Cost reduction can be aided by an analysis of mutation operators, but this requires the definition of specialized metrics. A better quality metric for mutants would be a first step towards improved cost reduction techniques. This work introduces such a metric and a firm mutation tool for WS-BPEL compositions. Experimental results obtained by comparing different metrics on several compositions are presented. Copyright © 2014 John Wiley & Sons, Ltd.

    2. Model-based mutation testing from security protocols in HLPSL

      Frédéric Dadeau, Pierre-Cyrille Héam, Rafik Kheddam, Ghazi Maatoug and Michael Rusinowitch

      Article first published online: 29 APR 2014 | DOI: 10.1002/stvr.1531

      Thumbnail image of graphical abstract

      This article proposes a model-based mutation testing approach for security protocols written in HLPSL. Mutation operators for protocol models, expressing real-world implementation choices or mistakes, are presented and evaluated on a large bench of real-world protocols. Finally, it describes a framework that helps automating the execution of the generated test cases. Copyright © 2014 John Wiley & Sons, Ltd.

    3. Using mutation to assess fault detection capability of model review

      Paolo Arcaini, Angelo Gargantini and Elvinia Riccobene

      Article first published online: 25 APR 2014 | DOI: 10.1002/stvr.1530

      Thumbnail image of graphical abstract

      Model review is a static analysis approach that aims at determining if a model owns certain quality attributes. This paper presents a methodology for evaluating the fault detection capability of the NuSMV model advisor. The approach is based on the use of mutation analysis; to improve the quality of the analysis, the equivalence between a NuSMV model and any of its mutants is checked. Some experiments assess the fault detecting capability, precision, and accuracy of the NuSMV model advisor. Copyright©2014 John Wiley & Sons, Ltd.

  6. Special Issue Paper

    1. Towards an automation of the mutation analysis dedicated to model transformation

      Vincent Aranega, Jean-Marie Mottu, Anne Etien, Thomas Degueule, Benoit Baudry and Jean-Luc Dekeyser

      Article first published online: 25 APR 2014 | DOI: 10.1002/stvr.1532

      Thumbnail image of graphical abstract

      Mutation analysis for model transformations is perceived as cumbersome because of the test set improvement manual step that requires a strong expertise and analysis from the tester. This paper presents a solution to go towards a full automation of the mutation analysis for model transformations. This solution uses model transformation traceability and mutation operator modelization embedded into an assistant that helps the tester to easily create new test models to improve the test set. Copyright©2014 John Wiley & Sons, Ltd.

  7. Special Issue Papers

    1. Employing second-order mutation for isolating first-order equivalent mutants

      Marinos Kintis, Mike Papadakis and Nicos Malevris

      Article first published online: 21 APR 2014 | DOI: 10.1002/stvr.1529

      Thumbnail image of graphical abstract

      This paper presents a new technique to tackle the equivalent mutant problem. The salient feature of this approach is the employment of higher-order mutation in the classification process. More precisely, the classification scheme utilises code coverage information and second-order mutation to classify a given set of first-order mutants as possibly killable or possibly equivalent ones. The results suggest that the proposed method outperforms its predecessors with respect to the recall metric, without significant loss of classification precision. Copyright © 2014 John Wiley & Sons, Ltd.

    2. Test generation from recursive tile systems

      Sébastien Chédor, Thierry Jéron and Christophe Morvan

      Article first published online: 2 APR 2014 | DOI: 10.1002/stvr.1525

      Thumbnail image of graphical abstract

      This paper presents a formal conformance test cases generation technique for recursive systems. These systems are modelled by recursive tile systems, which are defined by a finite collection of finite graphs (tiles), each of these equipped with a frontier that expresses how tiles are connected. Both on-line and off-line techniques are presented. Copyright © 2014 John Wiley & Sons, Ltd.

    3. UNICORN: a unified approach for localizing non-deadlock concurrency bugs

      Sangmin Park, Richard Vuduc and Mary Jean Harrold

      Article first published online: 9 MAR 2014 | DOI: 10.1002/stvr.1523

      Thumbnail image of graphical abstract

      UNICORN is an automated dynamic pattern-detection-based technique that finds problematic memory access patterns for non-deadlock concurrency bugs. It detects significant classes of bug types, including order violations and both single-variable and multi-variable atomicity violations. This paper describes the UNICORN approach and its implementations in Java and C++, and evaluates these implementations empirically. Copyright © 2014 John Wiley & Sons, Ltd.

    4. Killing strategies for model-based mutation testing

      Bernhard K. Aichernig, Harald Brandl, Elisabeth Jöbstl, Willibald Krenn, Rupert Schlick and Stefan Tiran

      Article first published online: 3 FEB 2014 | DOI: 10.1002/stvr.1522

      Thumbnail image of graphical abstract

      This paper presents a model-based test case generation approach that automatically derives test cases from UML state machines. The technique is based on the mutation analysis of test models. In the two case studies, different fault-based test case generation strategies are compared. Copyright © 2014 John Wiley & Sons, Ltd.

  8. Research Articles

    1. A model-free and state-cover testing scheme for semaphore-based and shared-memory concurrent programs

      Gwan-Hwan Hwang, Che-Sheng Lin, Teng-Shuo Lee and Chi Wu-Lee

      Article first published online: 13 JAN 2014 | DOI: 10.1002/stvr.1520

      Thumbnail image of graphical abstract

      We present a new framework for performing dynamic testing of semaphore-based and shared-memory concurrent programs. The proposed scheme only has to analyze the synchronization sequences (SYN-sequences) that are collected during the dynamic testing of the concurrent program - static analysis of the syntax and semantics of the program is unnecessary. If the tested concurrent program has an infinite number of SYN-sequences, state-cover testing of the target program can be performed (if the execution states of the target concurrent program are finite). Copyright © 2014 John Wiley & Sons, Ltd.

    2. Exploring the missing link: an empirical study of software fixes

      Maggie Hamill and Katerina Goseva-Popstojanova

      Article first published online: 7 OCT 2013 | DOI: 10.1002/stvr.1518

      Thumbnail image of graphical abstract

      The paper is focused on empirical characterization of software fixes. The results show that significant number of software failures require fixes in multiple software components and/or multiple software artifacts (i.e. 15% and 26%, respectively). The patterns of software components that are often fixed together are significantly affected by the software architecture. Furthermore, the types of fixed software artifacts are highly correlated with fault type, and they have different distributions for prerelease and post-release failures.

  9. Special Issue Papers

    1. Metallaxis-FL: mutation-based fault localization

      Mike Papadakis and Yves Le Traon

      Article first published online: 9 SEP 2013 | DOI: 10.1002/stvr.1509

      Thumbnail image of graphical abstract

      This paper proposes Metallaxis, a fault localization approach based on mutation analysis. The innovative part of Metallaxis is that is it uses mutants and links them with the faulty program places. Thus, mutants that are killed mostly by failing tests provide a good indication about the location of a fault. Experimentation using Metallaxis suggests that it is significantly more effective than statement-based approaches. This is true even in the case that mutation cost-reduction techniques, such as mutant sampling, are facilitated.


  1. 1 - 21