Software Testing, Verification and Reliability

Cover image for Vol. 25 Issue 3

Early View (Online Version of Record published before inclusion in an issue)

Edited By: Jeff Offutt and Robert M. Hierons

Impact Factor: 1.2

ISI Journal Citation Reports © Ranking: 2013: 37/105 (Computer Science Software Engineering)

Online ISSN: 1099-1689

Associated Title(s): Journal of Software: Evolution and Process, Software Process: Improvement and Practice, Software: Practice and Experience


  1. 1 - 18
  1. Research Articles

    1. Automatic fault localization for client-side JavaScript

      Frolin S. Ocariza Jr., Guanpeng Li, Karthik Pattabiraman and Ali Mesbah

      Article first published online: 14 APR 2015 | DOI: 10.1002/stvr.1576

      Thumbnail image of graphical abstract

      This work proposes an approach for automatically localizing Document-Object-Model-related (i.e. DOM-related) JavaScript faults in AJAX-based web applications. The technique has been implemented in a tool called AutoFLox, and it is capable of localizing faults in a wide array of web applications, including minified applications, as well as those using anonymous functions and eval. An evaluation of the proposed approach shows that it is accurate, with a recall of 96% and no false positives, and can localize real bugs from real-world web applications.

  2. Special Issue Papers

    1. Coverage-based regression test case selection, minimization and prioritization: a case study on an industrial system

      Daniel Di Nardo, Nadia Alshahwan, Lionel Briand and Yvan Labiche

      Article first published online: 7 APR 2015 | DOI: 10.1002/stvr.1572

      Thumbnail image of graphical abstract

      This paper presents a case study of coverage-based regression testing techniques on a real-world industrial system with real regression faults. The results show that prioritization techniques that are based on additional coverage with finer-grained coverage criteria perform significantly better in fault detection rates, test selection does not provide significant savings in execution cost (<2%), and test suite minimization using finer-grained coverage criteria could provide significant savings in execution cost (79.5%) while maintaining a fault detection capability level above 70%.

    2. Cause reduction: delta debugging, even without bugs

      Alex Groce, Mohammad Amin Alipour, Chaoqiang Zhang, Yang Chen and John Regehr

      Article first published online: 31 MAR 2015 | DOI: 10.1002/stvr.1574

      Thumbnail image of graphical abstract

      This paper presents a generalization of delta debugging (cause reduction) in which test cases are reduced based on an arbitrary property, rather than only with respect to ability to trigger a fault. The graph shows that basing test suites on tests cases that have been reduced with respect to statement coverage can result in much better code coverage for very small regression tests than using unreduced test cases. Cause reduction can also benefit symbolic execution and stress testing.

  3. Research Articles

    1. You have full text access to this OnlineOpen article
      Assessing and generating test sets in terms of behavioural adequacy

      Gordon Fraser and Neil Walkinshaw

      Article first published online: 20 MAR 2015 | DOI: 10.1002/stvr.1575

      Thumbnail image of graphical abstract

      Identifying a test set that adequately captures the essential behaviour of a program is a well-established problem. Behavioural coverage captures the idea that if it is possible to infer an accurate model of a system from its test executions, then the test set is adequate. Our approach enables the use of machine learning algorithms to augment standard syntactic testing approaches and shows how search-based testing techniques can be applied to generate test sets with respect to this criterion.

    2. Anomaly detection in performance regression testing by transaction profile estimation

      Shadi Ghaith, Miao Wang, Philip Perry, Zhen Ming Jiang, Pat O'Sullivan and John Murphy

      Article first published online: 9 MAR 2015 | DOI: 10.1002/stvr.1573

      Thumbnail image of graphical abstract

      We showed how the transaction profile (TP) can be used, via the TP run report, by regression testing teams as an automated way to discover performance regression anomalies. Furthermore, the TP is immune to changes in the load applied to the system, which will save time to repeat runs with the previous release load. Moreover, we proposed a novel way to infer the TP from already available performance data, mainly the resources utilizations and the transaction response times, and the queueing network model of the testing system.

  4. Special Issue Papers

    1. Defect prediction as a multiobjective optimization problem

      Gerardo Canfora, Andrea De Lucia, Massimiliano Di Penta, Rocco Oliveto, Annibale Panichella and Sebastiano Panichella

      Article first published online: 8 MAR 2015 | DOI: 10.1002/stvr.1570

      Thumbnail image of graphical abstract

      When performing defect prediction, a software engineer often wants to achieve conflicting objectives, for example, maximizing the number of identified defect-prone classes and reducing the code inspection cost. In this paper, we formulate the defect-prediction problem as a multi-objective problem and propose an approach called Multi-objective Defect Prediction (MODEP), which uses a genetic algorithm to build defect predictor models. Results of an empirical study indicate that MODEP outperforms single-objective predictors, trivial baselines, and local prediction models.

    2. A study and toolkit of CHECK-THEN-ACT idioms of Java concurrent collections

      Yu Lin and Danny Dig

      Article first published online: 9 FEB 2015 | DOI: 10.1002/stvr.1567

      Thumbnail image of graphical abstract

      A common usage of concurrent collections is composed of two operations where a check on the collection (e.g., collection contains an element) precedes an action (e.g., inserting an element). Unless the whole composition is atomic, the usage contains an atomicity violation bug. This paper presents an extensive empirical study of CHECK-THEN-ACT idioms of Java concurrent collections by analyzing 28 widely used open source Java projects, which shows that such idioms are commonly misused and can lead to bugs.

    3. Higher accuracy and lower run time: efficient mutation analysis using non-redundant mutation operators

      René Just and Franz Schweiggert

      Article first published online: 11 DEC 2014 | DOI: 10.1002/stvr.1561

      Thumbnail image of graphical abstract

      This paper studies redundancies in commonly used mutation operators and provides non-redundant versions of the conditional operator replacement, unary operator insertion, and relational operator replacement mutation operators. The empirical study in this paper shows how the inclusion redundant mutants affect the efficiency and accuracy of mutation analysis. In summary, the total mutation analysis run time decreased by more than 20% by removing redundant mutants, and the inclusion of redundant mutants led to an overestimated mutation score for all analyzed test suites.

    4. MuRanker: a mutant ranking tool

      Akbar Siami Namin, Xiaozhen Xue, Omar Rosas and Pankaj Sharma

      Article first published online: 11 AUG 2014 | DOI: 10.1002/stvr.1542

      Thumbnail image of graphical abstract

      MuRanker introduces and further implements the idea of ranking mutants. The tool ranks mutants according to their complexity in being detected by a test pool. The tool captures the distance between a mutant and the original program in deciding about ranking mutants. The empirical evaluation demonstrates the validity of the hypothesis stating that detecting medium and hard-to-kill mutants first will automatically detect easy-to-kill mutants, and thus, it reduces the number of test cases needed to achieve the mutation adequacy criterion.

    5. Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation

      Shin Hong, Matt Staats, Jaemin Ahn, Moonzoo Kim and Gregg Rothermel

      Article first published online: 23 JUN 2014 | DOI: 10.1002/stvr.1539

      Thumbnail image of graphical abstract

      We explore the impact of concurrency coverage metrics on testing effectiveness and examine the relationship between coverage, fault detection, and test size. We study eight existing coverage metrics and six new metrics formed by combining complementary metrics. Our results indicate that the metrics are moderate to strong predictors of testing effectiveness and effective at providing test generation targets. Nevertheless, metric effectiveness varies across programs, even for the combinations of metrics. This result highlights the need for improving concurrency coverage metrics.

    6. Reducing mutation costs through uncovered mutants

      Pedro Reales Mateo and Macario Polo Usaola

      Article first published online: 15 MAY 2014 | DOI: 10.1002/stvr.1534

      Thumbnail image of graphical abstract

      Mutant Schema Improved with extra Code (MUSIC) annotates the statements covered by test cases in the original program. Later, when mutants are executed, the execution engine avoids going through uncoverable mutants. Our empirical results prove that music is an efficient cost reduction technique. Copyright © 2014 John Wiley & Sons, Ltd.

    7. Quality metrics for mutation testing with applications to WS-BPEL compositions

      Antonia Estero-Botaro, Francisco Palomo-Lozano, Inmaculada Medina-Bulo, Juan José Domínguez-Jiménez and Antonio García-Domínguez

      Article first published online: 9 MAY 2014 | DOI: 10.1002/stvr.1528

      Thumbnail image of graphical abstract

      Mutation testing is a successful testing technique. However, it can be very costly. Cost reduction can be aided by an analysis of mutation operators, but this requires the definition of specialized metrics. A better quality metric for mutants would be a first step towards improved cost reduction techniques. This work introduces such a metric and a firm mutation tool for WS-BPEL compositions. Experimental results obtained by comparing different metrics on several compositions are presented. Copyright © 2014 John Wiley & Sons, Ltd.

    8. Model-based mutation testing from security protocols in HLPSL

      Frédéric Dadeau, Pierre-Cyrille Héam, Rafik Kheddam, Ghazi Maatoug and Michael Rusinowitch

      Article first published online: 29 APR 2014 | DOI: 10.1002/stvr.1531

      Thumbnail image of graphical abstract

      This article proposes a model-based mutation testing approach for security protocols written in HLPSL. Mutation operators for protocol models, expressing real-world implementation choices or mistakes, are presented and evaluated on a large bench of real-world protocols. Finally, it describes a framework that helps automating the execution of the generated test cases. Copyright © 2014 John Wiley & Sons, Ltd.

    9. Using mutation to assess fault detection capability of model review

      Paolo Arcaini, Angelo Gargantini and Elvinia Riccobene

      Article first published online: 25 APR 2014 | DOI: 10.1002/stvr.1530

      Thumbnail image of graphical abstract

      Model review is a static analysis approach that aims at determining if a model owns certain quality attributes. This paper presents a methodology for evaluating the fault detection capability of the NuSMV model advisor. The approach is based on the use of mutation analysis; to improve the quality of the analysis, the equivalence between a NuSMV model and any of its mutants is checked. Some experiments assess the fault detecting capability, precision, and accuracy of the NuSMV model advisor. Copyright©2014 John Wiley & Sons, Ltd.

  5. Special Issue Paper

    1. Towards an automation of the mutation analysis dedicated to model transformation

      Vincent Aranega, Jean-Marie Mottu, Anne Etien, Thomas Degueule, Benoit Baudry and Jean-Luc Dekeyser

      Article first published online: 25 APR 2014 | DOI: 10.1002/stvr.1532

      Thumbnail image of graphical abstract

      Mutation analysis for model transformations is perceived as cumbersome because of the test set improvement manual step that requires a strong expertise and analysis from the tester. This paper presents a solution to go towards a full automation of the mutation analysis for model transformations. This solution uses model transformation traceability and mutation operator modelization embedded into an assistant that helps the tester to easily create new test models to improve the test set. Copyright©2014 John Wiley & Sons, Ltd.

  6. Special Issue Papers

    1. Employing second-order mutation for isolating first-order equivalent mutants

      Marinos Kintis, Mike Papadakis and Nicos Malevris

      Article first published online: 21 APR 2014 | DOI: 10.1002/stvr.1529

      Thumbnail image of graphical abstract

      This paper presents a new technique to tackle the equivalent mutant problem. The salient feature of this approach is the employment of higher-order mutation in the classification process. More precisely, the classification scheme utilises code coverage information and second-order mutation to classify a given set of first-order mutants as possibly killable or possibly equivalent ones. The results suggest that the proposed method outperforms its predecessors with respect to the recall metric, without significant loss of classification precision. Copyright © 2014 John Wiley & Sons, Ltd.

    2. Killing strategies for model-based mutation testing

      Bernhard K. Aichernig, Harald Brandl, Elisabeth Jöbstl, Willibald Krenn, Rupert Schlick and Stefan Tiran

      Article first published online: 3 FEB 2014 | DOI: 10.1002/stvr.1522

      Thumbnail image of graphical abstract

      This paper presents a model-based test case generation approach that automatically derives test cases from UML state machines. The technique is based on the mutation analysis of test models. In the two case studies, different fault-based test case generation strategies are compared. Copyright © 2014 John Wiley & Sons, Ltd.

    3. Metallaxis-FL: mutation-based fault localization

      Mike Papadakis and Yves Le Traon

      Article first published online: 9 SEP 2013 | DOI: 10.1002/stvr.1509

      Thumbnail image of graphical abstract

      This paper proposes Metallaxis, a fault localization approach based on mutation analysis. The innovative part of Metallaxis is that is it uses mutants and links them with the faulty program places. Thus, mutants that are killed mostly by failing tests provide a good indication about the location of a fault. Experimentation using Metallaxis suggests that it is significantly more effective than statement-based approaches. This is true even in the case that mutation cost-reduction techniques, such as mutant sampling, are facilitated.


  1. 1 - 18