Software Testing, Verification and Reliability

This paper mechanises conformance verification in the setting of the CSP process algebra, using the FDR model checker, by establishing compositionality properties for the conformance relation (cspio) with respect to the CSP operators. We explore under which conditions cspio is equivalent to Tretmans ioco. All the results have been mechanically proved using the CSP-Prover. The proposed testing theory has been adopted in an industrial context involving collaboration with Motorola, on testing mobile applications.

A known problem of traditional coverage metrics is that they do not assess oracle quality—that is, whether the computation result is actually checked against expectations. In this paper, we introduce the concept of checked coverage, the dynamic slice of covered statements that actually influence an oracle. Our experiments on seven open-source projects show that checked coverage is a sure indicator for oracle quality and even more sensitive than mutation testing. Copyright ©2013 John Wiley & Sons, Ltd.

Statistical model-based testing is an automated approach to generating test cases for web applications, where logged user behaviour is used to build a usage-based model of web application navigation. This paper reports on our exploratory study of the tradeoffs and implications of the ways to represent user behaviour in a navigation model and the characteristics of the test cases automatically generated from various models. Our results suggest how testers can easily configure statistical model-based automatic test case generators for web applications.

Although intuitively longer tests can achieve higher code coverage, in search-based testing, there is always the threat of bloat, that is, abnormal growth of test length. We present techniques to overcome the problem of length bloat and evaluate all possible combinations of these techniques using different starting lengths for the search. Experiments on a set of difficult search targets, selected from several open source and industrial projects, show that controlling bloat with the appropriate techniques can significantly improve the search performance.

This paper addresses the integration testing of data-centric and event-based dynamic service compositions, in which abstract services are replaced by concrete candidate services at runtime. We consider data dependencies between services as potential points of failure and introduce the k-node data flow test coverage metric, which helps to significantly reduce the number of generated test combinations. We illustrate the end-to-end practicability based on an integrated scenario, which uses two diverse service composition techniques (Web Services Business Process Execution Language and WS-Aggregation).

An iterative approach is presented that combines techniques from grammatical inference and model-based testing domains to fulfil two main purposes: (i) reverse engineering of black-box components by inferring (partial) models and (ii) validating integrated systems of such components for compositional problems and generic errors by using their inferred models. The paper investigates how the ‘oracle’ assumption in the classic learning theory can be avoided practically and furthermore how existing model-based approaches can be exploited for testing components whose models are unavailable.

In this paper, model checking-based techniques are shown to complement the existing random inputbased and simulation-based methods in model-based testing of safety-critical embedded systems. They are also shown to enhance the effectiveness of test coverage by providing proofs of unreachability that also aids in debugging. Copyright © 2013 John Wiley & Sons, Ltd.

Software quality is very important in today's competitive business environment. Managing software quality risk is an important challenge. The conventional approach to quality risk management for ongoing software processes uses analytic models and structured methodologies to enhance processes and improve quality are not systematically applied, which are major deficiencies. This new method uses Monte Carlo Simulation and Six Sigma for ongoing quality risk management. Simulation predicts quality (reliability) and identifies and quantifies the risk. DMAIC (Define, Measure, Analyze, Improve, Control), tactically applied, enhances the process and improves quality. Simulation is superior to conventional risk models. DMAIC is a verified structured methodology for systematic process and quality improvements. These synergetic and significant enhancements eliminate observed deficiencies. The method has been successfully proven and applied practically on real in-house projects. Such an application and obtained results are presented. Substantial savings, quality improvements and customer satisfaction were achieved. This CMMI® compliant method offers very important gains. Copyright © 2013 John Wiley & Sons, Ltd.

Randomized algorithms are widely used to address many types of software engineering problems, but they are affected by chance and so require the use of appropriate statistical tests to be properly analysed. To address this issue, this paper provides guidelines on how to carry out and properly analyse randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing.

The SECCO approach ensures that test coverage achieved at source code is preserved during optimizing compilation. A typical use case of coverage preservation is source-based test-data generation. Experiments show that the guaranteed coverage preservation has a negligible impact on the resulting code performance.

This case study describes the motivations and design decisions behind the development of the testing tool, CoTester, and its deployment to multiple development teams. CoTester outperforms available testing tools by representing tests using an easy-to-understand scripting language, thus making the tests easily editable. A series of deployments to four development teams showed that CoTester worked very well for non-professional testers, and it was also found to be useful by some professional testers.

This paper presents methods for converting the Trampoline kernel code into formal models for the model checker SPIN and a series of experiments using an incremental verification approach. The conversion methods include functional modularization and treatment for hardware-dependent code, such as memory access for context switching. It also reports on potential safety issues found in the Trampoline operating system during the experiments and presents experimental evidence of the performance improvement using the embedded C constructs in SPIN. Copyright © 2012 John Wiley & Sons, Ltd.

This paper presents a model-based statistical testing (MBST) approach for generating functional test cases. The proposed approach consists of eight activities as shown in the figure. These activities answer the three major software testing research questions: (i) how to select test inputs; (ii) how to predict the expected results of a test; and (iii) when to stop testing software. This approach was experimented on two representative case studies from the automotive industry: more bugs were detected earlier and in a shorter time.

This paper presents a combined weak and strong mutation for noninterpretive Java mutation system. The new term ‘serialmutant’ is defined as a specialized program to conduct weak mutation against all mutants in an execution and report only weakly killed mutants as strong mutation candidates. Then strong mutation is conducted only for those reported mutants.

Fastest provides tool support for a solid, model-based testing framework for the Z notation known as the Test Template Framework (TTF). The Test Template Framework uses only Z and is aimed mainly to unit testing. The tool automates testing tactic application and semi-automates elimination of unsatisfiable test specifications, generation of abstract test cases and translation of test cases into natural language.

The study presents a comparative framework, including seven properties, which characterize the change impact analysis (CIA) techniques, identifies key applications of CIA techniques in software maintenance, and discusses the need for further research.

This paper proposes a new incremental testing strategy that turns the test of finite state machines (FSMs) a much more scalable process. The C-method has been developed to efficiently test modular compositions of FSMs, thus alleviating the effect of traversal sets on the size of generated test suites for such models. An analysis of an infinite family of FSMs shows that incremental testing can be exponentially more efficient than testing using the traditional W-method.

In this paper, we examine whether changes in coverage can be used to detect non-equivalent mutants. In a sample of 140 manually classified mutations of seven Java programs with 5000 to 100 000 lines of code, we found that: the problem is serious and widespread (45% of all undetected mutants are equivalent); manual classification takes time (15 min per mutation); coverage is a simple and effective means to identify equivalent mutants (with a classification precision of 75% and 56% recall).

Mutation testing of multithreaded code is highly time intensive: for each mutant, every test case may be explored for multiple possible thread schedules. This paper introduces a general framework for efficient exploration that can reduce the time for mutation testing of multithreaded code. The paper presents five techniques within the framework that are implemented in a tool calledMuTMuT. Evaluation of MuTMuT on mutation testing of 12 multithreaded programs shows that it indeed substantially reduces the time required.

This paper presents a linear programming-based approach to timing analysis of scenario-based specifications (SBSs) expressed by UML interaction models. With more general and expressive timing constraints in UML sequence diagrams, the algorithms in the approach solve the problems of the reachability, constraint conformance and bounded delay analysis of SBSs. These algorithms form a decision procedure for the loop-unlimited SBSs where any loop in any path is time-independent of the other parts in the path, and a semi-decision procedure for general SBSs. Copyright © 2010 John Wiley & Sons, Ltd.

Test suite minimization, Regression Test Selection (RTS) and test case prioritization are all techniques that aim to reduce the cost of regression testing. This paper presents the first survey of all three of these closely related areas and discusses open problems and the potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.