Selecting a software engineering tool: lessons learnt from mutation analysis
Résumé
Software developers employ many tools in every step of the development. As automation progresses, tools take a more and more important place. A common and difficult problem is choosing a tool among every tool for a given task.
As a particular instance of this problem, this paper considers mutation analysis tools. Mutation analysis is a way to evaluate the quality of a test suite. The quality is measured as the ability of the test suite to detect faults injected into the program under tests. A fault is detected if at least one test case gives different results on the original program and the fault-injected one. Mutation tools aim at automating and speeding both the generation of fault-injected variants, called mutants, and the execution of the test suite on those mutants.
This paper proposes a methodology to compare tools and applies it for comparing mutation tools. This methodology proposes to dress a list of comparison criteria as well as a list of usage profiles. Mutation tools for Java are compared on paper and by experiments. The work is then extended to other languages to assert the pertinence of the comparison criteria and the usage profiles. Finally, lessons are drawn from our selection process.