Recommendations for Mutation Testing as Part of a Continuous Integration Pipeline: With a focus on C++

Abstract

Mutation testing aims to judge the quality of a test suite by evaluating the ability of the tests to detect subtle changes in the code. Altered versions of the code called mutants are generated by a mutation tool, often a subtle change based on common syntactic mistakes. Each mutant is then run against the original test suite to see if the test results change. This thesis aims to identify recommendations to mitigate the fact that mutation testing can be notoriously expensive, making it difficult to integrate as part of a continuous integration (CI) pipeline. An evaluation of existing mutation testing tools for C++ was done to better understand the functionality and limitations of the available tools. Simultaneously, a case study was conducted at Zenseact to identify suggestions on how mutation testing can best be used within CI. First, a literature review was conducted to examine observations made by other studies on the application of mutation testing in practice, and to identify techniques and practices to perform mutation testing effectively. Next, stakeholder interviews were conducted to assess the developer’s view of effective mutation testing. The result of the tool evaluation showed that existing mutation tools for C++ differ in their capabilities, and the case study result showed that there exist plenty of techniques to effectively integrate mutation testing into CI from the developers’ perspective. Based on the result, only the mutation tools Dextool and Mull were recommended for CI integration. Furthermore, guidelines were developed such as prioritizing mutation testing on essential areas of the code, performing mutation testing on old code only when there is free machine time, and that inexperienced developers should be trained to understand the mutation result better.