Mutation Testing in Industry CI: A Value- Centric Approach - A case study about the intersection of software quality and developer experience.

Abstract

Context: Mutation testing is a robust testing technique to assess the sufficiency of test suites. The industry is still struggling to adopt mutation testing and maximize its benefits despite recent research suggesting it is becoming more mature.

Objectives: This study aims to share insights and recommendations to: 1. Assist developers during the integration process of mutation testing tools. 2. Present mutation testing results to maximize their benefit and minimize the cost of using them.

Methods: We perform a case study in an industry setting. We create an experience report that reflects on the integration process of a mutation testing tool at the partner company. We then focus on developer experience by using ten think-aloud sessions and semi-structured interviews to explore what information developers perceive as useful and how the information should be presented.

Results: A CI pipeline was developed to run mutation testing nightly and upload results to a developed dashboard. Integrating mutation testing tools is still a challenging process. When interacting with results, developers valued interactivity, getting an overview, and wanted to associate mutation testing results with other contextual information of the codebase, such as code coverage and complexity. A set of recommendations was created to facilitate integrating and using mutation testing in industry settings.

Conclusion: As-is, mutation testing is a standalone tool that should be easier to integrate and interact with other aspects of the codebase to become more adopted and successful.