Implementation and Evaluation of Automatic Prioritization for Continuous Integration Test Cases

Abstract

Nowadays, continuous integration is widely accepted and implemented by most industrial companies due to the benefits of adding new functions and accelerating the delivery of new products. But at the same time, frequent integration brings challenges, especially to large software companies. One challenge in the case company is to prioritize the test cases that are most likely to fail first in order to minimize the feedback loop from change to test result. In order to guarantee the product quality, a significant number of tests shall be executed after every integration, which has high demands on resources and thus has are associated with a high cost and time consumption. This thesis proposes a method based on mining correlations between historical data of test results and modified files. The Matthews Correlation Coefficient (MCC) and a scoring curves refers to Average Percentage of Faults Detected (APFD) are used to evaluate results and determine correlations. We aim to create an automatic way to provide a list of test cases that prioritized by their probability to fail, and further help the case company to shorten their feedback loop and time to market.