Adaptive Parameter Control for Search-Based Unit Test Generation

Abstract

Testing is a crucial task in software engineering, but one that is time-consuming and expensive. Test generation frameworks, such as Pynguin, are designed to automate the process of creating tests, thus alleviating some of the cost. However, the test suites generated by Pynguin do not always manage to reach 100% code coverage, thus indicating that there is room for improvement. This thesis aimed to explore whether adding reinforcement learning-based parameter control to update parameter values during the test generation process could yield improved code coverage. We created PynguinAPC, a version of Pynguin with the addition of reinforcement learning-based parameter control. To evaluate the impact of parameter control, we performed experimental simulation in two cycles, one where we applied control to individual parameters and one where we applied control to pairs of two parameters for a set of 24 Python modules. The results were analyzed using Bayesian statistical models, concluding that there was no overall gain for the final branch coverage achieved by the generated test suites and the branch coverage growth rate from the application of parameter control. The addition of a parameter control system resulted in an overall increase in the performance overhead. However, some modules were more receptive to parameter control for specific parameters, warranting an investigation of what traits these modules inhabit that caused this increased receptiveness. Paradoxically, by introducing a system to reduce the amount of necessary manual configuration, the system inadvertently introduced additional layers of configuration. These configurable layers could merit further exploration into their impact, as the choices made for this study may be a limiting factor in the results observed.