A Framework for Minimising Java Microbenchmark Suites - A Metric-Based Approach

Abstract

Performance is an important non-functional requirement. The success of software projects is highly dependent on the adequate performance of the software. However, one fundamental problem with performance testing is that it can become immensely time-consuming to conduct. In this paper, several different code metrics were explored and used for conducting test case minimization (TCM) of Java Microbenchmark Harness (JMH) suites. The microbenchmark suites were created with a tool called ju2jmh, which creates microbenchmark suites based on already existing unit testing suites of a project. The metrics used for conducting TCM were lines of code (LOC), loop count, cyclomatic complexity, and the combination of these metrics. The results indicate that suites derived using LOC and loop count performed worse than randomly sampled suites, while suites derived from the cyclomatic metric and the combination of LOC, loop count, and cyclomatic complexity performed similar or slightly worse than randomly sampled suites. Furthermore, it is not infeasible to assume that if these metrics were refined by utilising dynamic techniques, some of them could potentially outperform randomly sampled suites consistently.