JUnit-to-JMH: Automatic Generation of Performance Benchmarks from Existing Unit Tests in Java

Abstract

Software microbenchmarks have been proposed as a tool that could be used for performance regression testing, similarly to how unit tests are used for functional regression testing. However, studies on open-source Java projects have shown that unlike unit tests, suites of good microbenchmarks that could be used for performance regression testing are absent from most projects. To address this lack of usable microbenchmarks in existing projects, this thesis proposes an approach for automatically generating benchmarks from existing suites of unit tests, and presents a prototype implementation of the approach called JUnitto- JMH, capable of generating JMH benchmarks from existing JUnit 4 unit tests. The prototype was evaluated on the unit test suite of a real-world open source Java project as well as on a set of specially crafted workloads. The prototype was found to be able to produce compiling benchmarks for all of the study subject’s JUnit tests, of which 98.86% were able to run without executiontime errors, taking just under 30 seconds on average to produce the full suite of 11363 benchmarks. The generated benchmarks were found to produce more stable performance measurements compared to timing measurements obtained from executing the original unit tests. Generated benchmarks were also found to be able to detect performance differences between specially crafted workloads of different size, albeit less effectively than hand-written JMH benchmarks.