An Extensible Framework for Large-Scale Mining of Performance Microbenchmarks

Abstract

Many performance issues are caused by low-level implementation details, which are also more likely to be the cause of failures in software-intensive systems than erroneous feature implementations. A microbenchmark is a type of performance test that focus on measuring the performance of smaller, yet critical, code sections as opposed to a whole system. It is a complex task to design useful and correct performance microbenchmarks, and the lack of tool support is part of the reason. A goal of mining software repositories (MSR) research is to motivate the development of tools. To facilitate performing mining studies on a large scale mining frameworks can be used. This study focus on designing and implementing an open-sourced extensible framework for large-scale mining of software performance microbenchmarks. By mining existing microbenchmarks and their evolution researchers and practitioners could gain knowledge valuable for their work on tools facilitating the creation of correct and useful performance microbenchmarks. Thus, the objective is to find out if the created framework can be considered good enough for its purpose. An action research was conducted where MicroAnalyzer, the framework presented in this thesis, was designed and implemented. The study involved eliciting requirements for the framework and creating an architecture allowing mining studies to be highly reproducible. Two mining studies were performed in order to validate the framework. The results of this study show that MicroAnalyzer automates the major parts of the mining process and can be useful for large-scale mining of performance microbenchmarks. Mining studies performed with the framework can be considered highly reproducible.