An Empirical Study on the Trade-off of Diversity Types and Measures on Test Visualisation

Abstract

Test diversity is a well-researched topic and although much has been done in this area one part that is lacking is the investigation on how a functional dashboard can bring awareness to diversity into the software engineers’ toolkit. To expand that toolkit we have performed several types of analysis and an experiment where we address, i) What are the trade-offs in diversity for using different diversity measures (e.g., Jaccard, Levenshtein, NCD) and types of test artefacts (e.g., input, output), ii) what are the visual and structural differences on the clusters after using different dimensionality reduction techniques (e.g., MDS, t-SNE, UMAP) and iii) to what extent do different visual techniques influence decisions about test case selection. Our key findings are that Jaccard and Levenshtein perform similarly and NCD performs worse as expected on smaller data. However, Jaccard runs faster which gives the edge to Jaccard. When it comes to dimensionality reduction techniques, we have found that t-SNE and UMAP have a better effect on the structure of similarity maps compared to MDS. Furthermore, UMAP has a slight edge over the other techniques because of its hyperparameters being easier to understand and performs faster. Finally, when it comes to developing dashboards that visualise diversity it is worth pointing out that the colour of the data has a big effect on decision making, even better if the colours are discrete and are coupled with a visual technique (such as making a test case’s data point bigger if it found more faults). Additionally, we found that sliders and labels used to filter the data are not used much by developers when performing tasks on diversity visualisations. However, using these filtering techniques is recommended because they have the potential to spark discussion with people who are not as invested in the lower level details of the project (e.g., code, tests), such as managers, who possess more of a holistic view in most cases.