Exploring Software Evolution with Class Role Stereotype Visualisation Master’s thesis in Software Engineering KIN YAN LEE Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY UNIVERSITY OF GOTHENBURG Gothenburg, Sweden 2021 Master’s thesis 2021 Exploring Software Evolution with Class Role Stereotype Visualisation KIN YAN LEE Department of Computer Science and Engineering Chalmers University of Technology University of Gothenburg Gothenburg, Sweden 2021 Exploring Software Evolution with Class Role Stereotype Visualisation KIN YAN LEE © KIN YAN LEE, 2021. Supervisor: Truong Ho-Quang, Department of Computer Science and Engineering Examiner: Jennifer Horkoff, Department of Computer Science and Engineering Master’s Thesis 2021 Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg SE-412 96 Gothenburg Telephone +46 31 772 1000 Gothenburg, Sweden 2021 iv Exploring Software Evolution with Class Role Stereotype Visualisation KIN YAN LEE Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Abstract A major challenge for software developers in an organisation is to understand the software systems so as to perform software evolution tasks. The conventional ap- proach is to examine the code, which is often arduous and time-consuming. On the other hand, visualisation is often viewed as a better way to represent data. As the usefulness of class role stereotypes has been investigated, this study is conducted to shed light on the use of evolution data of role stereotypes in visualisation in relation to performing software evolution tasks. This study aims to develop a visualisation approach, which focuses on the evolution of class role stereotypes in object-oriented software systems, using the design science research approach, along with a user study evaluating the approach in performing software evolution tasks. The user study was conducted with six participants with the context of utilising the tool in the understanding part of evolution tasks, and they were asked to work on two tasks, one with the visualisation and other without. Qualitative analysis was carried out on the data collected from the survey and the video recordings of the tasks. The analysis shows the use of the visualisation tool, named Rologram, appears to help identify the changes of responsibility and collaborations of the classes, in comparison to the approach without. It is concluded that the visualisation approach tends to be helpful in performing software evolution tasks. Keywords: software evolution, data visualisation, object-oriented programming, class role stereotypes, software maintenance. v Acknowledgements I would like to express my sincere gratitude to my supervisor Truong Ho-Quang for his guidance and support in particular. I am also grateful to the experts who provided constructive feedback on the visualisation, as well as the participants in the user study. Kin Yan Lee, Gothenburg, June 2021 vii Contents List of Figures xi List of Tables xiii 1 Introduction 1 1.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Statement of the Problem . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Purpose of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Contribution of the Study . . . . . . . . . . . . . . . . . . . . . . . . 2 1.5 Structure of the Paper . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Background and Related Work 5 2.1 Role Stereotypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Usefulness of Role Stereotypes . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Interaction and Software Visualisation Techniques . . . . . . . . . . . 7 2.4 Existing Role Stereotype Visualisation Tool – RoleViz . . . . . . . . . 8 2.5 Alternatives of Software Evolution Visualisation Tools . . . . . . . . . 9 2.5.1 Visualising Change History . . . . . . . . . . . . . . . . . . . 9 2.5.2 Visualising Software Architecture - EVA . . . . . . . . . . . . 10 3 Research Questions 11 4 Methods 13 4.1 Research Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.2 Design Science Research Approach . . . . . . . . . . . . . . . . . . . 13 4.3 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.3.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5 Visualisation Tool 21 5.1 Role Stereotype Encoding . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2 Representations of Class, Package, System and Dependency . . . . . 21 5.3 Class Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.4 Abstraction Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 ix Contents 5.5 Class Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.6 Timeline Constitution . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.7 Pairwise Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.8 Consecutive Comparison . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.9 Pattern Finder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.10 Role Stereotype Changed Indication . . . . . . . . . . . . . . . . . . . 35 5.11 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.12 Interaction Components . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.13 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6 Results 39 6.1 Design Science Research . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.1.1 First Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.1.2 Second Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.1.3 Third Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6.2 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.2.1 Demography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.2.2 Task Performance . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.2.3 Other Quantitative Data . . . . . . . . . . . . . . . . . . . . . 47 6.2.4 Theoretical Constructs, Themes and Codes . . . . . . . . . . . 48 6.2.5 Theoretical Narrative . . . . . . . . . . . . . . . . . . . . . . . 52 6.2.5.1 Approach with the Visualisation . . . . . . . . . . . 52 6.2.5.2 Approach without the Visualisation . . . . . . . . . . 55 6.2.5.3 Benefits of the Visualisation . . . . . . . . . . . . . . 57 6.2.5.4 Improvement on the Visualisation . . . . . . . . . . . 59 7 Discussion 61 7.1 Answers to RQ1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 7.2 Answers to RQ2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 7.2.1 SQ 2.1: Is the proposed visualisation approach/tool helpful? . . 64 7.2.2 SQ 2.2: What approach do developers use? (with and without the tool) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 7.2.3 SQ 2.3: How can we improve Rologram? . . . . . . . . . . . . 67 8 Limitations and Delimitations 69 9 Conclusion 71 References 73 A Appendix I A.1 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I x List of Figures 2.1 Relationships between role stereotypes (a) and a common collabora- tion pattern between role stereotypes (b) in K-9 Mail . . . . . . . . . 7 2.2 RoleViz visualising K-9 Mail with role stereotypes . . . . . . . . . . . 8 2.3 Visualisation of evolution of Tomcat using change history . . . . . . . 9 2.4 Three types of views in EVA . . . . . . . . . . . . . . . . . . . . . . . 10 4.1 Research process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4.2 Procedure of conducting the user study with one participant . . . . . 18 4.3 Inputs and outputs for the steps in data analysis . . . . . . . . . . . . 19 5.1 Representations of class, package and system in the node-link diagram 22 5.2 Representation for dependencies . . . . . . . . . . . . . . . . . . . . . 22 5.3 Visualising lines of code metric . . . . . . . . . . . . . . . . . . . . . 22 5.4 Graph at the package level . . . . . . . . . . . . . . . . . . . . . . . . 23 5.5 Package-level dependencies . . . . . . . . . . . . . . . . . . . . . . . . 24 5.6 Graph at the class level . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.7 Visualisation with the default values for dependency type and level . 25 5.8 Visualisation showing the outgoing dependencies with three levels . . 26 5.9 Timeline constitution at the system level (top), package level (middle) and class level (bottom) . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.10 Visualisation at the package level . . . . . . . . . . . . . . . . . . . . 27 5.11 Visualisation at the class level . . . . . . . . . . . . . . . . . . . . . . 28 5.12 Timeline with current and comparing versions annotated . . . . . . . 28 5.13 Pairwise comparison at the system level . . . . . . . . . . . . . . . . . 29 5.14 Pairwise comparison at the package level . . . . . . . . . . . . . . . . 30 5.15 Comparison of source code in two versions . . . . . . . . . . . . . . . 31 5.16 Combination of changes of role stereotype and collaborating classes at the class level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 5.17 Slider and changes between two versions . . . . . . . . . . . . . . . . 32 5.18 Results of the user-defined pattern for the first class . . . . . . . . . . 33 5.19 Results of the user-defined pattern for two levels . . . . . . . . . . . . 34 5.20 Results of the selected pattern in pattern ranking list . . . . . . . . . 34 5.21 Results of the selected common pattern . . . . . . . . . . . . . . . . . 35 5.22 Highlighting classes which have changed in role stereotype . . . . . . 35 5.23 Visualisation with Information Holder and Interfacer filtered out . . . 37 6.1 Screenshot of the visualisation for the evaluation in the first iteration 39 xi List of Figures 6.2 Screenshot of the visualisation for the evaluation in the second iteration 40 6.3 Screenshot of the visualisation for the final evaluation . . . . . . . . . 42 6.4 Distribution of gender (left) and age (right) . . . . . . . . . . . . . . 44 6.5 Distribution of Java (left), programming (middle) and work experi- ence (right) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.6 Areas the participants had experience in . . . . . . . . . . . . . . . . 45 6.7 Attitude of the participants towards the visualisation . . . . . . . . . 48 xii List of Tables 2.1 Six role stereotypes characterising the responsibility of an object . . . 6 2.2 Seven categories of interaction techniques . . . . . . . . . . . . . . . . 7 5.1 Role stereotype and colour conversion . . . . . . . . . . . . . . . . . . 21 6.1 Comments from the second evaluation . . . . . . . . . . . . . . . . . 41 6.2 Comments on the visualisation from the final evaluation . . . . . . . 43 6.3 Combinations and orderings for the tasks and tools . . . . . . . . . . 45 6.4 Total points for tasks with and without visualisation . . . . . . . . . 46 6.5 Time taken for tasks with and without visualisation (MM:SS.ss) . . . 46 6.6 Points for question 1 regarding changes of class-level dependencies with and without visualisation . . . . . . . . . . . . . . . . . . . . . . 46 6.7 Points for question 2 regarding changes of package-level dependencies with and without visualisation . . . . . . . . . . . . . . . . . . . . . . 47 6.8 Points for question 3 regarding the responsibility of a class with and without visualisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.9 Points for question 4 regarding the changes of responsibility of a class with and without visualisation . . . . . . . . . . . . . . . . . . . . . . 47 xiii List of Tables xiv 1 Introduction 1.1 Context Software evolution involves two major areas: design for changing a software sys- tem over time and analysis of software histories to detect regularities and anomalies (Diehl, 2007, p. 129). Analysing the software evolution data in order to better per- form changes to software systems is not uncommon, especially structural changes. Developers that are new to the systems likely find it arduous to work on software evolution tasks. In this sense, software evolution visualisation helps present the mas- sive amount of data in a more accessible and understandable way, which facilitates the developers in performing those tasks. Visualising software evolution data is a subject that is studied by many researchers with proposed visualisation tools of software evolution (Eick et al., 2002; Holt & Pak, 1996; Lanza & Ducasse, 2002; Nam et al., 2018; Rysselberghe & Demeyer, 2004). There are also numerous studies focusing on software visualisation techniques (Bani-Salameh et al., 2016; Kienle & Müller, 2007) and interactions in information visualisation (Yi et al., 2007). A software evolution task often involves comprehension of the software system. The six role stereotypes proposed by Wirfs-Brock (2006) help understand the responsibil- ity of the classes in object-oriented software systems. Role stereotypes with depen- dencies between the classes show the relationships and interactions between different roles, for example, a class with the Information Holder role provides information to a Controller class. By getting the information of role stereotypes of all classes and how they collaborate, developers may tell the structure of a system. During the software evolution process, refactoring and alterations to the system structure are often inevitable, for instance, splitting a large class into several smaller classes, role stereotype changes are probably involved as the class structures are modified and dependencies to other classes are created. Change of role stereotypes can be seen in 5 to 14% of the classes in each of the three software systems investigated in the study of the evolution of role stereotypes (Fröding & Ngoc, 2020). Design degradation is not uncommon as a side effect of software evolution since developers may not have the original designs in mind when fixing bugs and enhanc- ing parts of the system. With the evolution data of role stereotypes, developers can possibly spot changes to system structures, which may help identify any incon- 1 1. Introduction sistencies and contradictions to the designs or evaluate whether a redesign is needed. The study of Ho-Quang et al. (2020) demonstrates the usefulness of role stereo- types in comprehending software systems with the proposed visualisation tool Role- Viz. Fröding and Ngoc (2020) study the evolution of role stereotypes and the anti- patterns identified in relation to them in three open-source software systems. With these studies as inspirations, it is interesting to look into whether visualising changes of role stereotypes could help developers perform software evolution tasks. 1.2 Statement of the Problem Visualising software data such as the numbers of methods, attributes and lines of code has been explored by numerous studies. In comparison with visualising these measurements, studies on role stereotypes, which are important information with a higher abstraction level that potentially help developers understand the system structures and class behaviours, are scarce. There are currently no software evolution visualisation tools showing the class role stereotypes available, and little research investigating the usefulness of the changes of class role stereotypes. As a software system evolves, huge costs and efforts are needed to understand how the system changes. This raises the questions of whether an interactive visualisa- tion tool highlighting the changes of class role stereotypes in the software evolution process could help reduce the time and efforts when working on software evolution tasks and how. 1.3 Purpose of the Study The purpose of this study is to validate the usefulness of software evolution visu- alisation focusing on class role stereotypes by creating an interactive visualisation and conducting a user study to evaluate how the visualisation can help developers understand the software systems and perform evolution tasks. 1.4 Contribution of the Study The study intends to propose and evaluate an interactive visualisation approach for developers performing software evolution tasks, showing software evolution data with focus on class role stereotypes. The main contributions of this study are the visualisation approach demonstrating the evolution of software systems, particularly the changes of class role stereotypes, the evaluation of the tool with the scenario of developers performing software evolution tasks and the approaches used with and 2 1. Introduction without the visualisation tool. IT practitioners and software developers can benefit from the results of this study if the visualisation of evolution of class role-stereotypes helps perform software evo- lution tasks and understand software systems. The study can be beneficial for researchers considering it would be an empirical piece added in studying the usefulness of visualising class role stereotypes in relation to software evolution tasks. This study would also provide researchers with data and knowledge on the visualisation approach and the evaluation. 1.5 Structure of the Paper This report consists of nine chapters. Chapter 1 introduces the problem to be addressed, purpose and the contribution of the study. Chapter 2 discusses the background and related work on role stereotypes and their usefulness, visualisation techniques, as well as existing role stereotype visualisation and software evolution visualisations. The research questions are presented in Chapter 3. Chapter 4 ex- plains the methods adopted in this study. Chapter 5 introduces the visualisation approach and the implementation. Chapter 6 describes the results of the study. Chapter 7 presents the answers to the research questions. Chapter 8 discusses the limitations and delimitations of the study. Lastly, Chapter 9 concludes the study and mentions the future work. 3 1. Introduction 4 2 Background and Related Work This chapter provides some background knowledge on role stereotypes and intro- duces the related work with focus on the usefulness of role stereotypes, techniques for information visualisation, and existing visualisation tools on role stereotypes and software evolution. 2.1 Role Stereotypes Responsibility-driven design proposed by Wirfs-Brock (1990) is an approach of object-oriented design. She defines role as a set of relevant responsibilities, responsi- bility as an obligation to perform a task or know information, and collaboration as an interaction of objects or/and roles (Wirfs-Brock & McKean, 2002). An object takes a specific role at a time and fulfils a responsibility with or without collaborating with other objects; the collaboration patterns disclose the control and information flow in the system, such that misassignment of responsibility to a class can be revealed (Wirfs-Brock, 1990, pp. 90-91). The notion of role stereotypes is fundamental to this study. Wirfs-Brock (2006) identifies six role stereotypes as simplified roles that carry predefined responsibili- ties for classes and interfaces in object-oriented software systems, namely Controller, Coordinator, Information Holder, Interfacer, Service Provider and Structurer. Not only do these role stereotypes help understand the class behaviours, but also the design patterns adopted in the software systems (Wirfs-Brock, 2006). Table 2.1 lists the role stereotypes with description of their responsibilities. 5 2. Background and Related Work Role stereotype Responsibilities Controller To make decisions and control complicated tasks. Coordinator To delegate work to other objects rather than mak- ing lots of decisions. Information Holder To know information and supply it to other ob- jects. Interfacer To transform requests or information between dif- ferent parts of a system. To interact with the user or communicate with external systems or between subsystems. Service Provider To perform certain work and provide services to other objects on demand. Structurer To maintain relationships between various objects and hold information regarding these relation- ships. Table 2.1: Six role stereotypes characterising the responsibility of an object 2.2 Usefulness of Role Stereotypes Ho-Quang et al. (2020) create the tool RoleViz visualising role stereotypes and com- pare it to the existing industrial visualisation tool Softagram, and it appears that the participants using RoleViz had better performance than those with Softagram when carrying out software comprehension tasks in the user study. Fröding and Ngoc (2020) investigate the changes of role stereotypes among different versions of three software systems and adopt the data to identify anti-patterns. The distribution of role stereotypes has changed remarkably between several versions in one of the software systems as shown in the results of their study. It is found that the distribution of anti-patterns in certain role stereotypes seems to have connection with the changes of the roles’ distribution. Their study on the evolution of role stereotypes (Fröding & Ngoc, 2020), is based on the data of three open-source software systems generated from a role stereotype classifying tool using machine learning, namely Class Role Identifier (Ho-Quang et al., n.d., as cited in Fröding & Ngoc, 2020). The tool provides 74% and 88% ac- curacy rate using the Random Forest algorithm without and with the use of the synthetic minority oversampling technique (SMOTE) respectively for the classifica- tion of different role stereotypes (Ho-Quang et al., n.d.). Nurwidyantoro et al. (2019) illustrate applications of role stereotypes, one of which is the patterns of collaboration between role stereotypes. Figure 2.1 (a) shows the rela- tionships between the role stereotypes in the K-9 Mail software system, in which the most common collaboration is Service Provider with Information Holder. Figure 2.1 (b) shows a common collaboration pattern between role stereotypes in the system, where Interfacer reacts to events of user interface by communicating to Coordinator, 6 2. Background and Related Work and then the Coordinator divides the tasks and forwards the information/requests to the associated Service Provider and Information Holder classes (Nurwidyantoro et al., 2019). Recurring collaboration patterns indicate the regularities in the design of software systems (Nurwidyantoro et al., 2019). Figure 2.1: Relationships between role stereotypes (a) and a common collaboration pattern between role stereotypes (b) in K-9 Mail 2.3 Interaction and Software Visualisation Tech- niques Yi et al. (2007) emphasise on the importance of interaction components in informa- tion visualisation and propose seven categories of interaction techniques, i.e. select, explore, reconfigure, encode, abstract/elaborate, filter and connect, in supporting user intents. Table 2.2 includes the description for the categories. Category Description Select Mark a specific item Explore Show subset of data Reconfigure Arrange in different way Encode Display in different representations Abstract/Elaborate Adjust data abstraction levels Filter Change according to different conditions Connect Display only relevant items Table 2.2: Seven categories of interaction techniques Bani-Salameh et al. (2016) propose six functional requirements for software evolu- tion visualisation tools, including views, details-on demand, filter and search, select, re-arrangement, and comparison, giving the ideas of the necessary functionalities a 7 2. Background and Related Work software evolution visualisation should have. Kienle and Müller (2007) identify the key quality attributes and functional requirements of software visualisation tools, which comprise views, abstraction, search, filters, code proximity, automatic lay- outs, undo/history and miscellaneous. 2.4 Existing Role Stereotype Visualisation Tool – RoleViz RoleViz is the only visualisation available visualising the role stereotypes of the classes in software systems (Figure 2.2). Polymetric view is used in the visualisa- tion, with the metrics of fan-in, i.e. the number of units depends on the unit (height), fan-out, i.e. the number of units the unit depends on (width) and role stereotype (colour) incorporated in the compilation units, which are represented by coloured box inside a larger grey box (package). Package-level dependencies are illustrated by bimetric lines showing the number of dependencies between the packages in both directions. The visualisation has also adopted some interaction techniques includ- ing hovering over a package to view its full path and get the relevant dependencies highlighted. Figure 2.2: RoleViz visualising K-9 Mail with role stereotypes The visualisation proposed in this paper basically adopts the same colours used in RoleViz to encode class role stereotypes, with some adjustments made. The 8 2. Background and Related Work intention of RoleViz is to visualise only one specific version of an object-oriented software system, incapable of illustrating the evolution of a system. On the contrary, the focus of the proposed visualisation approach is on the evolution of a software system. 2.5 Alternatives of Software Evolution Visualisa- tion Tools 2.5.1 Visualising Change History Rysselberghe and Demeyer (2004) introduce a visualisation on the evolution of the open-source system Tomcat based on the information extracted from Concurrent Versions System (CVS) including revision number, file name, author, date, time, lines changed and log message for each revision. Time and files are the two di- mensions used in the visualisation. The files are ordered by the full path name. A dot implies a change committed for one file at a specific time. The vertical line pattern indicates that the file has changed frequently across the revisions. Figure 2.3 shows the evolution of the change history of all files in Tomcat from 1999 to 2004. Figure 2.3: Visualisation of evolution of Tomcat using change history It is interpreted that badly designed components can be located by more dense lines in the visualisation as frequent changes are needed in order to synchronise with other parts of the system, and coherent entities can be identified by similar change pat- terns as the entities are changed simultaneously (Rysselberghe & Demeyer, 2004). Moreover, horizontal lines with no or little dots indicate a low productivity of the development team. 9 2. Background and Related Work The visualisation proposed by Rysselberghe and Demeyer (2004) is capable of visu- alising the evolution of the software system using a single diagram; however, it only focuses on the change history and lacks the evolution of the system structure. 2.5.2 Visualising Software Architecture - EVA EVA is a visualisation tool that illustrates software evolution with the aspect of software architecture and supports three different kinds of views: Single-Release Architecture (see Figure 2.4 (a)), 3-D Architecture-Evolution (see Figure 2.4 (b)) and Pairwise Architecture-Comparison (see Figure 2.4 (c)) (Nam et al., 2018). The tool extracts the issue data from the user-specified repository and uses them to map code-level entities to the corresponding architectural components. As shown in Figure 2.4, smaller coloured circles represent code-level entities, whereas the larger circles depict architectural components. The code-level entities from the same pack- age are encoded with the same colour. Stacked layers and lines are used to illustrate the changes from multiple versions, and the pairwise changes including the addition and deletion of code-level entities are annotated with labels on the circles. One can locate the changes of entity existence by the labels inside the circles which is efficient. The visualisation provides comparison of multiple versions, nevertheless, one needs to hover over each circle to view the changes and get the information of the entities, in which a considerable number of user actions is required in order to locate an entity or view the changes of all the entities. Figure 2.4: Three types of views in EVA 10 3 Research Questions The study involves two research questions regarding software evolution visualisation (RQ1) and evaluation of the visualisation approach (RQ2). RQ 1: How can data of evolution of class role stereotypes of a software system be visualised? This aims to suggest an approach for visualising the changes of class role stereo- types. The input of class role stereotype data is from the related studies, whereas other relevant software evolution data is extracted with tools. As an outcome of answering RQ1, an interactive visualisation tool that visualises the evolution data of role stereotypes change is built. RQ 2: How does the proposed visualisation approach/tool help developers perform software evolution tasks? This part aims to investigate and evaluate the visualisation approach from RQ1 with the following sub questions. SQ 2.1: Is the proposed visualisation approach/tool helpful? SQ 2.2: What approach do developers use? (with and without the tool) SQ 2.3: How can we improve Rologram? 11 3. Research Questions 12 4 Methods 4.1 Research Process This study adopts the design science research approach for the implementation of the visualisation (to answer RQ1) and a user study was conducted as part of the design science research and also a qualitative research (to answer RQ2). Figure 4.1 illustrates the process of this study. Figure 4.1: Research process Section 4.2 describes the design science methods used to build the visualisation tool (Rologram). Section 4.3 describes the recruitment of participants, tasks, procedures, data collection and analysis of the user study. 4.2 Design Science Research Approach The methodology for design science research proposed by Peffers et al. (2006, 2007) was adopted and the process model consisting of problem identification and moti- vation, objectives of a solution, design and development, evaluation, demonstration, 13 4. Methods and communication was followed. The activities in the process are described chrono- logically as follows. Problem identification and motivation. The problem-centred approach was adopted as the research was initiated by the problem identified in Chapter 1 (Pef- fers et al., 2007). The motivation and contribution are also mentioned in Chapter 1. Moreover, preparations for the data inputs were carried out prior to the step of design and development. The datasets of class role stereotypes from the study of Fröding and Ngoc (2020) were reused as the data inputs for the artefact, including K-9 Mail, with 37 commits ranging from February 2011 to February 2020, and Bit- coin Wallet, with 35 commits ranging from March 2011 to January 2020. The static dependencies between classes were extracted from the source code downloaded from GitHub using the Java class dependency analyser (jdeps). Objectives of a solution. The objective was to develop a solution of visualising the evolution of class role stereotypes of object-oriented software systems that in- tends to help developers to comprehend the systems and hence accomplish software evolution tasks. According to Hevner (2007), a design cycle iterates the research activities of build- ing and evaluation of the design artefacts and processes with refinement based on feedback. A design science research must produce a workable artefact (Hevner et al., 2004). The artefact, i.e. the visualisation tool, has gone through three iterations of design cycle. 1. First iteration: Design and development. The functionalities of the artefact for the solu- tion were analysed at the beginning and implemented. Evaluation. The aim of the evaluation was to improve the artefact. Reviews from three experts in software architecture, evolution and class role stereo- types, which were also the authors of a study on role stereotypes, software architecture and visualisation (Ho-Quang et al., 2020), were conducted by sending the link to the artefact through emails. The feedback on the artefact from the experts was collected and examined to see if it was feasible to im- plement in the next iteration. A list of the feedback and the corresponding implementation plan was sent to the reviewers for informing and verification purposes. 2. Second iteration: Design and development. The artefact was refined, and more functional- ities were added and enhanced with considerations of the feedback from the evaluation in the first iteration. Evaluation. The aim and procedure for this evaluation were the same as the first one. Reviews from three experts in software architecture, evolution and class role stereotypes were conducted through email communications, two of which participated in the first evaluation. Their feedback on the artefact was 14 4. Methods collected and examined. One of the experts also provided extra feedback on the artefact during an online meeting. 3. Third iteration: A user study served as both demonstration and final eval- uation was conducted in this iteration, with the purpose of evaluating how the actual use of artefact in the demonstration supports the objectives of the solution. The details of the user study are described in Section 4.3. Design and development. The feedback collected from the previous evalu- ations was taken into account during the development of the artefact. Demonstration. The resulting artefact was used by the participants in the user study, by working on the specific tasks with the context of understanding the evolution of the Bitcoin Wallet system. Evaluation. The user study was conducted in order to investigate how the artefact could help solve the identified problem. Communication. This report serves as the communication of the research, report- ing all the activities and the results of the research process. 4.3 User Study This section describes the user study carried out as an evaluation of the design science research. 4.3.1 Data Collection The participants were expected to complete the given tasks and provide the answers in the questionnaire. During the task completion, the operations and behaviours of the participants were screen recorded, and the video recordings were collected as qualitative data. The recordings only contained the video part without audio. The questionnaire, which can be found in Appendix, contained both qualitative and quantitative questions. Questions related to demographics, equipment, approaches used and feedback on visualisation were asked in the format of qualitative questions, whilst the quantitative questions were in the form of the 5-point Likert rating scale (from strongly disagree to strongly agree). Both qualitative and quantitative ques- tions regarding the tasks, usefulness of role stereotypes and the visualisation were formulated. Multiple-choice questions were not considered for the tasks so as to prevent the participants from randomly picking an answer, which could affect the results. The time taken for completing each task was collected. The questionnaire was provided to the participants through Google Form. Qualitative data was collected through the questionnaire with the following ques- tions: • Elaborate on how role stereotypes help complete the tasks. • Elaborate on how the visualisation helps complete the tasks. • Describe the approach with the visualisation. 15 4. Methods • Describe the approach without the visualisation. • Please comment on the visualisation regarding usability and other aspects you could think of. • Please provide some suggestions to improve the visualisation. Think of any missing features you would like to have. 4.3.2 Tasks The aim was to evaluate the tool with real evolution tasks. In reality, the real evolution tasks might comprise both understanding and development parts. In this user study, with the time restriction of 35 minutes per task, the focus was on the understanding part of the evolution task. It is considered as sufficient for this study owing to the fact that understanding is an utmost important part for developers to perform any development/evolution activities. The data of Bitcoin Wallet and K-9 Mail was used in this study. The Bitcoin Wal- let system was chosen for the tasks since it contained comparatively less amount of classes and packages compared to that of K-9 Mail system, and hence the time needed and complexity of the tasks could be reduced. The participants were given two different evolution tasks of the Bitcoin Wallet soft- ware system. Each participant performed one of the two tasks with help of the visualisation tool (Rologram) and the other task without the visualisation tool (the participant was not restricted in the tools or techniques to be used). Each task was to investigate the evolution of a specific class and package in the Bitcoin Wallet system across five commit versions, in the aspects of the functionali- ties of the class, and class- and package-level dependencies. The two tasks targeted different parts of the Bitcoin Wallet system (i.e. different packages); thus, a low learning effect was expected from performing one task to apply on the other task. Two packages that were of comparable size in terms of the numbers of classes within the package and of the package-level dependencies for the two tasks were chosen. The number of class-level dependencies for the two selected classes was also compa- rable; therefore, the same amount of workload was expected when performing the tasks. The questions for the tasks can be found in the questionnaire in Appendix. The sorts of the questions in each task are as follow. 1. Find two changes of class-level dependencies between the specific package and any of other packages between two commits. 2. Identify the changes of package-level dependencies centred the specific package over the five commits. 3. Identify the functionality/responsibility of the targeted class in the specific commit. 4. Identify the major changes of the targeted class in terms of its functional- ity/responsibility over the five commits. 16 4. Methods The participants were randomly assigned with different combinations of the tasks and tools. The maximum time for completing each task was restricted to 35 minutes and the time taken for each task was recorded. After data collection from the survey, the answers for the task questions were marked and given points according the definite solution found by examination of the software system using Eclipse. 4.3.3 Participants The sample size was 6. The non-probability sampling method convenience sampling was adopted to select the participants for user study as they were searched and recruited through the personal network for higher accessibility and response rate. The visualisation only supports Java projects at the moment as the software systems selected in the input datasets are written in Java; accordingly, the target population of the user study was developers with Java programming experience. The partici- pants must meet the criterion of having some Java programming experience. Actual working experience in the industry was preferred but not necessary. The exclusive criterion was the participants were not contributors of the Bitcoin Wallet system, which was used in the user study, so as to avoid the situation that they could answer the questions in the tasks directly, which might affect the results. 4.3.4 Procedure The activities before and during a user study are illustrated in Figure 4.2. Before a user study, participants were contacted and details of the user study along with the link to the trial version of the visualisation with the data of K-9 Mail software system, which was not used in the tasks to avoid the participants got familiarised with the Bitcoin Wallet system before the user study, and introduction of role stereo- types were sent once they agreed to participate in user study. Each participant was scheduled with a time to conduct a user study. 17 4. Methods Figure 4.2: Procedure of conducting the user study with one participant The user study was conducted remotely with video conferencing software with one participant each time, and the duration was restricted to a maximum of one and a half hours. The activities in a user study in chronological order were as follows. The participant was requested to share the screen with the facilitator and given time to get familiarised with the trial version of the visualisation (4 minutes). The facilitator then sent the link to the survey to the participant and the participant can start to provide some background information in the first section of the survey (1 minute). Communication with the participant during the whole user study was maintained for clarifying the questions. After completing the demographic questions, the par- ticipant can start working on the two tasks and the screen for the whole process of the two tasks was recorded. One participant refused to share his screen and thereby no corresponding video recording was collected. The link to the visualisation tool with data of the Bitcoin Wallet system was sent when the participant was ready to work on the task with visualisation. The facilitator started the timer and the participant worked on the first task, with 18 4. Methods a time restriction of 35 minutes. The participant may complete a task with less than 35 minutes. After 35 minutes, the participant was requested to proceed to the second task. The time taken for the first task was recorded. The procedure for the second task was handled in the same way. No break was given in between the two tasks unless the participant requested. The video recording of the participant working on the tasks and the two time for the duration of working on the first and second tasks were collected. The participant had to answer the after-task questions in the survey (15 minutes) after working on the second task. After the participant submitted the survey, the demographic, qualitative and quantitative data in the questionnaire was gathered. 4.3.5 Data Analysis The steps for qualitative data analysis in the book written by Auerbach and Silver- stein (2003) were followed (p. 43). The inputs and outputs produced for each step are illustrated by Figure 4.3. Figure 4.3: Inputs and outputs for the steps in data analysis Firstly, the research concern was about how the visualisation tool helped developers perform software evolution tasks and the theoretical framework was discussed in Chapter 2. Secondly, the relevant text was selected from the answers collected in the survey. Thirdly, the related relevant text was grouped together to derive the re- peating ideas (codes) using the coding methods Initial Coding and In Vivo Coding, and the behaviours of research participants observed from the video recordings were extracted using the coding method Process Coding as described in the book from Saldaña (2013, pp. 264-266). The three coding methods that were used are described with example codes as fol- lows. 19 4. Methods • In Vivo Coding. The exact phrase from the participants’ answers was ex- tracted into a code. For instance, the code “Easy to navigate a big project and dive deeper” was derived from “The visualisation is good and it is easy to navigate a big project and dive deeper.”, as written by a respondent. • Initial Coding. The qualitative data in the survey was examined and broken down into discrete parts. For example, the sentence “I like the zoom-in and out feature and also how the connections between different components are vi- sualized” was broken into two parts and two codes were extracted from it. • Process Coding. The participant actions and consequences were searched from the video recordings. The code “Viewing commit history” is an example by capturing a participant action. Afterwards, themes were produced by grouping the codes into coherent categories. By grouping and conceptualising the related themes, theoretical constructs were de- veloped. Finally, a theoretical narrative was created to convey the experience of the research participants with the tasks and the visualisation. 20 5 Visualisation Tool The chapter introduces the characteristics and features of the visualisation tool, along with the interaction components. 5.1 Role Stereotype Encoding Each circle in the graph represents a class in the software system and the colour of the circle represents its role stereotype. The six role stereotypes are encoded by colours as shown in the following table (Table 5.1). A legend for the role stereotypes is placed at the bottom of the visualisation, in which each role stereotype can be hovered over to highlight the corresponding classes, and thus users having problems distinguishing colours are also able to identify the role stereotype of the classes. Role stereotype Colour Controller Purple Coordinator Green Information Holder Red Interfacer Yellow Service Provider Blue Structurer Pink Table 5.1: Role stereotype and colour conversion 5.2 Representations of Class, Package, System and Dependency The concept of a software system is illustrated by a node-link diagram with grey- bordered rectangles representing the packages, and circle nodes representing the classes (Figure 5.1). An arrow (link/edge) illustrates a dependency from a class to the pointing class. For example in Figure 5.2, the class WalletActivity depends on the class Application. In the context of responsibility-driven design, a class node with body colour indicating the role stereotype represents an object with a specific role, an arrow showing the dependency between two class objects represents the collaboration of the classes to fulfil a responsibility. Chances are high that isolated class nodes without arrows contain static variables and functions, in which no dependencies are created. 21 5. Visualisation Tool Figure 5.1: Representations of class, package and system in the node-link diagram Figure 5.2: Representation for dependencies 5.3 Class Metrics Two class metrics are incorporated into the visualisation. Role stereotype. The six role stereotypes are encoded by colours (See Table 5.1). Lines of code. The size of class nodes is used to encode the number of lines of code for each class, so as to give a rough idea of the differences of classes in terms of lines of code. Figure 5.3 shows the visualisation of the metric lines of code on the class nodes where it is visible that the class WalletActivity has the largest number of lines of code while the class Constants has the smallest number. Figure 5.3: Visualising lines of code metric 22 5. Visualisation Tool 5.4 Abstraction Levels A software system is represented by three abstraction levels. System level. The visualisation shows data of the latest version at the system level by default. All packages and sub packages are displayed in the rendered graph. The sub packages within packages are placed inside the corresponding packages to show the hierarchies. All the classes inside the packages and the class-level dependencies are also shown. Figure 5.1 is an example showing the system level. The system level is accessed by clicking on the grey background outside the parent packages at the package or class level. Package level. By clicking on a rectangle representing a package, the package level is entered. All the sub packages within the selected package, which is highlighted by the thick black borders, and the classes within both the selected package and its sub packages are shown, as well as the classes in other packages which have dependencies with the classes in the selected package. Take the selected package wallet.util as an example (Figure 5.4), all the classes belonging to it are displayed, as well as the class AbstractWalletActivity in package wallet which depends on the class ActionBarFragment in the selected package. Figure 5.4: Graph at the package level Package-level dependencies are noticeable at this level. The class-level dependencies across packages reveal how packages utilise each other. For instance, Figure 5.5 shows the package-level dependencies between the selected package wallet.util and the packages wallet, wallet.camera and wallet.ui. 23 5. Visualisation Tool Figure 5.5: Package-level dependencies Class level. By clicking on any circle class node at the system or package level, the class-level visualisation is displayed. The dependencies between the selected class and classes in all packages are shown. For example, Figure 5.6 shows the selected class ActionBarFragment highlighted by a black border at the class level with the class AbstractWalletActivity in the package wallet depends on it. Figure 5.6: Graph at the class level 5.5 Class Dependencies The inclusion of dependency types and levels is intended to demonstrate how the classes are related to each other within a software system and how a class is utilised by other classes. The latent connections between the classes may be discovered. The options of dependency type and level are only available at the class level. Three op- tions are provided for the dependency type: “All”, “In” and “Out”. The supported options for the dependency level are 1, 2 and 3. The dependency level is set to 1 by default and thereby the graph only shows the directly connected classes to the selected class. Figure 5.7 shows both incoming and outgoing dependencies for the selected class PeerListViewModel within the first dependency level. 24 5. Visualisation Tool Figure 5.7: Visualisation with the default values for dependency type and level “All” is set by default for the dependency type, and thereby both incoming and outgoing dependencies are displayed. An incoming dependency implies the con- necting class depends on the pointing class while an outgoing dependency has the opposite meaning. Figure 5.7 above shows the selected class PeerListViewModel has an incoming dependency from PeerListFragment and outgoing dependencies to WalletApplication and BlockchainServiceLiveData. When the dependency level is set to 2 or 3, the incomers to the selected class and the incomers of incomers, and so on, are shown for the option “In”. Similarly, outgoers for the option of “Out”. The dependencies at different levels are represented by the width of the arrows. The thickest arrows illustrate the first-level dependencies, while the medium-sized and thinnest arrows represent the dependencies at the second and third level respectively. Figure 5.8 demonstrates the dependencies for the class PeerListViewModel with only outgoing dependencies within three levels. To simplify the results and reduce the number of dependencies, some classes are filtered. As mentioned previously, the class has outgoing dependencies with WalletApplication and BlockchainService- LiveData within the first level, and those first-level dependencies are shown with the thickest arrows. As can be seen in the graph, the arrows representing the depen- dencies from WalletApplication to Event, BlockchainState and Configuration are of medium-sized. This is an indication of second-level dependencies as they are related to the class WalletApplication, which is within the first dependency level of the selected class PeerListViewModel. The thinnest arrow from Configuration to ExchangeRate shows the third-level dependency. 25 5. Visualisation Tool Figure 5.8: Visualisation showing the outgoing dependencies with three levels 5.6 Timeline Constitution A timeline is created in the visualisation to show the changes of the class role stereotype at the class level, and the changes of the dominant role stereotypes at the system and package levels. Figure 5.9 shows the timeline with different shapes at the three levels. Figure 5.9: Timeline constitution at the system level (top), package level (middle) and class level (bottom) Each commit version of a software system is illustrated by a coloured shape at dif- ferent levels. The timeline is put together by a series of shapes with one for each version. At the system and package levels, the colour of the square implies the domi- nant role stereotypes, in other words, the role stereotype with the highest number of occurrences, in the system or in the selected package respectively. Multiple colours in the square indicate there are more than one role stereotypes having the highest number of occurrences. Grey colour indicates the inexistence of the selected package 26 5. Visualisation Tool in the corresponding version. For example, the graph in Figure 5.10 shows the role stereotypes Information Holder (red) and Interfacer (yellow) are of equally largest number of occurrences in the selected package wallet.ui.scan; thus, the shape on the timeline representing the current version has the corresponding two colours (red and yellow). Moreover, the figure shows the inexistence of the package during the first 28 versions. Figure 5.10: Visualisation at the package level At the class level, the timeline is formed by the coloured circles representing the class with certain role stereotypes in different versions. The grey colour depicts the absence of existence of the selected class in the corresponding version. Figure 5.11 shows an example of the class BlockchainService. The first three grey circles on the timeline indicate that the class does not exist in the first three versions. On 2012- 04-02, it is added and has the role stereotype Interfacer, then becomes Information Holder for 23 versions. After that, it becomes Interfacer again for 8 versions. In this way, the timeline serves as an indication of how and when the changes of role stereotype of the selected class occur. 27 5. Visualisation Tool Figure 5.11: Visualisation at the class level The current viewing version is annotated on the timeline, and it can be switched by selecting a shape representing another version on the timeline. Moreover, by switching to the comparison mode using the toggle button on the left, the comparing version can be chosen on the timeline. Figure 5.12 shows the toggle button with the comparison mode and the annotations of the current and comparing versions on the timeline. Figure 5.12: Timeline with current and comparing versions annotated 5.7 Pairwise Comparison The visualisation enables pairwise comparison of two selected versions at any of the three levels: system, package and class. In order to do the pairwise comparison, one must select a version to compare. There are two ways to select a comparing version. First, the dropdown select box in the compare dialog can be used. Second, the toggle button with comparison mode next to the timeline can be used to select a version for comparison within the timeline. After selecting the version to compare, the changes between the current viewing ver- sion and the selected comparing version are displayed immediately. The differences between them are displayed as changes from the earlier version to the later version, no matter if one is comparing the current version to an earlier or a later version. Figure 5.13 illustrates the changes between two versions at the system level. The 28 5. Visualisation Tool changes of class are grouped into four types, i.e. added, unchanged, removed and role-changed, while the changes of dependency are grouped into added, unchanged and removed. The word unchanged means the role stereotype of the class remains the same, and the class or dependency exists in both versions, and this is represented by a circle or an arrow with faded colour. A role-changed class, which is illustrated by a coloured circle with a border of a different colour, implies that the role stereotype of the class in the current version (body colour) is different from that in the comparing version (border colour). An added class or dependency is denoted by a circle or an arrow with solid colour, so as to distinguish with unchanged classes and dependencies. The grey colour on the circle indicates a class is removed in the current version while a dashed arrow line represents a removed dependency. A legend of the notations used for the changes is placed inside the compare dialog. Figure 5.13: Pairwise comparison at the system level In the example shown in Figure 5.13, the versions on 2011-03-26 and 2011-07-02 are compared. The class FilesBlockStore in the package wallet is removed while the class Service is added in the version on 2011-07-02. Dependencies from the class WalletActivity to the class Service and from Service to Application are added. The class ActionBar having the colour green in the body and pink in the border indicates that the role stereotype is changed from Structurer to Coordinator, as the current version is 2011-07-02. The classes Constants, PreferencesActivity, WalletActivity and Application, and the dependency from WalletActivity to Application are unchanged between the two versions. The representations of different types of changes are the same for both the package and class levels. However, one has to be aware that the meaning of added and re- moved classes at the package and class levels is different from that at the system 29 5. Visualisation Tool level. They simply imply the existence or nonexistence of classes in the system at the system level. On the other hand, the addition or removal of classes in the comparison at the package and class levels means the related classes have added or removed dependency from or to the selected package or class. In other words, the meaning of added or removed classes at these two levels can be interpreted as linked or unlinked classes respectively, or added or removed collaborating classes in terms of collaborations. A removed collaborating class may or may not be removed from the system. However, chances are high that a removed class at the class or package level is removed from the system. As can be seen in Figure 5.14, the dependency from the class AbstractWallet- Activity to ActionBarFragment is removed in the version on 2012-04-02. The class AbstractWalletActivity is considered as a removed class in connection to the selected package wallet.util. The visualisation does not keep track of relocation of classes to another package; however, this may be noticed by addition and removal of classes with the same name. For example, AbstractWalletActivity is removed from wallet and added in the package wallet.ui. Likewise, rename of a class may be recognised in the same way with similar names. Figure 5.14: Pairwise comparison at the package level The visualisation also provides code comparison of the selected class between two versions with highlighted differences, with the source code for the earlier version on the left. An example for the class BlockListViewModel can be seen in Figure 5.15. 30 5. Visualisation Tool Figure 5.15: Comparison of source code in two versions Combining the information of changes of role stereotype on the timeline and the changes of collaborating classes shown in the diagram, one may trace where the responsibilities of the selected role-changing class shift to, as a change of role stereo- type probably indicate the responsibilities of the class is redistributed to its col- laborating classes. For example, Figure 5.16 shows the added collaborating class BlockchainServiceImpl and removed dependencies from WalletActivity and to WalletApplication for the selected class BlockchainService when its role stereo- type changed from Interfacer to Information Holder, which is probably an indication of the responsibilities of BlockchainService are transferred to its new collaborating class BlockchainServiceImpl (Interfacer). Figure 5.16: Combination of changes of role stereotype and collaborating classes at the class level 31 5. Visualisation Tool 5.8 Consecutive Comparison A set of slider control and previous and next buttons is included for showing the changes between two consecutive commit versions each time. By selecting a version with the slider control, the graph displays the changes between the selected version and one version before it. Alternatively, the previous and next buttons can be used to select a version. Continuously clicking on the next button allows users to view the changes chronologically for the entire range of versions. The representations for added, removed and unchanged classes and dependencies as well as the role-changed classes in this view are the same as those in the comparison of two versions. Figure 5.17 shows the changes centred the package wallet.util between the selected ver- sion on 2012-04-02 and the previous version, for example, the dependency from the class AbstractWalletActivity to ActionBarFragment is removed and the classes IOUtils and Iso8601Format are added. Figure 5.17: Slider and changes between two versions 5.9 Pattern Finder The visualisation enables searching for specific collaboration patterns within the system. A collaboration pattern is the collaborations between the objects within different dependency levels. The pattern finder provides three ways to look for a specific collaboration pattern: user-defined pattern, ranking lists and common pat- terns. It supports two levels of collaboration patterns, in which at most three classes are involved in a case matching the pattern. The feature of user-defined pattern is created for users who are knowledgeable on role stereotypes and know what they are looking for. When only the role stereotypes for the first class are selected, the classes with the selected role stereotypes are displayed and their total counts are shown in the chart, which illustrates the changes of the occurrences over the versions. For example, Figure 5.18 shows the role stereotypes Controller and Coordinator selected for the first class; thus, the graph displays only the classes with the selected role stereotypes. The line chart shows the total 32 5. Visualisation Tool numbers of the selected role stereotypes over the versions, with 12 counts in the current version. Figure 5.18: Results of the user-defined pattern for the first class For user-defined patterns with one or two levels, the visualisation displays the result of cases matching the selected role stereotypes at the levels. For example, Figure 5.19 shows the results of the defined pattern Interfacer to Structurer to Information Holder. Three results matching the collaboration pattern are found. One of them is the class ExchangeRatesFragment (Interfacer) depending on ExchangeRatesAdapter (Structurer), of which depends on BlockchainState (Information Holder). 33 5. Visualisation Tool Figure 5.19: Results of the user-defined pattern for two levels The visualisation also identifies all possible collaboration patterns within two levels and adds up the number of occurrences to generate a ranking of all the patterns existing in the current version. After selecting a pattern in the list, the diagram shows all the cases matching the selected collaboration pattern, and its changes of the number of occurrences are shown in the line chart. For example, Figure 5.20 shows parts of the ranking list on the right and the results of the selected pattern, i.e. from Interfacer to Information Holder, on the left. Figure 5.20: Results of the selected pattern in pattern ranking list 34 5. Visualisation Tool In addition to the user-defined pattern and ranking list of all available patterns, some common patterns are suggested for users who are unfamiliar with role stereotypes and have no ideas what patterns to look for. Figure 5.21 shows the results of a common pattern (Coordinator to Interfacer to Service Provider). Figure 5.21: Results of the selected common pattern 5.10 Role Stereotype Changed Indication The visualisation provides a feature to highlight all the classes that have changed role stereotypes over all versions. Figure 5.22 shows the highlighted class WalletUtils in the graph, indicating that the role stereotype of the class has changed over the versions, while that of other classes without a dashed border have never changed. Figure 5.22: Highlighting classes which have changed in role stereotype 35 5. Visualisation Tool 5.11 Miscellaneous The visualisation also provides the following features. History. When an element in the diagram is selected, a record holding the current version and the selected element is stored in the history list. The viewed elements (system, package or class) are listed, and the list can be exported to a text file. It is enabled to go back to previously viewed elements by clicking an item in the history list. It is also possible to clear the history list. Graph export. The rendered graph can be downloaded in the format of Portable Network Graphic (PNG). Source code. The source code for classes is displayed within the visualisation for better comprehension between the visualisation and the actual code without having to look for it. The syntax is highlighted with the styles used in Eclipse, a widely used Java integrated development environment (IDE). 5.12 Interaction Components The visualisation design has adopted all of the seven categories of interaction tech- niques proposed by Yi et al. (2007). Select. Users can select a specific class or package by clicking a node or a rectangle respectively. The node or rectangle is highlighted with a black border around it; therefore, users can keep track of the class or package easily after applying different features. Explore. Users can explore the data at the system level by default, or subset of data by clicking a node or rectangle to view the data at class or package level respectively. Reconfigure. The visualisation enables users to choose between different types of layout, currently compact and hierarchy layouts are available. Encode. The class nodes are by default of the same size, showing the role stereotype only. Moreover, alternative representation is provided by altering the size based on the encoded number of lines of code for the classes. Abstract/Elaborate. When the class names are set to not shown, users can hover over a class node to view its name. Moreover, zooming and dragging are possi- ble for all elements in the graph. Users can alter the scale of the graph to look at the detailed view or the overview of the dataset by zoom-in or zoom-out respectively. Filter. Users can alter the graph by filtering different role stereotypes. Figure 5.23 shows the graph rendered after filtering out the classes with the role stereotypes 36 5. Visualisation Tool Information Holder and Interfacer. The filtered classes are in solid colours while those filtered out are in faded colours. Figure 5.23: Visualisation with Information Holder and Interfacer filtered out Connect. The directly connected classes and their neighbours within all depen- dency levels are highlighted when hovering over a specific class node. When hovering over the circle representing a role stereotype in the legend, the classes having the corresponding role stereotype are highlighted. Moreover, the graph highlights the package and all collaborating classes when hovering over a package. 5.13 Implementation The visualisation tool is implemented as a web application, using the HTML tem- plating engine Pug, SCSS and JavaScript for front-end development. Express and Node.js are adopted as the web framework and the runtime environment respec- tively. The data for the software system to be visualised in the artefact is stored in the format of comma-separated values file (CSV), with an alternative option of connection to the graph database Neo4j. The graph visualisation is implemented with the use of the JavaScript graph theory library Cytoscape.js and the Klay lay- out algorithm for Cytoscape.js. CodeMirror is used to show the source code and highlight code changes. The tool currently requires data inputs of role stereotypes and dependencies. The visualisation tool can basically be applied to any object-oriented software systems, provided that it is integrated with the role stereotype classifying tool (Nurwidyan- toro et al., 2019; Ho-Quang et al., n.d.) and modified to achieve automated extrac- tion of dependencies. 37 5. Visualisation Tool 38 6 Results This chapter presents the results of implementing the methodology described in Chapter 4. 6.1 Design Science Research The final artefact produced, i.e. the visualisation tool named Rologram, of which the name is derived from “role evolution diagram”, using the design science research approach is introduced in Chapter 5. The implementation of the visualisation has gone through three iterations of design cycle. The results from the three iterations of development and evaluation are described as follow. 6.1.1 First Iteration During the design and development stage of the first iteration, the component con- secutive comparison was implemented. Figure 6.1 shows the screenshot of the visu- alisation built. The visualisation provided simple features of showing the changes between two consecutive versions by using the slider on top to select a version and a legend for mapping the colours and role stereotypes. Figure 6.1: Screenshot of the visualisation for the evaluation in the first iteration 39 6. Results After the design and development in the first iteration, evaluation on the visuali- sation tool was conducted with three experts in software evolution and class role stereotypes. They pointed out the following issues with the visualisation: • Navigation between the versions • Lack of legend of the representations for changes, in particular the circles with a different border colour • Popup of class name • No highlight of selected class and package Moreover, they suggested extra features including: • Viewing of source code for the classes • Addition of class metrics, for example, lines of code • Filtering of role stereotypes • User guide for the visualisation 6.1.2 Second Iteration After the first iteration of the design cycle, the visualisation was refined based on the feedback from the first evaluation, and all feedback was considered and incor- porated into the visualisation, except the user guide was not fully completed. As a result of the design and development activity, the functionalities of pairwise com- parison, pattern finder, display of code, history and options for changing the layout and metric were implemented. Figure 6.2 shows a screenshot of the artefact, with the data of K-9 Mail system. Figure 6.2: Screenshot of the visualisation for the evaluation in the second iteration 40 6. Results Two experts from the first evaluation and one other participated in the evaluation of the built artefact during the second iteration with the following comments (Table 6.1). Acknowledgment Suggestions 1. nice representation of the lines of code metric for the classes 2. nice pattern builder (the user- defined pattern) 3. chart showing the evolution of the number of occurrences of the de- fined pattern 4. fast and smooth visualisation 1. missing legend 2. selected layout should be pre- served 3. shape used on the timeline for the system and package levels is the same as that for the class level which causes confusion 4. filtered classes can be faded and remained in the graph 5. unable to know all available func- tionalities 6. some circles are overly large when visualising the lines of code met- ric 7. want dependencies to other pack- ages at the package level Table 6.1: Comments from the second evaluation 6.1.3 Third Iteration With consideration of all the feedback from the evaluations in the first two itera- tions, the final version of the artefact is created. Figure 6.3 shows the screenshot of the artefact for the evaluation in the third iteration, with the data of Bitcoin Wallet system. 41 6. Results Figure 6.3: Screenshot of the visualisation for the final evaluation The user study was conducted as the final evaluation of the design science research and the feedback collected from the participants are shown in Table 6.2. 42 6. Results Acknowledgment Suggestions 1. easy to go back with history func- tion 2. easy to navigate a big project 3. easy to use 4. visualising the content of differ- ent commits is fast 5. display of the code for selected class is fast 6. efficient tool 7. friendly user interface 8. zoom-in and out feature 9. visualisation of the connections between different components 10. help finding dependencies across packages 11. useful representation of metric lines of code 12. highlighted newly added code 1. accidental double click causes misbehaviour 2. too many lines prevent from click- ing the package/class 3. often jump to other pages or re- fresh unwantedly 4. removed dependency should be in bold colour 5. history does not keep track of the comparing version 6. lack of animation to display the changes 7. user guide should pop up after initialisation 8. missing definition of role stereo- types 9. dependencies should not be preloaded when entering the page/include an option to change the visibility of dependencies 10. unable to hide some versions 11. use of pink colour causes confu- sion with the dimmed red colour 12. unclear meaning of icons on tog- gle button 13. unable to filter the versions 14. indicators such as year on the timeline Table 6.2: Comments on the visualisation from the final evaluation 6.2 User Study 6.2.1 Demography The participants for the user study were in various occupations, including product owner, postdoctoral researchers in software engineering, back-end developer, soft- ware engineer and IT placement. All respondents have never used nor contributed to the software system Bitcoin Wallet involved in the tasks, although one of them has studied the source code. The demographic characteristics of the participants are summarised in the charts below. The pie charts in Figure 6.4 show the distribution of gender and age of the participants. 43 6. Results Figure 6.4: Distribution of gender (left) and age (right) Figure 6.5 shows the distribution of the Java experience, programming experience and work experience in the Information Technology (IT) industry of the participants. The range of participants covered varying experience in terms of programming and work experience, and yet no participants had Java programming experience for longer than five years. Figure 6.5: Distribution of Java (left), programming (middle) and work experience (right) The participants were asked whether they have experience in the following areas: software development, software maintenance, software testing, refactoring, debug- ging, Android development and iOS development. Figure 6.6 shows the numbers of participants having experience in each of the mentioned areas. As can be seen from the figure, the majority of the respondents had experience in software development, followed by software maintenance and Android development. 44 6. Results Figure 6.6: Areas the participants had experience in 6.2.2 Task Performance The combinations and ordering of the tasks and tools for the participants are listed in the following table (Table 6.3). A and B refer to the tasks related to the packages wallet.service and wallet.ui.send respectively. Participant Task 1 Tool for Task 1 Task 2 Tool for Task 2 1 A with visualisation B without visualisation 2 A without visualisation B with visualisation 3 B with visualisation A without visualisation 4 B without visualisation A with visualisation 5 A with visualisation B without visualisation 6 B with visualisation A without visualisation Table 6.3: Combinations and orderings for the tasks and tools The task performance of the respondents was measured by the time taken and the points judging from the answers. Table 6.4 lists the total points the participants received in task with the usage of visualisation in comparison to that without, whilst the points for the four task questions are shown in Table 6.6 to 6.9. Table 6.5 shows the time taken for working on the tasks with and without the visualisation tool. As can be seen in Table 6.4, the participants tended to receive higher points in task with the use of the visualisation compared to that without. Four participants stopped working on the task without the use of the visualisation after 35 minutes, which was the maximum time allowed for working on a task (See Table 6.5). 45 6. Results Participant With Without 1 19/20 5/20 2 20/20 7/20 3 18/20 5/20 4 14/20 10/20 5 20/20 7/20 6 10/20 2/20 Table 6.4: Total points for tasks with and without visualisation Participant With Without 1 34:43.82 31:18.73 2 31:59.31 35:01.71 3 20:40.31 35:01.21 4 24:33.09 35:01.07 5 34:48.09 35:01.24 6 29:06.23 30:43.96 Table 6.5: Time taken for tasks with and without visualisation (MM:SS.ss) As can be seen from Table 6.6 to 6.9, some participants did not manage to answer all the questions in the task, and this is marked in the table with “No answers”. Overall, the participants tended to complete all the questions in task with the use of visualisation within the given time, whereas some did not manage to answer some of the questions in task without the use of visualisation. Question 2 regarding changes of package-level dependencies had the greatest number of empty answers, followed by question 4 and question 1. Participant With Without 1 5/5 5/5 2 5/5 No answers 3 5/5 No answers 4 5/5 5/5 5 5/5 0/5 6 0/5 0/5 Table 6.6: Points for question 1 regarding changes of class-level dependencies with and without visualisation 46 6. Results Participant With Without 1 4/5 No answers 2 5/5 No answers 3 3/5 No answers 4 4/5 No answers 5 5/5 No answers 6 0/5 0/5 Table 6.7: Points for question 2 regarding changes of package-level dependencies with and without visualisation Participant With Without 1 5/5 No answers 2 5/5 5/5 3 5/5 5/5 4 5/5 5/5 5 5/5 5/5 6 5/5 0/5 Table 6.8: Points for question 3 regarding the responsibility of a class with and without visualisation Participant With Without 1 5/5 No answers 2 5/5 2/5 3 5/5 No answers 4 0/5 No answers 5 5/5 2/5 6 5/5 2/5 Table 6.9: Points for question 4 regarding the changes of responsibility of a class with and without visualisation 6.2.3 Other Quantitative Data The participants were asked regarding their attitude towards the visualisation after working on the two tasks, on a scale from 1 to 5 (See Figure 6.7). 47 6. Results Figure 6.7: Attitude of the participants towards the visualisation 6.2.4 Theoretical Constructs, Themes and Codes The theoretical constructs (capitalised), themes (italics) and codes extracted from the qualitative data are listed below, with codes extracted by Process Coding from the screen recordings in the format of gerund and those by In Vivo Coding in quotes. The number of occurrences of the corresponding code in survey answers and resulted from Process Coding (observed behaviours and consequences) are included in the parentheses and square brackets respectively. I. Approach with the visualisation A. Accessing abstraction levels 1. Viewing the system [26] 2. Viewing a package [74] 3. Viewing a class [39] B. Selection of element in the diagram 1. Clicking on background to access to system level [10] 2. Clicking a package (2) [32] 3. Clicking a class (1) [24] 4. Hovering over a package [30] 5. Hovering over a class [77] 6. Dragging a package [4] C. Operations on timeline 1. Hovering over a version on timeline [657] 2. Selecting the current version on timeline (2) [68] D. Ways to compare two versions with the visualisation 1. Clicking toggle button (2) [101] 2. Using slider (1) [2] 3. Using compare dialog (2) [27] E. Selection of comparing version 48 6. Results 1. Selecting a comparing version in compare dialog (2) [10] 2. Selecting a comparing version on timeline [83] 3. Selecting a version with slider [7] 4. Using the previous button to select a version on slider [10] 5. Using the next button to select a version on slider [34] F. Change identification with the visualisation 1. Viewing changes of system [1] 2. Viewing changes of package [74] 3. Viewing changes of class [18] 4. Viewing added class list [1] 5. Viewing removed class list [1] 6. Reading change legend (1) [6] G. Dependency identification with the visualisation 1. Identify package-level dependency with arrows (3) 2. Identify added/removed dependencies between classes (1) 3. Identify class-level dependencies (1) H. Role stereotype utilisation 1. Identify role stereotype with colour (3) 2. Spot the change in role stereotype with the colours (2) 3. Identify significant change of role stereotype with timeline (3) I. Codes in the visualisation 1. Reading code in visualisation (3) [10] 2. Viewing code comparison in visualisation [8] 3. Highlighting code in visualisation [4] 4. “Read highlighted codes to observe the change and new functionality” (1) 5. Searching a keyword in source code [1] J. Supporting features 1. Going back to previously viewed element using history feature [5] 2. Hovering over role stereotype legend [1] 3. Downloading the graph as image [1] 4. Zooming in (1) [26] 5. Zooming out (1) [19] K. Unexpected behaviours 1. Double clicking on a package [7] 2. Double clicking on a class [2] 3. Double clicking on one version on timeline [1] 4. Clicking on an edge [21] 5. Opening the code dialog at package level [2] II. Approach without the visualisation A. Code retrieval 1. Cloning code repository using GitHub Desktop (1) [2] 2. Downloading code from GitHub website (1) [2] 3. Opening code repository on GitHub website (3) [33] B. Utilisation of tools 49 6. Results 1. Using Eclipse (1) [1] 2. Using Visual Studio Code (1) [1] 3. Using Sublime Text [1] 4. Using Sourcetree [1] 5. Using command line [1] 6. Using GitHub Desktop [2] 7. Using GitHub website (2) [5] C. Navigation 1. Clicking an untargeted package folder [20] 2. Clicking a targeted package folder (2) [106] 3. Hovering over an untargeted package folder [47] 4. Hovering over an untargeted class [25] 5. Clicking a class to open code (3) [129] 6. Going to previous page [23] 7. Navigating to another class from code [5] 8. Switching tabs (1) [552] D. Code viewing 1. Reading code (6) [65] 2. Reading import declaration [24] 3. Highlighting code [28] E. Searching 1. Searching a keyword in webpage [32] 2. File searching [2] 3. Searching by commit hash [1] 4. Searching by commit message [1] F. Utilisation of version control 1. Viewing commit history [17] 2. Reading commit message [11] 3. Resetting working branch to a commit [4] 4. Viewing file changes of a commit [8] G. Comparison without the visualisation 1. Comparing the classes in the same package of between versions (1) [6] 2. Viewing code comparison with diff tool for a class [22] 3. Comparing the same class of different commits (1) [10] H. Dependency change identification without the visualisation 1. Read code to identify changes of dependencies (2) 2. Jot down package names (1) 3. Search package name (2) I. Difficulty without the visualisation 1. Difficult to compare code of different commits on GitHub (1) 2. Difficult to complete task with limited time (1) 3. Troublesome to observe changes in code on GitHub (1) III. Benefits of the visualisation A. Ease of use 1. “Easy to navigate a big project and dive deeper” (1) 50 6. Results 2. “History function makes it easy to go back” (1) 3. Easy to compare two versions (3) 4. Easy to see the dependencies (1) 5. Easy to locate a package (1) 6. Easy to view a package (1) B. Merits over other tools 1. Capability of identifying package-level dependency is not provided in existing tools (1) 2. “It makes dependency more obvious/visible compared to other tools” (1) C. Supporting change identification 1. Help check the changes since last commit (1) 2. Help spot changes in role stereotype (3) 3. Help check changes in dependency (2) 4. Capability to show addition and removal of classes (1) 5. Easy to view the changes in code (1) 6. Able to observe the change of number of lines of code (1) D. Supporting recognition in interrelation 1. See how a class is utilised (1) 2. Help understand class relationships (2) 3. Provide overview of system structure (4) E. Benefits of role stereotype utilisation 1. Indication of changing aspects with change of class role stereotype (1) 2. “Colour change in timeline shows the change of role of certain class easily” (4) 3. Help understand the role of classes without code inspection (2) 4. Help expect dependencies (1) F. Satisfaction with representations 1. “Dimmed colour and dotted arrow” (1) 2. Use of graph (1) 3. See dependency by connections of dots (1) 4. “I like how the connections between different components are visualized” (1) 5. Useful metric lines of code (1) 6. Able to observe importance of classes (1) G. Quality attributes 1. “Visualizing the content of the different commits and corresponding code was fast” (1) 2. “The user interface is friendly” (1) 3. “I like the zoom-in and out feature” (1) 4. Provide the code for classes (1) 5. “The tool was very usable and easy to use” (3) 6. “Highlighted newly added code. Very good” (2) IV. Improvement on the visualisation A. Difficulty of finding a version 51 6. Results 1. No year indicator on timeline (1) 2. Have to overlay pointer on the shapes on timeline (1) B. Problems on representation 1. Use of pink colour causes confusion (1) 2. “Too many lines prevent me from clicking the package/class” (1) 3. Removed dependency should be in bold colour (1) C. Ambiguous components 1. Vague meaning of icons on toggle button (2) 2. “I missed the role change visualisation part” (2) D. Usability problems 1. Unhandled double click issue (2) 2. “Often jump to other pages/refresh” (1) 3. User guide not shown at the beginning (1) E. Incapability of the visualisation 1. Absence of animation on changes (2) 2. No filtering on versions by year (1) 3. Incapability to remove/filter versions (2) 4. Incapability to open any code repositories (1) 5. No option to change the visibility of dependencies (1) 6. Missing role definition (1) 7. Keep track of the comparing versions in history (1) 8. Incapability to see the selected element with the overall structure (1) 6.2.5 Theoretical Narrative This section presents the theoretical narrative created using the theoretical con- structs, themes and codes listed in Section 6.2.4. The attitude and experience of the research participants are described with the theoretical narrative below, regarding the research concern of how the visualisation tool helped them perform software evolution tasks. The theoretical narrative is separated into four subsections with regard to the four theoretical constructs. The theoretical constructs in capitalised letters and the themes in italics are included in parentheses. 6.2.5.1 Approach with the Visualisation With the described approaches by the participants in the survey and the observed behaviours from the video recordings, the participants’ approaches with the use of the visualisation can be seen (APPROACH WITH THE VISUALISA- TION). The participants had performed various operations of selection in the dia- gram (Selection of element in the diagram), in particular clicking the target package or class, for example, one participant stated that “I clicked on the package of interest to identify possible changes in dependencies and role”. It was notable that none of the participants attempted to click an untargeted package. Moreover, they also clicked on the background of the diagram to access to the system level. Some of the participants clicked on the background of the diagram accidentally, especially when a dialog was opened. Selection of different elements allowed them to access to various abstraction levels (Accessing abstraction levels). One participant 52 6. Results clicked the already selected package multiple times. By clicking a class, package or the background, the view of the class, the package or the system was displayed respectively. During the tasks, the participants accessed to the package level the most frequently, followed by the class and system levels. Most participants had used the mouse to hover over a package or a class node to highlight the selected element and its related elements, as well as dragged a package (Selection of element in the diagram). Among all the selecting operations, the most frequently occurring was hovering over a class, followed by clicking a package. The timeline was the only possible way to change the current viewing version (Operations on timeline). Among all the participants’ actions, hovering over a version on the timeline was shown as a recurring action, which happened 657 times, with 131 times on average. This was resulted from the hidden version data on the timeline, making the hovering actions unavoidable. The participants used the timeline to select the current version, and also the comparing version (Selection of comparing version). Some participants tended to click the versions adjacent to the targeted version on the timeline prior to locating the desired one. Some participants chose to click the targeted element before selecting the specific version, whereas some participants did it the other way around. There were three ways to perform comparison between two versions in the visual- isation (Ways to compare two versions with the visualisation). First, the toggle button was used the most by the participants, it was observed that they clicked it 101 times in total. The participants switched between the two modes to select either the current or the comparing version on the timeline, by clicking the toggle button (Selection of comparing version). Second, some participants used the compare dialog to select the version for comparison (Selection of com- paring version). Third, the slider was used by two participants, which had the least number of usage (2 times). The participants clicked a point on the slider or used the previous or next button to select a version (Selection of comparing version). Immediately after selection of the comparing version, the changes of the system, selected package or class were displayed in the diagram, according to the current abstraction level (Change identification with the visualisation). The counted number of occurrences shows that the participants mostly viewed the changes at the package level, with 74 times, whereas the numbers for the class and system levels are 18 and 1 respectively. Moreover, albeit uncommon, two participants had viewed the added or removed class list in the compare dialog. Some participants had opened the compare dialog to read the legend of the representations of the changes of class and dependencies, in order to understand the notations of the changes in the diagram, as one participant described his approach: “I also used the notations and legend to grasp the changes that have been done (types of arrows and different class colours)”. The research participants identified the dependencies between the elements in the software system using the visualisation (Dependency identification with the 53 6. Results visualisation). All participants reported the found changes of dependencies with- out validating with the code. The dependencies between the classes were examined at the package and class levels. At the package level, the participants identified the package-level dependencies between the selected package and the related packages with arrows in order to find answers to the question 2 of the task. One partici- pant stated, “Identify dependencies of packages very easily by simply looking at the arrows” and “It is much clearer to identify the whole structure and the relation- ship between packages of the system”. Furthermore, the participants identified the changed dependencies among classes between two versions, in particular the added and removed dependencies. A participant mentioned that “Instead of having to look at the code to know the dependency I could see if by the connection of the dots, and while comparing version I saw if the dependencies had changed”. Most participants utilised the role stereotypes when working on the task (Role stereotype utilisation). The major usage was to identify the role stereotype of a class by its colour with the use of legend. One of the participants mentioned the use of colour to identify the role stereotype of a class with the purpose of understanding the functionality of the class: “Identify the brief role of a class by looking its colour, which is helpful in understand the code and hence its functionality”. This also showed that the participant considered both the information of role stereotype and the cor- responding code in order to perform the task. Another participant focused on the change of colour for certain class to identify the change of role stereotype: “using the stereotypes colors, it was easy to spot the change in the role”. Some participants used the timeline component to identify significant change of role stereotype for the selected class, as stated by two participants: “Look at the change of colour in the timeline to see if there is a big change in the role” and “For the role changes I used the timeline to spot the change in colour indicating the change in stereotype role”. When selecting the comparing version, it was obvious that some participants deliberately pick a version where a change of role stereotype occurred, as shown in the timeline. In contrast to reading the source code of classes in order to answer the question regarding the responsibility of a class, some participants relied on the data of role stereotype to identify the responsibility and its changes over the targeted versions. The participants used the code feature in the visualisation to view the source code for the investigating classes (Codes in the visualisation). The participants ex- pressed “I have seen the code of the element of interest and used it to make con- clusions” and “Then use the code function to bring up the code to get a better understanding”. Some highlighted the code while reading the source code of the inspecting class, as well as searching a keyword in the code. One participant men- tioned the highlighted differences of the code of the same class between two versions: “Read highlighted codes to observe the change and new functionality”, to identify the changes on the source code. All the participants had used the code comparison fea- ture to view the changes in the source code of the same class. Nevertheless, only three participants had viewed the source code of a class alone without code com- parison. 54 6. Results The participants also used some supporting features provided by the visualisation (Supporting features). They used the zooming feature to zoom-in and zoom- out: “I also zoomed in- and out to better visualize the package of interest and that was helpful”. Some participants had used the history feature to reverse back to the previously viewed elements, instead of selecting the version and locating the pack- age/class again. One participant had downloaded the graph as image once; however, no preceding action relating the image file was observed. Some unforeseen behaviours were observed (Unexpected behaviours). Two par- ticipants had performed double clicking on a package or a class, altogether 9 times. Moreover, one of them double clicked a version on the timeline once. Some partici- pants tried to click an edge in the diagram, with 21 times in total. It was observed that the intention of clicking the edges was to click the package behind them. Two participants also attempted to open the source code at the package level, in which no code was available. 6.2.5.2 Approach without the Visualisation All research participants also worked on a task without the use of the visualisation (APPROACH WITHOUT THE VISUALISATION). To start with, they opened the listed code repository on the GitHub website (Code retrieval). Two of them chose to clone the code repository using GitHub Desktop, whilst three chose to view it directly on the GitHub website. One participant downloaded the code from the website after attempting to view the code repository on the website. Five software and tools were used by the participants when working on the task (Utilisation of tools). Given that the links to the GitHub code repository of the five commits were provided, all participants inevitably opened the links and used the GitHub website. As mentioned previously, two participants had used GitHub Desk- top to retrieve the code repository. One used Sourcetree and Visual Studio Code for viewing the commit history. Two participants used Sourcetree or command line to reset the current branch to a commit. Three participants used Eclipse, Visual Studio Code or Sublime Text to open and view the downloaded files. Regardless of the tool the participants used, navigation in the code repository was inevitable (Navigation). With the purpose of looking for a targeted package folder, the participants had hovered over an untargeted package folder (47 times) and clicked an untargeted package folder (20 times). The participants had clicked a targeted package folder 106 times in all. To locate a class, they hovered over an un- targeted class 25 times altogether. In all, they had clicked a class to open the code 129 times. A participant described his process: “Open the folders in separate tabs, containing the classes and then open the code to see how the changes to dependen- cies have taken place”. Two participants navigated to another class from the source code. The most frequent behaviour observed was switching tabs, with the sum of 552 times. Two of the participants had performed it over 200 times. Additionally, the participants navigated to the previous page 23 times in total. 55 6. Results All participants had read source code when working on the task without the vi- sualisation, and this was mentioned by all of them when describing their approach (Code viewing). One participant stated, “Read and understand the meaning of the code to understand its role and functionality”. Four participants had focused on the import declaration of the classes in order to answer the questions. The participants also highlighted source code when reading it. Searching was an action that most participants had performed (Searching). They used the search feature of the browser to search a keyword in the webpage. Two participants had performed file searching in Visual Studio Code or Sublime Text. Furthermore, a participant searched for a commit by using the commit hash or the commit message. All participants utilised the version control components (Utilisation of version control). They read the commit messages of the packages/classes/commits. Three participants had viewed the commit history, with 17 times in all. Some participants had reset the working branch to certain commit (4 times), as they put it: “reset the code to target version” and “Checking out the different versions of the commits”. Four participants also viewed the f