A Requirement-centric Framework to Build Tool Integration for Traceability and Con- sistency Master’s thesis in Computer science and engineering Dongying Cao Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY UNIVERSITY OF GOTHENBURG Gothenburg, Sweden 2024 Master’s thesis 2024 A Requirement-centric Framework to Build Tool Integration for Traceability and Consistency Dongying Cao Department of Computer Science and Engineering Chalmers University of Technology University of Gothenburg Gothenburg, Sweden 2024 A Requirement-centric Framework to Build Tool Integration for Traceability and Consistency DONGYING CAO © DONGYING CAO, 2024. Supervisor: Eric Knauss, Department of Computer Science and Engineering Advisor: Daniel Larsson, Shiva Kumar Gone, Volvo Car Examiner: Jennifer Horkoff, Department of Computer Science and Engineering Master’s Thesis 2024 Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg SE-412 96 Gothenburg Telephone +46 736582601 Cover: Description of the picture on the cover page (if applicable) Typeset in LATEX Gothenburg, Sweden 2024 iv A Requirement-centric Framework to Build Tool Integration for Traceability and Consistency Dongying Cao Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Abstract Many automotive companies use the ISO 26262 and Automotive SPICE (ASPICE) standards to develop software. These standards highlight the importance of trace- ability and consistency. However, in practice, different companies use various soft- ware tools for different tasks such as software testing and requirement management. Current methods to achieve traceability often depend heavily on specific suppliers or tools. Some methods require a lot of manual work, like maintaining Excel sheets, while others automate the process but are not efficient enough and stick to cer- tain standards. The issue of ensuring consistency has received less attention. A more generic and flexible approach is needed to establish traceability and identify inconsistencies using data, not relying on specific tools. This thesis study was conducted at Volvo Car Corporation with a software inte- gration and release team. Not just this team, but other departments also face a challenge: there is no effective workflow to establish traceability for automated HIL (Hardware in the Loop) testing. Requirements are managed in a tool called Car- Weaver, while test cases are executed either automatically in dSPACE Automation- Desk or manually in dSPACE ControlDesk. Currently, traceability and visualization are only managed for manual testing using Excel sheets. As automated HIL test- ing becomes more common, a method to also report the status of requirements is needed. Furthermore, the consistency required by ASPICE is lacking in the current approach. This thesis proposes a requirement-focused framework to create a tool integration framework for establishing traceability and checking consistency. This framework does not depend on any specific software vendor in the automotive industry. It involves collecting data on test results and requirements from various software tools and then processing and analyzing this data centrally. The study provides a practical example, including adjusting the test workflow to capture the right data and using APIs to transform requirement data. The central processing unit uses Python scripts for data handling and Power BI for visualization. This framework could also be useful in other fields where requirement traceabil- ity and consistency are issues, especially when different tools are used for various purposes. Design science research is adopted as the research methodology with two cycles of iteration, each involving problem investigation, solution design, design validation, solution implementation, and implementation evaluation. Keywords: tool integration, requirement traceability, consistency, automotive indus- v try, Automotive SPICE. vi Acknowledgements I would like to express my deepest gratitude to my supervisor, Eric Kanuss, for his invaluable guidance, patience, and expert advice throughout this project. His encouragement and support have been cornerstones of my journey. I am also im- mensely thankful to Jennifer Horkoff, my examiner, for her insightful comments and suggestions, which have significantly contributed to the refinement of this work. A special thanks goes to Shiva Kumar Gone, Jesper Fritsch, and Daniel Larsson at Volvo Car Corporation. Their industry insights and expertise were vital in bridging the gap between academic research and practical application. Their contributions have been crucial to the success of this project. I am also grateful to the entire Integration and Release team and the Complete Software team at Volvo Car Corpo- ration. Working alongside such talented groups has been both inspiring and reward- ing. Their collaboration and willingness to share their knowledge and experiences have enriched this experience. Dongying Cao, Gothenburg, 2024-06-18 viii x Contents 1 Introduction 1 1.1 Case Company . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Purpose of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.5 Scope and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.6 Significance of the Study . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.7 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Background 7 2.1 CI/CD and Automated Testing . . . . . . . . . . . . . . . . . . . . . 7 2.2 Requirement Traceability . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Tool Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.4 Challenges in Achieving Effective Traceability in Automative Industry 11 2.5 Traceability Visualization . . . . . . . . . . . . . . . . . . . . . . . . 12 2.6 Hardware-in-the-loop Testing . . . . . . . . . . . . . . . . . . . . . . 12 2.7 Tools and Software Used at the Company . . . . . . . . . . . . . . . . 13 2.7.1 dSPACE AutomationDesk . . . . . . . . . . . . . . . . . . . . 13 2.7.2 CarWeaver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.7.3 XML, XSLT, XSD . . . . . . . . . . . . . . . . . . . . . . . . 14 2.7.4 Power BI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.7.5 Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3 Research Methods 17 3.1 Design Science Research . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 DSR Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.1 Problem Investigation . . . . . . . . . . . . . . . . . . . . . . 18 3.2.2 Solution Design . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2.3 Design Validation . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.4 Solution Implementation . . . . . . . . . . . . . . . . . . . . . 21 3.2.5 Implementation Evaluation . . . . . . . . . . . . . . . . . . . 22 4 Findings 25 4.1 Cycle I Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1.1 Problem Investigation . . . . . . . . . . . . . . . . . . . . . . 25 xi Contents 4.1.2 Solution Design . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.1.3 Design Validation . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.4 Solution Implementation . . . . . . . . . . . . . . . . . . . . . 35 4.1.5 Implementation Evaluation . . . . . . . . . . . . . . . . . . . 46 4.2 Cycle II Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2.1 Problem Investigation . . . . . . . . . . . . . . . . . . . . . . 52 4.2.2 Solution Design . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.3 Design Validation . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.4 Solution Implementation . . . . . . . . . . . . . . . . . . . . . 54 4.2.5 Implementation Evaluation . . . . . . . . . . . . . . . . . . . 55 5 Discussion 57 5.1 Revisiting the Research Questions . . . . . . . . . . . . . . . . . . . . 57 5.2 Contribution to Knowledge . . . . . . . . . . . . . . . . . . . . . . . . 59 5.3 Contribution to State-of-the-Art . . . . . . . . . . . . . . . . . . . . . 60 5.4 Validity Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4.1 Construct Validity . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4.2 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.4.3 External Validity . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.4.4 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 6 Conclusion and future work 65 Bibliography 67 A Appendix I A.1 Evalution Survey in the First Iteration . . . . . . . . . . . . . . . . . I xii 1 Introduction In today’s automotive industry, software plays an increasingly critical role. The vol- ume of software integrated into vehicles is expanding and the size reaches 100 Million lines of code [1]. At the same time, software complexity and its application in con- trolling safety-critical functions have surged as well [2]. During the development process, numerous artifacts are produced, such as requirements, test cases, imple- mentation code, and so on. Therefore traceability should be managed and achieved effectively [3]. The automotive industry adheres to standards like ISO 26262 and Automotive SPICE (ASPICE) throughout the development cycle [4], which often highlight the significance of traceability and consistency. Well-established traceabil- ity facilitates the verification of requirements [5], thereby ensuring the high quality and safety of the software and promoting product delivery. However, due to the large number of artifacts in modern automotive systems and various software in use, establishing and interpreting traceability has become a chal- lenging task. For interpretation, visualization tools aid in presenting the complicated relationships between requirements and test cases in a more understandable way [6]. Software should be tested thoroughly before the delivery to stakeholders. Testing activities in the automotive industry often require interaction with external elements like signals. These activities are typically carried out in simulated environments to ensure accuracy and safety before they are implemented in an actual vehicle. Volvo Car Corporation employs hardware-in-the-loop (HIL) testing for this purpose, using AutomationDesk for most of the automated testing and ControlDesk for manual testing. In manual HIL testing, a process is in place to record test results and track requirement traceability using Excel sheets and manual work. However, automated HIL testing lacks a similar process. The software systems do not automatically connect, resulting in missing data. This creates a gap in traceability, leading to issues such as no visualization of requirements and no checks for inconsistencies. Also, reports from AutomationDesk do not include links between test cases and requirements. If stakeholders want to know which requirements are linked to failed test cases, they must undergo a time-consuming manual search. Therefore, it is necessary to develop a new solution to simplify access to traceability. Due to the complexity of software, automated testing has become essential [7]. Volvo Car Corporation is shifting its focus from manual to automated HIL testing. The lack of traceability in automated testing is becoming a major problem. This study uses design science research methodology to explore traceability challenges in auto- 1 1. Introduction mated HIL testing at Volvo Car Corporation. It suggests a framework for integrat- ing tools to establish requirement traceability and ensure consistency checks, which can be applied in broader contexts. This framework could help other companies improve traceability and consistency across different software platforms, such as dSPACE AutomationDesk for automated HIL testing and SystemWeaver for man- aging requirements. This framework differs from existing methods for integrating tools to track requirements, such us using software product provided by particular vendors or filling Excel templates, as it places few restrictions on the software or architecture used and does not depend heavily on specific suppliers or tools. It also moves away from the widespread yet inefficient practice of using Excel to manage traceability in complex, multi-software environments. 1.1 Case Company Volvo Car Corporation, with its research and development department headquar- tered in Sweden, is a leading international manufacturer of luxury vehicles. The company has adopted a hybrid agile methodology called the Volvo Cars Agile Frame- work (VCAF). At the heart of VCAF’s organizational architecture is the Product Stream concept. A Product Stream is comprised of multiple Agile Release Trains (ARTs), which are long-term, dedicated, cross-functional teams working toward a unified goal. This thesis is conducted within the Integration and Release (IAR) Team in the Power Electronics Software ART (PESW), which is part of the Electric Propulsion Product Stream under the Engineering department. PESW is responsible for developing optimal drive control software for electric and hybrid vehicles to ensure superior customer satisfaction. The IAR team mainly works with the integration and delivery of inverter software. The IAR team integrates inverter software developed within the application layer of the Autosar stack as described in [8]. At this stage, automated HIL testing is performed as smoke testing, which is a quick and preliminary test of a software build, focusing on basic functionality to ensure the major components are working before deeper testing is performed. The team then delivers the integrated software to external suppliers, who build the complete inverter software. Upon receiving the finalized complete software, comprehensive manual HIL testing, known as accep- tance testing, will be performed to confirm that all relevant requirements defined in a software called CarWeaver are met. The complete inverter software will then be sent to the internal stakeholders of the company. Currently, HIL smoke testing is automated through Jenkins pipelines, running test cases with dSPACE AutomationDesk. In contrast, HIL acceptance testing is man- ually performed using dSPACE ControlDesk. Both processes use Excel sheets to build and visualize traceability. However, as the company shifts from manual to automated HIL testing, there is a growing need for an automated traceability sys- tem. This new system will support quicker and smoother software development and release cycles. 2 1. Introduction 1.2 Purpose of the Study The purpose of this study is to develop a method to automate traceability in complex environments where different software is used for various purposes, particularly in the automotive industry. This will be achieved by creating a framework that focuses on data integration and requirements. The main beneficiaries are all stakeholders involved in the software integration and delivery process at Volvo Car Corporation. This includes integration and delivery engineers, developers, testers, suppliers, and project managers. The study seeks to offer a flexible and broadly applicable method that can be used by various companies across different industries to establish traceability between different systems and software. For Volvo Car Corporation, this research is expected to support more strategic decision-making and improve the efficiency of software delivery. 1.3 Problem Statement In the automotive industry, HIL testing is typically conducted before real vehicle testing. The case company, Volvo Car Corporation, is transitioning from manual to automated HIL testing before software release. However, they face challenges due to the lack of an effective method for establishing requirement traceability in automated HIL testing. While there is a workflow for building traceability in manual HIL testing, it is unsuitable for automated testing because it requires significant manual effort. This lack of traceability has led to issues such as missing information in test reports, the inability to automatically analyze project requirements, and the absence of proper visualization to aid software release. Many companies, including Volvo Car Corporation, struggle with traceability man- agement and maintaining consistency among software artifacts. Factors contributing to this include distributed teams, frequent software updates, changing trace links, and varied stakeholder backgrounds. The limitations of current tool integration practices make it challenging for companies to address these issues. Therefore, a more generic and flexible method is required to build requirement- centric traceability in a complex environment where various software interact. 1.4 Research Questions The proposed study aims to address three research questions. RQ1 - What is the problem with the current status of requirement traceability in the software build and automated test workflow? This question aims to investigate the challenges and limitations regarding require- ment traceability in the existing workflow faced by the case department. RQ2 - What improvements in test reporting and visualization are sought by stake- holders? 3 1. Introduction This question aims to understand stakeholders’ expectations for traceability within the test report and the following visualization of test case execution results. RQ2.1 - How should traceability information be integrated into the HIL testing report? This question aims to determine the specific traceability details between HIL test cases and corresponding requirements that stakeholders want to see included in the test reports. It also considers the presentation of this traceability data, including style and layout preferences. RQ2.2 - In what way would the stakeholders prefer traceability data to get visual- ized? This question aims to discover the preferred methods and formats for visualizing the execution results of automated HIL test cases. RQ3 - How can improved traceability in test reports and visualizations enhance the efficiency of software integration and release processes? 1.5 Scope and Limitations HIL testing at Volvo Car Corporation is categorized into four levels: Component, Domain, Complete, and Vehicle HIL testing. Discussions with the team highlighted their desire to incorporate traceability information in reports and export execution results in JSON format across all testing levels. However, given the limited time- frame of this master’s thesis project, this study will concentrate on Component HIL testing. The case company frequently updates its vehicle platforms and introduces new tools at the onset of new platform development, leading to variations in working meth- ods. This thesis investigates a specific platform project where CarWeaver is used for managing requirement information and test case definitions. It’s important to note that while previous projects may have employed different tools for requirement man- agement, this study is limited to projects utilizing CarWeaver and does not address the transition of requirement management tools for other projects. Moreover, adding a traceability table into HIL test reports requires modifications to existing test cases. It needs to be clarified that this study does not update test case details such as requirement IDs and test case IDs in CarWeaverthis is a responsibility that remains with the test case developers. The framework suggested in this study can work with any software that can share its data, such as through an API or direct export. However, if the software does not have a method to export data, the framework will not be completely automated. In such cases, manual effort to export data will be necessary. 4 1. Introduction 1.6 Significance of the Study In this design science research, the author contributes to both academia and in- dustry by proposing an innovative and automated framework centered on data and requirements for developing a tool integration framework that is independent of any specific software vendor or standard. This framework offers a more universal, adapt- able, and automated solution compared to existing solutions, facilitating traceability establishment and consistency maintenance. Therefore, it is applicable not only to automotive companies but also to other industries facing similar challenges with traceability. Volvo Car Corporation is the biggest beneficiary here since the study bridges the gap in requirement validation for future fully automated HIL testing processes. This improvement will bring smoother software delivery. The framework developed in this study can be applied across Agile Release Trains (ARTs), product streams, and departments at Volvo Car Corporation. With the tool’s source code available in- house, it also serves as a benchmark for future tools designed to extract or augment data from AutomationDesk and CarWeaver for advanced analytics. 1.7 Outline The rest of this report is structured as follows: It begins with a section on background information, which describes the main tools and techniques used by the case team. This is combined with a review of related literature that provides support for the thesis. Next, the research methodology used in this study is detailed. The findings of the two iterative phases carried out during the thesis are then presented, leading to a discussion of these results. The report concludes with a summary and perspectives on future research directions. 5 1. Introduction 6 2 Background This chapter introduces the key concepts, techniques, and tools relevant to the study. 2.1 CI/CD and Automated Testing Continuous Integration (CI) is a widely established development practice in the soft- ware development industry [9]. It refers to the practice of merging all developers’ working copies to a shared mainline several times a day [10]. Automated software building and testing are usually included in CI practice [11]. As summarized by Shahin et al. [12], CI reduces build and test time and increases visibility and aware- ness of build and test results. The subsequent delivery process benefits from CI due to the improvement of dependability and reliability. Jenkins is widely used as a CI tool in companies. Continuous Delivery is a software development discipline where software is developed in a manner that allows it to be released to production at any time [13]. The discipline is achieved through optimization, automatization, and utilization of the build, deploy, test, and release process [14]. Both CI and CD underscore the critical role of employing automated testing. Auto- mated testing is when software tools execute tests before it is released into produc- tion, ensuring that the software behaves as expected. General software testing is composed of five phases. The first phase, test planning, is mainly to plan all the test activities that are to be conducted in the whole testing process. The second phase is test development, where the test cases are developed to be used. Test execution is the next phase. It encompasses the execution of the test cases. The fourth phase is test reporting where the relevant bugs are reported. Test result analysis is the last stage in which the software developers analyze the defect. It can also be performed together with the client as it will help in the better understanding of what to ignore and what to fix, enhance, or modify [15]. Nowadays about 90% of all the current car innovations are based on electronics and software and the software used in a car has increased a lot reaching the size of 100 Million lines of code [1]. Therefore, automated testing is becoming increasingly important in the automotive industry to address the needs of complex software systems effectively. In the context of automotive systems, test automation consists of the following three steps: test case generation, test execution, and result analysis. 7 2. Background The generation step usually utilizes state charts, control flow, information flow, and MSCs (Message Sequence Charts). When the second step, test execution, is completed, the test results should be analyzed to unveil the error that caused a test to fail and discover errors that need to be fixed [7]. 2.2 Requirement Traceability Requirement traceability is defined as the ability to describe and follow the life of a requirement in both forward and backward directions (i.e., from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases) [16]. Traceability helps trace interdependencies between different types of artifacts [17]. For example, it can tell which test cases verify a certain requirement and identify whether the requirement is issued by a person or document or a group of persons [5]. Hyun Cho discussed the growing complexity of systems and the evolving needs of customers [3], highlighting the importance of traceability in software development: traceability ensures the quality and timely delivery of systems under tightening de- velopment schedules and budgets; it also aids in making informed decisions, assess- ing implementation completeness, and identifying potential for reusing components, thereby facilitating more effective system development processes. Volvo Car Corpo- ration is currently using a platform called CarWeaver for requirement management. 2.3 Tool Integration In the automotive industry, many companies follow the international standard ISO 26262 to guarantee high-level functional safety. ISO 26262 emphasizes the impor- tance of traceability throughout the entire development process to ensure that each safety requirement is implemented and verified. In addition to ISO 26262, global automakers are required to apply the Automotive SPICE (Software Process Improvement and Capability Determination) framework to evaluate the development capabilities and qualities of suppliers for preventing accidents caused by errors in the electronic components [18]. It provides a set of requested practices in the development life-cycle, including system and software testing. According to the Automotive SPICE process reference model, there are five processes directly dealing with system and software testing: System Integration and Integration Test, System Qualification Test, Software Unit Verification, Software Integration and Integration Test, and Software Qualification Test [19]. ASPICE highlights the importance of bidirectional traceability and consistency between ar- tifacts for the testing processes. Falcini et al. shortly described the base practices of the five testing processes according to [20]. Seven base practices of the software qualification test process are listed in Table 2.1. As can be seen in BP5 and BP6, bidirectional traceability needs to be built between requirements and test cases and there should be no contradiction between them. 8 2. Background Id. Definition BP1 Develop software qualification test strategy including re- gression test strategy consistent with the project plan and the release plan. BP2 Develop the specification for software qualification test including test cases based on the verification criteria, ac- cording to the software test strategy to provide evidence for compliance of the integrated software with the soft- ware requirements. BP3 Select test cases from the software test specification. The selection of test cases shall have sufficient coverage according to the software test strategy and the release plan. BP4 Test the integrated software using the selected test cases. Record the software test results and logs. BP5 Establish bidirectional traceability between software re- quirements and software qualification test cases. BP6 Ensure consistency between software requirements and software qualification test cases. BP7 Summarize the software qualification test results and communicate them to all affected parties. Table 2.1: SWE.6 Software Qualification Test Process Base Practices However, in real-life practice, achieving effective traceability management and ensur- ing consistency are often difficult. Some reasons can be software projects usually in- volve distributed teams, experience constant changes to software artifacts and trace links, and include multiple stakeholders with different backgrounds [21]. Amalfitano et al. [22] also observed insufficient support for traceability creation and management activities in the automotive industry. A significant factor is the lack of integration among the tools employed throughout the software development lifecycle. Besides, traceability relationships are still manually established and maintained sometimes. As suggested by Maro et al., the challenges for traceability, stemming from the use of diverse artifacts and tools, can potentially be resolved through the adoption of an integrated tool platform or tool integration [23]. Research into tool integration began in the 1990s [24]. It aims to link software development tools into a cohesive workflow to support the entire software lifecycle. By allowing data and information to flow smoothly between different development stages and components, toolchain integration becomes highly beneficial in complex areas like the automotive industry. Proper integration of these tools can enhance traceability and help with consistency, ensuring that the final product meets all requirements and quality standards that the project should follow. Some research on toolchain integration within the automotive domain has been conducted based on particular projects [25] [26] [27]. The dSPACE company intro- duced SYNECT as a system integration tool for Model-Based Design systems [28]. 9 2. Background It provides centralized data management for the integration of artifacts of testing and development. The tool can be easily applied to Hardware-in-the-Loop (HIL), Model-in-the-Loop (MIL), and Software-in-the-Loop (SIL) testing. However, its use is limited to companies that employ dSPACE products for simulation testing. Com- panies using standalone requirement management tools might not use SYNECT, or they may need significant effort and resources to integrate it. Here, a clear gap remains. The projects explored above often depend on specific tools and platforms, which constrains the generalization and applicability of the proposed solutions. Such diversity means that those solutions are custom-fit for certain situations, which is a challenge when trying to use them in different contexts. A more commonly used software, Microsoft Excel, is adopted by some companies to establish traceability. Navas et al. reported manual efforts in a case study [29]. The case company in this study uses a requirement traceability matrix document to create and maintain traceability links of requirements and test cases. This document must be manually completed by the person in charge of testing. The manual process is tedious, and tracking artifacts becomes challenging when changes occur and the document is updated. Consistency validation was not covered in this study. Some companies have automated their use of Excel for maintaining traceability instead of building it manually, but issues remain. Excel only records the links or IDs of artifacts, without knowledge of the artifacts themselves. As a result, traceability is managed separately from the actual artifacts, increasing the risk of inconsistencies between the links and the artifacts themselves [30]. Klespitz et al. introduced a method called Augmented Lifecycle Space (ALS), build- ing on Open Services for Lifecycle Collaboration (OSLC), to fill bidirectional trace- ability gaps and correct inconsistencies in either homogeneous or heterogeneous de- velopment environments [31]. In heterogeneous environments, information remains isolated and inaccessible to other tools. Conversely, in homogeneous environments, tools within the system can exchange data freely, and all artifacts are available to any tool that requires them. The core concept of ALS involves generating workflows automatically. The steps to get ALS proposed by Klespitz et al. include categoriz- ing existing artifacts within the tool environment, exploring their relationships, and automatically generating any missing artifacts based on the model’s traceability and consistency requirements. To illustrate this approach further, Klespitz et al. provided a practical example where a heterogeneous system was established using IBM Rational DOORS for re- quirements management and Atlassian JIRA to manage test information, including test cases and results [32]. Integration between these tools was accomplished through a simple CSV file exchange. Initially, DOORS generates a CSV file containing refer- ences to relevant test cases. This file is then updated by JIRA, which appends test results and aligns them with the required test identifiers. The updated CSV file is sent back to DOORS for comprehensive analysis. This method has shown its ability to improve traceability and consistency through practical demonstration, yet its broad applicability is limited. The success of the ALS heavily depends on the adoption of the OSLC standard across all utilized tools. 10 2. Background Challenges arise when some tools lack OSLC support or when there are inconsisten- cies in their implementation. Additionally, the ALS approach requires significant effort and expertise to integrate various tools with diverse functionalities, interfaces, and data formats, particularly in heterogeneous environments. The need to develop OSLC adapters for tools that do not support the standard natively adds to this complexity. Moreover, the ALS should ideally identify inconsistencies automatically, which is not always achievable. There is a growing need for a more universal and automated solution that automo- tive companies can adopt to enhance traceability across integrated toolchains and improve consistency among artifacts. This solution should minimize dependency on external vendors and specific standards. Such a solution would not only improve the efficiency of traceability management but also promote a smoother adaptation process for companies looking to implement or improve their traceability systems. 2.4 Challenges in Achieving Effective Traceability in Automative Industry A comprehensive study by Maro et al. identified 22 traceability challenges through two literature reviews and a case study conducted at an automotive supplier com- pany [23]. These challenges are broadly categorized into seven areas: knowledge of traceability, tools, human factors, organization and process, uses of traceability, mea- surement of traceability, and exchange of traceability information. Notably, three challenges are closely related to the problems addressed in the thesis. Inaccessibility of artifacts: The first challenge involves the difficulty in accessing artifacts that need to be connected by the traceability link, especially in projects with a large number of artifacts. Tools that support searching by ID or keywords can make it easy for users to find necessary artifacts, so the basic method to build traceability, which is manually copying the ID from one tool to another, becomes available. Perceived as an overhead: The second challenge is the perception among devel- opers that creating traceability is an additional, disruptive activity to their workflow. This perception arises partly because the individuals creating traceability links often do not benefit directly from them, leading to low motivation and potentially incor- rect or missing links. The proposed solutions are to ensure that the traceability links created provide immediate benefit to the creators and also to automate the tasks whenever possible. Lack of proper visualization and reporting tools: The third challenge focused on the lack of visualization and reporting tools for traceability links, which are pre- sented in large tables or lists. This makes it hard to comprehend the links or detect the flaws in them. Insufficient visualization can arise from the manual creation of traceability links, such as copying IDs between artifacts. Some requirement man- agement tools used in the industry don’t support such kind of visualization. The absence of visualization also results in the traceability links never being used or 11 2. Background being under-utilized. To address these issues, it is essential to provide end users with proper visualization tools and powerful reporting tools capable of producing overviews and statistics for reviews. Effective visualization should enable users to see which requirements are already implemented and tested or which tests do not have corresponding requirements. Additionally, tool vendors are encouraged to develop features that allow custom reports to be generated from traceability information based on user needs. 2.5 Traceability Visualization Visualization is a process of "representing data, information, and knowledge in a visual form to support the tasks of exploration ... and understanding" [33]. It occurs when interacting with visual elements, which represent data that have been collected, stored, fetched, filtered, transported, and transformed to and from some data storage [34]. In extensive and complex systems, there are more traceability links and the com- plexity of their usage increase. Therefore proper visualization helps understand traceability. Li et al. discussed three main concepts of traceability visualization: the traceability information (what to visualize), the visualization technique (how to visualize), and the task context (when to visualize) [6]. The authors suggested visualizing the artifact types, artifact metadata, granularity, artifact attributes, link information as well as confidence. Techniques like matrices, graphs, lists, and hyperlinks are found to be useful and helpful. Several techniques of traceability visualization were proposed by Heim et al. and Duan et al. as well, aiming to help users understand the "cloud of links" and get to the underlying information smoothly [35] [36]. Traceability visualization can be employed in various tasks of software development, ranging from management, design, and implementation, to testing. In software testing, it validates if requirements have been implemented correctly [37]. It is usually linked to artifacts like test cases and requirements. Visualizations of software testing support hypothesis generation, and decision-making, and can map to information needs [38]. The visualization of test results, including traceability information, is important. It helps understand the test coverage and the relationship between tests and require- ments, making identifying gaps in the testing strategy easier [39]. Good visualization supports better decision-making by providing clear insights into the software quality and areas that may need more focused testing or development efforts. 2.6 Hardware-in-the-loop Testing Hardware-in-the-loop (HIL) simulation is a technique for performing system-level testing of embedded systems in a comprehensive, cost-effective, and repeatable man- ner [40]. It rigorously tests the functionalities, system integration, and communi- cation of electronic control units (ECUs) within a simulated environment. A HIL 12 2. Background simulator is connected to a test object and an HIL PC to enable HIL testing. In the case department, there are three ways to perform HIL testing: • Manual execution of in software called dSPACE ControlDesk. • Direct trigger of automated HIL testing via software called dSPACE Automa- tionDesk. Users select the project to be tested and import the necessary libraries to execute HIL test cases directly through the AutomationDesk soft- ware. • Indirect trigger of automated HIL testing via Jenkins. There are two types of pipelines defined for this purpose. The first pipeline is triggered on demand when testers need to perform HIL testing on a project. The second pipeline operates as follows: a developer commits changes to the source code. Jenkins will query the Version Control System for any source code updates. Upon detecting an update, Jenkins triggers a build job including various steps in a pre-defined sequence. When the software has been compiled and packed in a format suitable for installation on the test object, a test request is dispatched to the HIL PC. A script on the HIL PC then downloads, unpacks the software, and launches a specific test suite, such as an acceptance test. The outcomes of the test phase are reported back to Jenkins. Finally, the end user can access the Jenkins Web server to review the build and test results. 2.7 Tools and Software Used at the Company 2.7.1 dSPACE AutomationDesk The case company uses a tool named AutomationDesk from dSPACE, a leading supplier of HIL simulators at Volvo Car Corporation, for conducting automated HIL testing. AutomationDesk offers a user-friendly graphical interface, enabling test case developers to create and modify test routines without the need for programming expertise. Essential concepts in AutomationDesk related to this thesis include project, se- quence, library, data object, result, and report: • A project in AutomationDesk serves as a container for all instantiated re- sources that implement a specific automation task, termed an automated test case in this thesis. It encompasses the implementation of HIL test cases, out- comes of executed tasks, and their corresponding reports. The Project Man- ager pane allows users to handle and display project elements in a hierarchical layout. • A sequence represents the implementation of an automation task as a control flow, defined with automation blocks. It is editable via the Sequence Builder pane. The main components of a sequence are blocks. Users can create blocks independently, or select from template blocks available in AutomationDesk libraries, both built-in and customized. The graphical interface facilitates the 13 2. Background easy organization of blocks into a control flow chart to form a sequence and execute test cases. • A library acts as a storage for templates that aid users in creating sequences, data objects, or automation blocks within their projects. Accessible through the Library Browser panel, libraries in AutomationDesk are categorized into built-in and custom libraries. Built-in libraries come as predefined software components, with source code hidden from users, whereas custom libraries allow users to create elements per their requirements. Valid elements that can be added to a custom library are templates for data objects, blocks, and sequences. • Data objects are designed to hold values of various types such as string, float, int, and file, among others. They are instantiated specifically for a project, sequence, or an automation block. A data object can also be referenced to other data objects. • A result is a set of data derived from executing a project, project folder, or sequence. Based on the result, users can generate a report document. While AutomationDesk provides default PDF and HTML formats for reports, users have the option to customize style sheets via XSL files for PDF report gener- ation. 2.7.2 CarWeaver To manage system designs and requirement documentation, the case department employs CarWeaver, a derivative of SystemWeaver which is an information manage- ment platform by Systemite. CarWeaver is enhanced with various plugins by Volvo Car Corporation to customize both the model and its functionalities to meet specific needs. CarWeaver’s primary function within the department is to describe test case, along- side establishing and managing a multi-level requirements framework. This setup supports a many-to-many traceability matrix, allowing a single test case to corre- spond to multiple requirements, and vice versa, ensuring comprehensive coverage and integration. However, CarWeaver’s capabilities are focused on describing test case information, such as test case ID, test case purpose and so on, rather than implementing or executing these test cases, which is handled by other software. CarWeaver does not support the establishment of traceability between test case implementation, test case execution and requirement specification. The thesis mainly utilizes the requirement ID and test case information in Car- Weaver. 2.7.3 XML, XSLT, XSD XSD (XML Schema Definition), XML (eXtensible Markup Language), and XSLT (eXtensible Stylesheet Language Transformations) are interconnected technologies 14 2. Background used to transform and represent XML data. They’re used in test report generation, web services, and so on. XSD, also known as XML Schema, defines the structure and content of XML doc- uments. It outlines how to describe XML document elements, including data types, sequences, and so on. XSDs ensure XML documents adhere to defined constraints and structures. It also validates if an XML document meets the expected format. XML is a markup language used for encoding documents in a format readable by both humans and machines. Its primary purpose is data storage and transportation, with a focus on simplicity, generality, and usability across the Internet. Users can create and define their own tags and document structure in XML. XSLT is a language for transforming XML documents into other XML documents, HTML for web pages, or other document types such as PDF. It applies stylesheets to XML documents, transforming and rendering the original document into a new format using XPath. In summary, XSD defines the data structure, XML represents the data itself, and XSLT determines how to display the XML data. AutomationDesk generates reports based on execution results, which are provided as XML files. These XML files follow the rules set by XSD files that are provided by AutomationDesk. To customize the report’s stylesheet, users must write XSLT files. 2.7.4 Power BI Microsoft Power BI, developed by Microsoft, is an interactive tool designed for data visualization. It supports data import from various sources including databases, web pages, PDFs, and structured files like spreadsheets (CSV, XML, JSON, XLSX) and SharePoint. The case company uses Power BI to display diverse data types through an easy-to-use interface. 2.7.5 Jenkins Jenkins is an open-source software that streamlines the automation of various soft- ware development tasks such as building, testing, and deploying. This capability supports the principles of continuous integration and continuous delivery by ensur- ing that new changes are seamlessly integrated. Within the scope of the the thesis, Jenkins is used for HIL testing, which can be initiated automatically after a success- ful build or manually, based on user requirements. 15 2. Background 16 3 Research Methods The research methodology adopted in this study is design science research (DSR). The thesis project follows the regulative DSR cycle described in [41] and the guide- lines for applying DSR outlined in [42] [43]. 3.1 Design Science Research Design Science Research (DSR) is a methodological paradigm within Information Systems. It seeks to extend the boundaries of human and organizational capabil- ities by creating new and innovative artifacts [43]. These artifacts, from models and methods to processes and software systems, are developed to address specific organizational challenges or opportunities. DSR is inherently a problem-solving process. The core of DSR lies in its iterative process of building and applying these artifacts to gain knowledge and understanding of both the problem domain and the solution space. A design science contribution can be based on a new artifact. It can also be an im- provement upon an established solution to a well-known problem or just a marginal modification of an existing artifact [44]. The main goal of this study is to build tool integration for requirement traceability in complex environments. It involves the development of a requirement-centric framework that has high generality. Such an objective aligns with the principles of design science research, making it the chosen methodology for our study. In line with the general guidelines for conducting design science research, the study also adhered to the seven guidelines for running a DSR master thesis with industry provided by Knauss [42]. 3.2 DSR Iterations The practical DSR framework applied in this thesis is the regulative cycle described by Roel Wieringa [41]. The regulative cycle can be found in Figure 3.1. Design science research should be conducted in several iterations. Each iteration starts by investigating a practical problem. From the second iteration onwards, each iteration is built on the previous one, meaning the problem also comes from solving earlier 17 3. Research Methods practical problems. The next step is designing solutions for the problem identified earlier. These solutions are then validated. One chosen design will be implemented. The outcomes of this implementation are then evaluated, which could be the start of a new turn through the regulative cycle. Figure 3.1: A regulative cycle This thesis research was carried out over two iterations, each containing the complete range of design science research steps, from the initial problem investigation to the implementation evaluation. The following sections give an overview of both the definition and application of DSR activities. They also include a summary of how these activities were implemented in the context of this thesis. 3.2.1 Problem Investigation Problem investigation is the first step in the regulative DSR cycle. It seeks to gather insights and comprehension about the problem without changing it. The problem should be clearly described and explained after this step. Additionally, it should outline the potential outcomes if the problem remains unaddressed. The problem investigation can be categorized into four types depending on the underlying reasons and purposes [41]. Problem-Driven Investigation: This is essential in the early stages of DSR. The purpose of problem-driven investigation is to describe, explain, and diagnose the problems. At the very beginning of the thesis study, the author examined the team’s respon- sibility, current workflow, and way of working. This was done by reading through documents hosted on the company’s documentation platform named Confluence, participating in the teams introductory presentation, checking the latest acceptance testing report, and communicating with team members. Upon gathering basic background information about the company and team, the au- thor proceeded to organize multiple meetings and discussions with a system engineer from the team. These interactions focused on exploring the traceability issues en- countered by the case ART. As the system engineer has six years of experience in the 18 3. Research Methods company, working closely with software integration and delivery, while others in the team have limited experience in requirement traceability, the initial problem inves- tigation was strongly supported by the system engineer. During the problem-driven investigation, the author also looked into the existing HIL testing reports, current web pages of test result visualization, and the workflow to build the visualization. A literature review was also performed on the challenges and potential solutions regarding traceability and consistency in both the general software industry and the automotive sector specifically. This review helped in a deeper understanding of the reason and situation of common traceability issues, thereby equipping the author with insights for designing solutions applicable beyond the case company. Goal-Driven Investigation: This is motivated when there is no problem expe- rienced but changes are expected to happen in agreement with some goals. Goal- driven investigation usually involves an analysis of goals to be achieved. This can be done through the description of stakeholder goals, definition and operation of them, as well as prioritizing the goals. The goal-driven investigation was not included in this study because the problems related to the study already exist within the case department. Solution-Driven Investigation: This is based on the search for new technology that can solve the problems. The investigation can start by exploring new technology to understand its properties. This thesis study doesn’t focus on any specific new technology. Impact-Driven Investigation: This is to investigate the results of previous ac- tions, instead of preparing for future solutions. Therefore it’s also called evaluation research. This kind of investigation usually involves describing the earlier imple- mented solutions, analyzing their impacts, identifying stakeholder goals, and so on. The second iteration cycle used the impact-driven investigation to identify issues in the artifact implemented during the previous cycle. The problem investigation activities were part of the evaluation stage of the first cycle as can be seen in Fig- ure 3.2. The first iteration concluded with a workshop presenting the framework, followed by a survey. The overlap between the evaluation of the first cycle and the problem investigation of the second cycle occurred after the survey. It began with a focused discussion group to analyze the feedback from the workshop and continued with an initial code review session. The problem investigation in this cycle concen- trated on the security of the designed framework and potential improvements to the implemented workflow. 19 3. Research Methods Figure 3.2: Evaluation activities of the first cycle and problem investigation activities of the second cycle. The workshop, survey, code review and focused discussion group together evaluated the artifacts developed in the first cycle. The code review and focused discussion group also identified problems in the second cycle. 3.2.2 Solution Design Solution Design is the process of creating innovative solution candidates. In the initial iteration cycle, the author reviewed the literature on existing solutions to traceability and consistency challenges. This review provided valuable insights into tool integration. Although the study identified multiple problems in the current traceability workflow, it did not create separate solutions for each issue. Inspired by existing toolchains, the author designed a more generic and broadly applicable framework to connect different software within the context of a safety-critical envi- ronment, focusing on requirement-centric approaches. Once the idea of the framework was decided upon, the author explored the software tools involved in-depth to develop detailed designs. This included the requirement management tool CarWeaver and the automated HIL testing software dSPACE Au- tomationDesk. A critical part of this process was reviewing documentation on Au- tomationDesk to understand its features fully and gain insights into the detailed design of the framework. This review focused on the methods for incorporating specific information into test sequences as desired and exporting test results in a format that could be converted for compatibility with data from other software. Discussions with the company’s system engineer, as mentioned earlier, also signifi- cantly influenced the solution design during the first cycle. The design of the solution for the second iteration cycle primarily stemmed from the focused group discussions and the initial code review session held after the evaluation workshop in the first cycle. This cycle emphasized improving the frame- work implemented previously. Therefore, devising potential solutions became more straightforward than in the initial cycle. 20 3. Research Methods 3.2.3 Design Validation Design validation is a knowledge task used to assess if a proposed design will help stakeholders achieve their goals once it is implemented. Roel Wieringa proposed three validation questions to guide this evaluation, listed as follows [41]. • Internal validity: "Would this design, implemented in this problem context, satisfy the criteria identified in the problem investigation? This contains two subquestions: In problem domain D, would solution S have effects E? Does E satisfy stakeholder criteria C?" • Trade-offs: "How would slightly different designs, implemented in this context, satisfy the criteria?" • External validity: "Would this design, implemented in slightly different con- texts, also satisfy the criteria?" The thesis study followed the guidance to perform design validation. In iteration one, all three questions were answered for the proposed framework. In the second iteration, the focus was only on internal validity, since this iteration mainly made improvements to the previously developed framework according to the evaluation feedback. 3.2.4 Solution Implementation One or multiple solutions that are designed and validated in the previous steps are implemented in this stage. Implementation of iteration one focused on building the framework, while the one for the second iteration mainly addressed the feedback collected from the evaluation of the first iteration to improve the framework. As the proposed framework fetches data from different software and processes them to get analysis results, the implementation covers data generation, data fetching, data processing, data analysis, and visualization. The framework implementation started with the traceability establishment from Au- tomationDesk. The first part is a suite to generate a traceability table in the test report. It includes a customized library in AutomationDesk to add traceability in- formation to the test result, an updated XSL script for report styling, and guidance for updating the test cases. The second part features a key Python script to parse the test result from XML format to JSON format that contains necessary traceabil- ity data. The script can be executed with various parameters, either through the Jenkins pipeline script or directly within AutomationDesk via a built-in GUI tool that is also developed by this study. The implementation then proceeded with the interaction with CarWeaver, which included retrieving requirement data via API (Application Programming Interface) and processing the data to make it comparable with the test results in JSON format. This part was programmed with Python. 21 3. Research Methods When the data generation, retrieval, and process are done, data analysis begins. It compares the JSON files, analyzes the requirement status, and stores the results in a CSV file which will later be visualized. When the status of all requirements of the scope under test is known, the status of the scope can be easily known as well. Python scripts were created during this stage. The final step is to import the CSV result into Power BI to build a visualization of the project status. In the second iteration, the implementation involved running the API utilization code on a different Jenkins node than the one communicating with AutomationDesk. Additional changes were made based on the code review and the focused group discussion. Implementation details will be discussed in the Findings section. 3.2.5 Implementation Evaluation The implementation evaluation stage evaluates if the implemented solutions can solve the problem identified correctly. This phase may overlap with the problem investigation of the next cycle if there is one. In the first iteration, the author evaluated the implementation by conducting a workshop at the case company to demonstrate the solutions. Subsequently, a survey was distributed to gather detailed feedback on the functionality and usability of the solution. This was followed by a focused discussion group to explore the feed- back received from the workshop further. Additionally, an initial code review was conducted to assess the quality of the code and identify any issues. In the second iteration, an evaluation meeting was held with a system engineer to validate the updates and assess how the framework performs in various contexts. # Participant Role Activities Workshop Survey Code Review Focused Discussion Group Evaluation Meeting Person-A System Engineer X X X X Person-B Software Engineer X X X Person-C Software Engineer X X Person-D Integration and Release Engineer X X Person-E Integration and Release Engineer X X Person-F DevOps Engineer X Person-G DevOps Engineer X Person-H Electrical Engineer X X Person-I Electrical Engineer X X Person-J Master’s Student X X Person-K Test Engineer X X Person-L Test Engineer X X Person-M Test Engineer X Person-N Test Engineer X Person-O Software Verification Engineer X Person-P Software Engineer X Person-Q Integration and Release Engineer X Table 3.1: Participants in the evaluation activities of the two cycles. Table 3.1 lists the participants in the evaluation activities across two cycles. Fifteen individuals took part in the workshop, of whom nine responded to the survey. The code review was conducted by the author and a software developer. The focused 22 3. Research Methods discussion group included four stakeholders from the case company. Both the code review and the focused discussion group contributed to the problem investigation in the second cycle. At the end of the second cycle, an evaluation meeting was held with a system engineer. 23 3. Research Methods 24 4 Findings 4.1 Cycle I Findings 4.1.1 Problem Investigation In the first iteration cycle, a problem-driven approach was adopted as outlined in the Research Methods chapter. The initial investigation was based largely on many discussions with an experienced system engineer at the case company, who specializes in software integration and release. These discussions were intentionally unstructured to foster deeper insights. Document reading also strongly supported the problem identification. The case com- pany uses Confluence as one of the workspaces for knowledge sharing and document management. This platform houses extensive information across various teams and departments and is regularly updated, making the investigation process more effi- cient by reducing reliance on time-consuming interviews, workshops, and meetings. Although the author reviewed many documents on Confluence, only those directly related to the thesis study will be described in the subsequent section. A comprehensive literature review was conducted to understand the general back- ground of the thesis study. In the first iteration, the review focused on challenges related to requirement traceability and consistency in the automotive industry. Sev- eral studies suggested tool integration as a solution to these challenges, and it was then explored. The problems identified in this cycle will be elaborated as follows: First, the the- sis motivation and the case team’s operational methods will be discussed. Subse- quently, a thorough analysis was conducted to answer Research Question 1 and Research Question 2, covering how testing is executed in the integration and release processes, the state of traceability before this study, issues with the test simulation tool AutomationDesk, and the current status of visualization. The case team in this thesis study is responsible for integrating and releasing in- verter software. The development of this software adheres to AUTOSAR (Auto- motive Open System ARchitecture), a standardized framework for automotive soft- ware. This architecture is structured into distinct layers, as illustrated in Figure 4.1. The foundational Hardware layer consists of physical elements, including electronic control units (ECUs). Above this, the AUTOSAR Runtime Environment (RTE) 25 4. Findings serves as middleware, supporting the interaction between the application software and the hardware via standardized interfaces. The System Software, also known as basic software, is organized into four functional groups. At the uppermost level, the Application Layer houses the software components that perform specific vehicle functions [8]. Figure 4.1: AUTOSAR-System Software Stack Overview The integration and delivery activities within the case team are based on this archi- tecture. Software components for the application layer are developed separately by different in-house teams within the company. The case team then integrates these components with the runtime environment to build the integration software. The third layer, system software, comes from external suppliers. Once the integration software is assembled, the case team executes important HIL test cases to ensure everything works as a preliminary check, known as a smoke test, before sending the software to the supplier. Upon receiving the integration software, the supplier develops it into fully functional software and sends it back to the case team. This final software undergoes acceptance testing through HIL, carried out by another internal acceptance testing team. The workflow is depicted in Figure 4.2. Although this description simplifies the actual delivery process, it provides enough detail to understand how HIL testing and traceability are applied currently. As indicated in Figure 4.2 and mentioned above, two testing activities happen in the integration and delivery process. 1. The first activity occurs after the integration software is assembled and before it is sent to the external supplier. At this stage, smoke testing is carried out to quickly verify the functionality of the most important features. Only a se- lect few of the defined test cases are performed, as these are deemed the most critical. In some projects, test cases for smoke testing are automated using dSPACE AutomationDesk, whereas in others, they are manually executed us- 26 4. Findings Figure 4.2: Integration and delivery process in the case team. ing dSPACE ControlDesk. The automated tests can also be initiated through the Jenkins pipeline. 2. The second testing activity takes place within the Acceptance Testing team, where acceptance testing is conducted on the fully developed software received from the external supplier. Similar to the smoke testing phase, some test cases for acceptance testing are automated, while others require manual execution. For both smoke and acceptance testing, regardless of how test cases are executed, the results must be manually recorded in an Excel sheet template by the tester. This template, commonly used in the case department across various testing stages, begins with a test overview section. This section includes metadata about the project under test, tester information, and test results. Following the overview is a detailed test specification section, where test cases are listed along with corresponding test steps and outcomes. The specifics of the test cases vary depending on the project under test and the type of testing being conducted. This template is illustrated in Figure 4.3. A limitation of this template is that it focuses exclusively on test execution without including information on related requirements. To establish traceability using this template, a separate Excel sheet is employed. This sheet maps requirements to test cases as specified in CarWeaver, and it is created based on the data exported from CarWeaver. Although this secondary Excel sheet contains multiple columns, the most important ones include ownership of test cases and requirements, test case name, test case ID, requirement ID, requirement handle, testing level, and test outcome. Figure 4.4 shows some of the template’s key headers. This template is stored on Confluence. Testers must download a new local copy each time they intend to use it. During use, the tester needs to filter out the relevant test cases to be executed and enter the verdicts manually. Upon completion, the sheet 27 4. Findings Figure 4.3: Template to record smoke/acceptance testing result. Figure 4.4: Part of headers of the template for traceability building. must be uploaded to a custom web portal designed in-house. This portal processes the data, forwards it to a company-specific message bus, and visualizes the project status in Power BI. Figure 4.5 provides an example of the resulting visualization. Figure 4.5: Visualization dashboard of the project status. Since the traceability template is derived from data exported from CarWeaver, any updates made to specific information within CarWeaver require the template to be regenerated. Currently, this process of version control is handled manually, which poses potential issues such as human error, network disruptions, and other opera- tional challenges. Additionally, the existing method for managing traceability in- volves several manual steps, making it time-consuming and complex. The workflow described above for building traceability and visualization is only suited to manual HIL testing and does not support automated HIL testing. This limitation stems from a lack of appropriate data, such as the information typically found in an Excel template. However, the company is now aiming to increase the 28 4. Findings use of automated test cases. More test cases will be transitioned to AutomationDesk and executed automatically, rather than manually. This shift in testing methodology requires an automated approach to establishing traceability, which can eliminate the need for manual input into Excel templates. Given that similar workflows of building traceability for manual HIL testing are already in use across various departments within the company, and considering the broad impact of increasing automation, developing a new solution for traceability in automated HIL testing could benefit not just the case department but also the whole company. More issues related to inconsistency emerged as the author investigated the practical application of AutomationDesk and CarWeaver. As previously discussed in the Background section, in the context of HIL testing, CarWeaver is used to designing and defining test cases, while their implementation and execution are carried out in dSPACE AutomationDesk. In dSPACE AutomationDesk, each test sequence contains a "Requirement" data object, which specifies the particular requirements it aims to test. Ideally, this data object should contain the requirement ID from CarWeaver. However, test cases are developed by various teams, each focusing on different components, and inconsistencies arise in documenting the requirement information. For some test cases, the Requirement data object correctly matches the requirement values stored in CarWeaver. However, certain data, while initially aligned with CarWeaver’s records, have become outdated; changes made in CarWeaver are not updated in the AutomationDesk test cases. For other test cases, the Requirement data object values do not directly match those found in CarWeaver. Some have relatively clear requirement descriptions that are possible to interpret and locate in CarWeaver, whereas others are only described with less informative text that may be understood only by the developers who created them. This inconsistency in documentation standards makes it difficult for different teams to have a uniform understanding and access to requirement details. Another challenge in tracking requirement information in the data object is that some values are still linked to the old requirement management tool. Since the com- pany transitioned from this old platform to a new one, which is CarWeaver, updates to both test cases and requirements are necessary. Although CarWeaver monitors these changes across the two platforms, AutomationDesk does not recognize them. As a result, the requirement ID linked to a HIL test case in AutomationDesk might not reflect the latest version. Additionally, some test cases refer only to a subset of relevant requirements in the Requirement data object. This lack of consistency in maintaining up-to-date traceability information between AutomationDesk and CarWeaver presents risks for product releases and complicates the testing process, ultimately affecting the efficiency of software releases. Additionally, Merely having a Requirement data object is insufficient to establish effective traceability. Given the many-to-many relationship between test cases and 29 4. Findings requirements, it is not feasible to automatically determine which test case imple- mented in AutomationDesk corresponds to which test case definition in CarWeaver. As discussed above, it is possible to manually trace CarWeaver information in Au- tomationDesk to some extent and vice versa if the Requirement data object is cor- rectly configured. However, this workflow has a significant gap due to the absence of a direct connection between these two software systems. When users need to trace a particular item, they find it tedious and time-consuming because they have to man- ually navigate through CarWeaver or AutomationDesk to locate the corresponding information. This situation is far from ideal, as it affects the efficiency of track- ing and managing project artifacts across these platforms. Moreover, the observed inconsistency between CarWeaver data and AutomationDesk requirement informa- tion cannot be addressed due to the inadequate implementation of traceability for automated HIL testing. The challenge extends to the reporting mechanism in AutomationDesk. The core problem lies in the fact that these reports, serving as artifacts within Jenkins jobs in the CI pipeline as well, do not include vital requirement information. The lack of traceability information in the report makes it tedious to identify if the planned requirements for a release cycle are all validated or not. It also hinders the identifi- cation of requirements of error-prone test cases. Owing to the issues outlined above, the case team, the acceptance testing team, and other departments employing a similar workflow have been unable to adhere to the ASPICE standard for qualification testing, which prioritizes traceability and consistency. Other than the inconsistency issue, many test cases defined in CarWeaver are not implemented in AutomationDesk, meaning they’re either just defined but not im- plemented, or performed manually instead of automatically. This issue was noticed because the author examined a small subset of test cases and their corresponding requirements in CarWeaver, then searched for them in AutomationDesk using either the test case names or the requirement IDs. Apart from the lower implementation rate of automated test cases, other challenges in the current visualization flow for automated testing have been identified. Cur- rently, traceability visualization has been limited to manual testing at the case company. The visualization of requirement traceability in automated testing is not available because there is no suitable data format for recording test execution results. It has been noted that while automated integration and verification test results from some ongoing projects are displayed in Power BI, this visualization is restricted to test verdicts per baseline. Specifically, it only includes the Baseline name, Car- Weaver Test case Test System IDs, Test Case Name, and Test Result. That means there is no traceability between tests and requirements, making it impossible to effectively get the requirement coverage. 30 4. Findings 4.1.2 Solution Design The main problems identified in this iteration are the absence of traceability in automated HIL testing and the mismatch between AutomationDesk test cases and CarWeaver requirements. Data plays an important role in establishing traceability and identifying these mismatches. Moreover, it is an essential resource for all systems involved in traceability and reporting. To address the issues outlined earlier, several key questions must be considered: How is data stored across different software applications? How can this data be re- trieved? And how is data integrated across various software systems? Accordingly, the proposed solution should be data-centric. Since these issues arise from interac- tions among multiple software programs, a standalone code solution that operates independently of external software is not suitable. Inspired by the literature review during the problem investigation phase, tool integration has proven to be a viable approach. Currently, there is no tool or plugin available that directly connects CarWeaver and AutomationDesk. As a result, this thesis cannot rely on any existing support tools for integration. The integration methods from existing research, which depend on specific vendors, do not meet the requirements of this study. Furthermore, the use of Excel, previously discussed, will not be included as this study seeks to establish a more automated traceability process that eliminates the need for manually main- taining Excel sheets. The ALS approach described before is also unsuitable because the software in this study does not comply with the OSLC standard. Therefore, the integration must be accomplished in a more generic manner, which involves reading data from the software. Developing a framework that incorporates the software in use could effectively address these issues. The solution is illustrated in Figure 4.6. It is a requirement-centric framework that can link AutomationDesk and CarWeaver, enabling the establishment of traceability and consistency checks. The core concept of this framework is to extract test results from AutomationDesk and requirements specifications from CarWeaver. The data is then processed to enable comparisons, ultimately generating outputs that reflect the status of both the requirements and the overall project. Eight components developed during this thesis are highlighted in the framework. COMP 1-1 Updated test sequences in AutomationDesk One of the issues identified was the lack of test case ID information from CarWeaver in the AutomationDesk test cases. Furthermore, significant inconsistency exists between the Requirement data object and the actual requirement IDs documented in CarWeaver. To establish traceability, it is crucial to ensure that test cases provide accurate information. This requires updating the Requirement data object and introducing a Testcase data object that records the respective test case ID from CarWeaver. The data objects described above can be viewed as attributes of the test sequence. However, the test results don’t output these data objects by default, which prevents 31 4. Findings Figure 4.6: A framework to build traceability for automated HIL testing and check consistency between AutomationDesk test cases and CarWeaver requirements. their inclusion in the test report and JSON test results. Consequently, the test sequence itself needs to be updated to add a new step that will enable the display of data objects in the test results. COMP 1-2 Test report in AutomationDesk AutomationDesk supports report generation, enabling users to either use the default report template or customize it according to their needs. The case company uses a customized template for generating their reports. Inspired by the existing HIL information tables in the report, the concept of adding another table specifically for requirement traceability was developed. The new traceability table in the report template allows testers to easily view the CarWeaver requirements and test case IDs of the executed tests. The creation of the traceability table should be as automated as possible. Its content includes the test case name in AutomationDesk, the test case ID in CarWeaver, the requirement ID in CarWeaver, and the outcome of the test. The prototype can be found in Figure 4.7. The data presented in the test report are derived from the results produced by AutomationDesk following the test execution. COMP 1-3 JSON test result in AutomationDesk 32 4. Findings Figure 4.7: Prototype of traceability table Although AutomationDesk outputs test results in an XML file, this format includes excessive details that are not required for establishing traceability. In the framework, compatibility and comparability between data from different software are expected. The JSON format is particularly suitable because it is both human-readable and straightforward to process. Therefore, it is necessary to convert the native test results into a JSON file, which only contains information required for building trace- ability. Each entry in the JSON file will be structured as follows: { "test-case": "A dummy test case name", "verdict": "Passed", "carweaver-test-case-id": "TC-123456", "requirement-id": [ "REQ-987654", "REQ-456789" ] } COMP 2 CarWeaver API usage The CarWeaver API, developed internally by the case company, offers a wide range of endpoints to access resources hosted within CarWeaver. The API responses are in JSON format and provide detailed information about each requested resource. The requirements within CarWeaver are organized hierarchically. It is possible to retrieve requirements from various scopes by using the appropriate endpoints. COMP 3-1 JSON requirement specification Although the CarWeaver API responds in JSON format, the amount of information and its structure are not directly suitable for building traceability. Additionally, accessing some information may require nested API calls. Hence, the requirement specifications retrieved through the API need to be reorganized and stored in a separate JSON file. COMP 3-2 Consistency checking Using the JSON files that store test results and requirement specifications, incon- sistencies between two software systems can be conveniently reported by comparing the data with a script. By setting specific conditions within the code, these incon- sistencies can be detected. Examples of potential inconsistencies include: • The Requirement data object of a test sequence in AutomationDesk contains 33 4. Findings only a portion of the requirements defined in CarWeaver. • In AutomationDesk, the mapping relationship between the Testcase data ob- ject and the Requirement data object doesn’t align with the definition in CarWeaver. • Either the Testcase or Requirement data object doesn’t exist in a specific scope in CarWeaver. • Some test cases that should be executed according to the requirement specifi- cation in CarWeaver are not included in AutomationDesk. • Duplicated test cases are executed in AutomationDesk. COMP 3-3 Project status (CSV) In CarWeaver, requirements for various scopes are clearly defined. Each requirement can be linked to multiple test cases. To thoroughly validate a requirement, it is essential to cover all associated test cases during the test run. Executing HIL test sequences that belong to the same requirement scope allows for the analysis of the requirement status. By understanding the status of all requirements within a specific scope, such as a project, it is possible to determine the overall status of that scope. Requirement statuses are divided into four categories: fully validated and passed, fully validated but failed, partly validated, and not tested. • Fully Validated and Passed: This status indicates that for a requirement de- fined in CarWeaver, all connected test cases have been covered in the test run and have passed successfully. • Fully Validated but Failed: This status applies when all test cases connected to a requirement in CarWeaver are covered in the test run, but at least one test case has failed. • Partly Validated: This status means that only some of the test cases connected to the requirement in CarWeaver have been covered in the test run. • Not Tested: This status is assigned when none of the test cases connected to the requirement have been tested. Since requirement information is managed in CarWeaver, and HIL testing is con- ducted using AutomationDesk, integrating these tools is a key step. To analyze the requirement status, it is important to prepare and harmonize data from both software systems. This integrated data can then be used to assess and compare the fulfillment of project requirements. By analyzing the JSON data of test results and requirement specifications, a CSV file can be created to display the requirement statuses for a specific scope. This file also helps determine the overall requirement status of the scope. COMP 4 Data visualization in Power BI The CSV file containing the requirement statuses can be visualized using software 34 4. Findings like Power BI. Various diagrams like a pie chart can be created to provide a user- friendly view of the data, such as showing the coverage of requirements. 4.1.3 Design Validation The three validation questions proposed by Wieringa are answered in this iteration. • Internal validity: The designed framework supports the creation of requirement- centric traceability and helps in the identification of inconsistencies. When the framework is used, the HIL testing report will include a traceability table. Python scripts will be used to process and analyze data from test results and requirements, eliminating the need for manual data entry and maintenance in Excel sheets. This ensures robust traceability and consistency checks for automated HIL testing, addressing the issues identified in the first iteration cycle. • Trade-offs: The framework will still operate with minor variations. For in- stance, although JSON is chosen for structuring test results and CarWeaver requirements, other formats like XML and YAML are also suitable due to their capability to organize data effectively. This adjustment affects compo- nents COMP 1-3 and COMP 3-1, as modifications to the code are required to support file creation in these formats. For the storage of test results and requirements, using a centralized database that supports a search mechanism could also support the frameworks functionality. In such a scenario, data in- teractions would occur between the processing center and the database rather than separately from CarWeaver and AutomationDesk. The specific imple- mentation would depend on the chosen database. Furthermore, visualization tools are not confined to Power BI; alternative dashboard applications can be employed, although the procedures to create graphs will differ based on the software used. • External validity: The framework does not depend heavily on specific ex- ternal vendors, allowing for wide applicability across various contexts. Even if CarWeaver is replaced with another requirements management platform, or if different software is used for testing instead of AutomationDesk, the frame- work remains functional as long as the new software supports data exportation. This thesis uses an API to retrieve data from CarWeaver, but alternative meth- ods such as direct database access or web scraping are also viable. The core concept of the framework involves data preparation, transportation, processing, analysis, and visualization. Therefore, as long as valid data is available, the framework can be adopted. This versatility makes it highly adaptable for companies that employ different workflows and software for test- ing and requirements management. 4.1.4 Solution Implementation The framework is implemented by assembling the defined components within it. 35 4. Findings In the thesis study, the implementation started with updating test sequences in AutomationDesk and adding a new traceability table in the test report. A deep investigation was performed to understand how the report generation works in the existing workflow. In AutomationDesk, a report can be generated automatically after a sequence has been executed, or from an existing result, meaning that a report is always based on a result. AutomationDesk uses a standardized XSD file as a schema for test results. The result data is stored in XML format post-execution and conforms to the XSD schema. Without this XML data, a report that includes the execution results cannot be created. An investigation of the XSD file used by AutomationDesk for execution results revealed that the key elements supporting customized report generation are called "blocks". Therefore, as long as the data is included in the test result, it can be retrieved, parsed, and placed in the final traceability table. AutomationDesk offers two default formats for presenting reports: PDF and HTML. Users can also create their style sheets for the reports by writing XSL files per the XSD schema. The mechanism to generate the test report is illustrated in Figure 4.8. The case department has developed a customized style sheet, consisting of a series of XSL files, to enhance the presentation of the report. Compared to the standard formats, these customized reports provide a more user-friendly display of informa- tion, featuring a glanceable header section. This section includes additional data such as the system under test, a summary of the test case, and information about the test object. For ease of use and quick reference, it was decided to place the desired traceability table in the header section. COMP 1-1 Updated test sequences in AutomationDesk Data objects can be treated as attributes of the test sequences and are not auto- matically displayed in the test report or results. To have them included in the test results, a block that contains the data object information must be developed and explicitly added to the test sequence. The thesis does not undertake the actual updating of existing test cases, as this requires relevant metadata, typically only known by the test case developers. The test case developers need to perform the updating activities by themselves. As discussed in the solution design phase, the updates that should be made to the test sequences are outlined as follows • Update Requirement data object Test case developers should find the respective requirement ID in CarWeaver. Afterward, by selecting the data object in AutomationDesk and entering the requirement ID, they can modify the value. • Add Testcase data object Similar to the first update, test case developers should navigate in CarWeaver 36 4. Findings Figure 4.8: An explanation of how the test report is generated in dSPACE Automa- tionDesk. to retrieve the relevant test case ID. Subsequently, in AutomationDesk, they need to create a new data object under the test case and input the acquired test case ID. Its important to note the distinction between the test case ID in CarWeaver and the test case ID generated by AutomationDesk. In AutomationDesk, test cases are saved in blkx files, which are XML files containing definitions of saved or exported AutomationDesk elements such as sequences. Each test case stored in a blkx file is assigned a unique ID automatically by AutomationDesk. However, this ID is unrelated to the test case ID defined in CarWeaver and cannot be used for establishing traceability. • Design a new block for data object display and add it in the sequence The author explored various block types available in AutomationDesk to deter- mine the most automated and efficient solution. A summary of these findings is presented in Table 4.1. Since the requirement ID is a data object of the test sequence, it’s logical 37 4. Findings # Attempt Block Type Adopted 1 AddDataObjects No 2 AddText No 3 Exec Yes Table 4.1: Various blocks that were explored during implementation phase. to drag a data object block called AddDataObjects from the built-in Report library to the sequence flow. This block acts as a container for a data object and can show its information. To connect it to the corresponding requirement, a new data object should be created within this block, and its reference field should be linked to the requirement data object in the sequence. In this way, the requirement information is integrated into the automation sequence and should also appear in the report. Following the execution of the updated sequence, the report successfully displayed the requirement ID. However, it was presented in a table format, which is not ideal for a single value and complicates the extraction of the requirement ID for the traceability table. An example of this table format, showing only the requirement information, is depicted in Figure 4.9. Figure 4.9: The table automatically inserted into the test report after adding a data object block in the automation sequence. The block is connected to the requirement data object of the test sequence. Inspired by the data object block, the author considered replacing the block with another type of block, which is the AddText block. It’s also a built-in block type provided by AutomatinDesk itself. It’s a container of several data objects. By default, it will print out in the test report the string specified by users if it’s included in the sequence. After dragging the AddText block into the test case sequence, it’s necessary to edit the value field to include requirement information. The author modified the field to "CarWeaver Requirement: example require- ment id" and executed the test case. As anticipated, the report correctly displayed the added text. However, in practical scenarios, testers must man- ually enter "CarWeaver Requirement: " followed by the value of the related Requirement data object. This manual entry process can lead to errors, as any typographical mistake or omission in "CarWeaver Requirement: " could cause parsing failures. A more automated procedure should be developed to connect and enter data automatically, reducing the need for manual edits by testers and lowering the risk of errors. The optimal solution for maximizing automation is to use the Exec block, which works like a script by embedding Python code. AutomationDesk in- 38 4. Findings cludes a built-in Python environment and provides various Python libraries. These tools help execute code within the Exec block and integrate it with test- ing processes. In this iteration, the code retrieves data from the Requirement and Testcase data objects of a test sequence and logs this information. The details are then displayed in the XML test result. This block can be inserted at any point in the test sequence, though placing it at the beginning is often most convenient. COMP 1-2 Test report in AutomationDesk A traceability table in the test report can offer a clear and concise overview that al- lows testers to easily see the relationships between test cases, requirements, and test verdicts. Minimizing the effort required to update test cases will decrease manual labor and lower the likelihood of errors for testers. The software testing process generally follows a sequence of steps, beginning with test definition, followed by test execution, and concluding with test analysis. In the case company, when the predefined HIL test cases are executed, an XML file known as ReportPool.xml is created. This file is a comprehensive repository of the test results, which includes details such as runtime logs, metadata related to test cases, the verdicts of the tests, and so on. The ReportPool.xml is integral to the testing process as it serves both as the output from the test execution phase and the input for the test analysis phase. Based on the information contained in this XML file, a customized test report is generated, providing a tailored overview of the testing outcomes. To incorporate a traceability table into the test report, reverse engineering tech- niques were employed to devise a method for its generation. The placement of this table in the test report, derived from the results contained in the ReportPool.xml file, necessitates that relevant data be included as part of this file. The data encap- sulated within the ReportPool.xml originates from the execution phase and directly references the test sequences under test. Therefore, including traceability information in the test results is only feasible if the test sequences are configured to support this functionality. This realization led to the proposal of introducing a new block within each test sequence that records the requirement ID and test case ID as part of the test results. Given that the existing test sequence definitions already incorporate a data object named Requirement, it is straightforward to introduce an additional data object named Testcase. This new object acts as an attribute and represents the test case ID defined in CarWeaver, which corresponds to the AutomationDesk test sequence. For the XSL processing to accurately identify and retrieve the elements that store the requirement ID and test case ID within the test results, it is imperative that the logging details inputted into the ReportPool.xml are distinctive. The imple- mented solution involves specifically logging entries such as "CarWeaver Test Case ID: example-test-case-id" and "CarWeaver Requirement ID: example-requirement- id". 39 4. Findings When the test result was prepared, the author proceeded to define the elements of the table and the styling attributes within the XSL file. The Requirement ID and test case ID are extracted from the Reportpool.xml file and then recorded in the report with the help of the XSL file. This setup can generate the traceability table in the report, which is a visualization of the link between test cases, requirements, and their respective test results. COMP 1-3 JSON test result in AutomationDesk Although test results are accessible in the ReportPool.xml file and are detailed further in a test report for better visibility, they lack data that is directly usable for status analysis. Therefore, it is necessary to develop a method to produce data ready for analysis, such as JSON. This would involve converting the test results from AutomationDesk into JSON format, which supports requirement status assessments. The JSON data should include essential elements like the test case name, test case ID in CarWeaver, requirement ID in CarWeaver, and the test verdict. There are two methods to initiate HIL testing in AutomationDesk. The first method is to manually execute the test sequences directly within AutomationDesk. The second method uses a Jenkins pipeline to trigger testing on a HIL machine by calling the AutomationDesk API. In both scenarios, the execution results must be generated as JSON data, regardless of the testing trigger method. When testers manually initiate testing directly in AutomationDesk, the exportation of JSON data can be implemented through a GUI tool. This tool reads the latest test results and generates a JSON file. The design choice of integrating a GUI tool within AutomationDesk, accessible via a simple click rather than through script execution, is intended to streamline the process. This approach simplifies the operation by eliminating the need for explicit script invocation. The GUI tool was developed primarily using the Tkinter library in Python. When a user clicks the GUI icon within AutomationDesk, it triggers a Python script. This script defines a class to identify the active project in AutomationDesk and access the default fold