A Framework for Verification and Validation of Simulation Models in esmini With an Example of Framework Application Master’s thesis in Computer science and engineering Daniel Olsson, Eric Rylander Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY UNIVERSITY OF GOTHENBURG Gothenburg, Sweden 2022 Master’s thesis 2022 A Framework for Verification and Validation of Simulation Models in esmini With an Example of Framework Application Daniel Olsson, Eric Rylander Department of Computer Science and Engineering Chalmers University of Technology University of Gothenburg Gothenburg, Sweden 2022 A Framework for Verification and Validation of Simulation Models in esmini With an Example of Framework Application Daniel Olsson, Eric Rylander © Daniel Olsson, Eric Rylander, 2022. Supervisor: Sam Jobara, Department of Computer Science and Engineering Advisor: Mikael Andersson, Volvo Car Group Examiner: Robert Feldt, Department of Computer Science and Engineering Master’s Thesis 2022 Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg SE-412 96 Gothenburg Telephone +46 31 772 1000 Cover: A simplified flowchart displaying the primary steps of the framework. Typeset in LATEX Gothenburg, Sweden 2022 iv A Framework for Verification and Validation of Simulation Models in esmini With an Example of Framework Application Daniel Olsson, Eric Rylander Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Abstract As the use of simulations is becoming more prominent in the industry, the need for verified and validated simulation models becomes more important. For the simula- tor esmini, there are no concrete methods or techniques for the V&V process, which this thesis aims to unravel. This thesis presents a verification and validation (V&V) framework for the traffic simulator esmini, with the purpose of providing a founda- tion for the V&V process and contribute to the research field of simulation valida- tion. The framework was developed by performing desk research on existing V&V methodologies and combining found techniques with software engineering concepts such as quality attribute scenarios (QAS) and acceptance test-driven development (ATDD). The research was done in collaboration with Volvo Cars. Simultaneously, a lane independent routing model for esmini was developed to solve a case provided by Volvo Cars, to which the framework was applied in order to provide an example of how the framework could be applied to a simulation model and to evaluate the frame- work in terms of usefulness and effectiveness. The resulting framework consisted of six iterative steps: creating an assumptions document, defining requirements and acceptance tests, developing the model, testing and refactoring, reconnecting with stakeholders, and performing empirical and visual validation. The fundamental con- cept of the framework is to perform the steps iteratively, in order to continuously improve the simulation model. The evaluation of the framework was primarily done through fault injection and developer feedback from Volvo Cars which were encour- aging. Overall, the conclusion is that the framework provides necessary guidance for the V&V process in esmini and that it could be used for V&V in other similar simulators. Keywords: V&V, Verification, Validation, Framework, esmini, OpenDRIVE, Open- SCENARIO, Simulation models v Acknowledgements We would like to thank our supervisor, Sam Jobara, for providing support and feedback throughout the thesis. We would also like to thank our advisor at Volvo Cars, Mikael Andersson, for taking the time to help us understand the case and esmini, and providing helpful insights on our thesis. We would also like to thank Emil Knabe, the main developer on esmini, for helping us to locate and fix bugs in our model. Daniel Olsson & Eric Rylander, Gothenburg, June 2022 vii Acronyms ABS Agent-Based Simulation API Application Programming Interface ASAM Association for Standardization of Automation and Measuring System ATDD Acceptance Test-Driven Development CQAS Concrete Quality Attribute Scenario DES Discrete-Event Simulation DSRP Design Science Research Process esmini Environment Simulator Minimalistic LIRM Lane Independent Routing Model MBT Model-Based Testing OEI Observable emerging information PBT Property-Based Testing QAS Quality Attribute Scenario V&V Verification and Validation ix x Contents List of Figures xv List of Tables xvii 1 Introduction 1 1.1 Statement of the Problem . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Purpose and Significance of Study . . . . . . . . . . . . . . . . . . . . 2 1.3 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 Limitations and delimitations . . . . . . . . . . . . . . . . . . . . . . 3 1.5 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 Theory 5 2.1 Verification and validation in regard to simulation models . . . . . . . 5 2.2 Existing verification and validation methodologies . . . . . . . . . . . 6 2.3 Modeling and Simulation techniques . . . . . . . . . . . . . . . . . . 11 2.3.1 Simulation techniques . . . . . . . . . . . . . . . . . . . . . . 11 2.3.2 Traffic modeling techniques . . . . . . . . . . . . . . . . . . . 12 2.4 ASAM Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4.1 OpenDRIVE . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4.2 OpenSCENARIO . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5 Environment Simulator Minimalistic (esmini) . . . . . . . . . . . . . 16 2.5.1 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5.2 Event and Action . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.5.3 Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.5.4 Similar simulators . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.6 Quality Attribute Scenarios . . . . . . . . . . . . . . . . . . . . . . . 19 2.7 Acceptance Test-Driven Development . . . . . . . . . . . . . . . . . . 20 2.8 Test automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3 Methods 25 3.1 Design Science Research Process . . . . . . . . . . . . . . . . . . . . . 25 3.2 Research methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3 The creation of the framework . . . . . . . . . . . . . . . . . . . . . . 27 3.3.1 Development process . . . . . . . . . . . . . . . . . . . . . . . 27 3.3.2 Motivation for chosen procedures . . . . . . . . . . . . . . . . 28 3.3.3 The application of the framework . . . . . . . . . . . . . . . . 29 3.4 The development of a lane independent routing model (LIRM) . . . . 29 xi Contents 3.4.1 Model criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.4.2 Model design . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.4.3 Model development . . . . . . . . . . . . . . . . . . . . . . . . 30 4 Framework 33 4.1 Assumptions Document . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.1.1 Model functionality . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.2 Model flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.3 Model subsystems . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.4 Simplifying assumptions . . . . . . . . . . . . . . . . . . . . . 35 4.1.5 Model limitations . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.1.6 Model input data summary . . . . . . . . . . . . . . . . . . . 36 4.1.7 Important sources of information . . . . . . . . . . . . . . . . 36 4.1.8 Acceptance criteria . . . . . . . . . . . . . . . . . . . . . . . . 36 4.1.9 Assumption validation . . . . . . . . . . . . . . . . . . . . . . 36 4.2 Requirements, Concrete QAS and Acceptance Tests . . . . . . . . . . 36 4.2.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.2.2 Concrete Quality Attribute Scenario . . . . . . . . . . . . . . 37 4.2.3 Acceptance test . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.3 Model development . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4 Calibration, Verification, and Validation . . . . . . . . . . . . . . . . 38 4.4.1 Iterative process . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4.2 Create unit tests . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.4.3 Run tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.4.4 Perform refactoring and calibration . . . . . . . . . . . . . . . 39 4.4.5 Reconnect with stakeholders . . . . . . . . . . . . . . . . . . . 39 4.4.6 Empirical and visual validation . . . . . . . . . . . . . . . . . 39 5 Framework Application Example 41 5.1 Assumptions document . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.1 Model functionality . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2 Model flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.3 Model subsystems . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.4 Simplifying assumptions . . . . . . . . . . . . . . . . . . . . . 42 5.1.5 Model limitations . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.1.6 Model input data summary . . . . . . . . . . . . . . . . . . . 43 5.1.7 Important sources for information . . . . . . . . . . . . . . . . 44 5.1.8 Acceptance criteria . . . . . . . . . . . . . . . . . . . . . . . . 44 5.1.9 Assumption validation . . . . . . . . . . . . . . . . . . . . . . 44 5.2 Requirements, CQAS and Acceptance Tests . . . . . . . . . . . . . . 44 5.2.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.2.2 Concrete quality attribute scenario . . . . . . . . . . . . . . . 45 5.2.3 Acceptance tests . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.3 Model development . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.4 Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.4.1 First Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.4.2 Second Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 49 xii Contents 5.4.3 Final Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6 Results 59 6.1 Developing a framework . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.2 Evaluation of framework . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.2.1 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.2.2 Simulated experiment . . . . . . . . . . . . . . . . . . . . . . . 60 6.2.3 Usefulness and effectiveness . . . . . . . . . . . . . . . . . . . 63 7 Discussion 65 7.1 The developed model, LIRM . . . . . . . . . . . . . . . . . . . . . . . 65 7.2 The framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 7.3 Uniqueness of framework . . . . . . . . . . . . . . . . . . . . . . . . . 67 7.4 Threats to validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7.5 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 7.6 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 8 Conclusion 71 A Lane Independent Router Case from Volvo I B All Requirements and CQASs III xiii Contents xiv List of Figures 1.1 Flowchart of research questions. . . . . . . . . . . . . . . . . . . . . . 4 2.1 Example of OpenDRIVE reference line [26] © ASAM e.V.(Used with permission) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Example of OpenDRIVE common junction [27] © ASAM e.V. (Used with permission) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Example of OpenDRIVE direct junction [27] © ASAM e.V. (Used with permission) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4 UML diagram of acts and all related classes in esmini. . . . . . . . . . 17 2.5 Example of a general QAS for the quality attribute availability. . . . 19 2.6 Example of a concrete QAS for the quality attribute availability. . . . 20 2.7 Flowchart of a typical development workflow . . . . . . . . . . . . . . 20 2.8 Flowchart of acceptance test-driven development workflow . . . . . . 20 2.9 The ATDD cycle visualized as a graph . . . . . . . . . . . . . . . . . 21 2.10 The ATDD cycle graph in a more detailed visualization . . . . . . . . 22 3.1 The six steps of the Design Science Research Process. . . . . . . . . . 26 3.2 A flowchart of the logic in the lane change controller. . . . . . . . . . 32 4.1 Flowchart of the V&V framework . . . . . . . . . . . . . . . . . . . . 34 5.1 A simplified flowchart of the model for the case. . . . . . . . . . . . . 42 5.2 An overview of the road network used for the acceptance tests 1&2. . 47 5.3 An overview of the road network used for the acceptance test 3. . . . 48 5.4 Calculated and expected path for route 2, using route strategy: shortest. 53 5.5 Calculated and expected path for route 3, using route strategy: fastest. 54 5.6 Calculated and expected path for route 4, using route strategy: least intersections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.7 Graphs of lateral and longitudinal positions from expected data re- garding a lane change. The upper graph displays relative lateral po- sition (m) over time (s). The bottom graph displays relative longitu- dinal position (m) over time (s). . . . . . . . . . . . . . . . . . . . . . 56 5.8 Graphs of lateral and longitudinal positions from calculated data in esmini. The upper graph displays relative lateral position (m) over time (s). The bottom graph displays relative longitudinal position (m) over time (s). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 xv List of Figures 6.1 Graphs of lateral and longitudinal positions from calculated data in esmini after the injected fault. The upper graph displays relative lateral position (m) over time (s). The bottom graph displays relative longitudinal position (m) over time (s). . . . . . . . . . . . . . . . . . 62 xvi List of Tables 4.1 Table for defining a Concrete Quality Attribute Scenario. . . . . . . . 37 5.1 A selection of requirement for the developed model. . . . . . . . . . . 45 5.2 CQAS for requirement 1 . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.3 CQAS for requirement 2 . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.4 CQAS for requirement 3 . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.5 The test results from running the acceptance and unit tests during the first iteration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 5.6 The test results from running the acceptance and unit tests during the second iteration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.7 All requirements for the developed model in the final iteration. . . . . 51 5.8 Start and target positions of the routes used in validation, written in the format (road-ID, lane-ID, S-position). . . . . . . . . . . . . . . . . 53 6.1 A description of the injected faults. . . . . . . . . . . . . . . . . . . . 61 xvii List of Tables xviii 1 Introduction Verification and validation (V&V) are two procedures that software engineers should use to increase the correctness, accuracy, and reliability of developed software prod- ucts. Verification is the procedure to determine if a developed software represents the conceptual design and specifications [1], [2], while validation is the procedure to determine if the behavior of the developed software fulfills a set of criteria for the intended use [2]. In other words, verification answers the question “have we built the software right?” while validation answers the question “have we built the right software?” [2]. V&V are even more important for simulation models, as these procedures should be used to determine the correctness of the simulated models’ data compared to a realistic scenario. The aim of the study is to create a framework for applying V&V to simulation models developed in Environment Simulator Minimalistic (esmini) at Volvo Cars. The framework should be presented as an informal process containing a set of sys- tematic procedures applied in given steps. To create the framework, the plan is to define new methods, but also take ideas or advantages from existing framework tech- niques to improve on our framework. It should define the methods needed to assess both verification and validation of a simulation model. The framework is intended to be tested during the study by using it for the V&V process on a newly developed simulation model, in order to evaluate the framework’s usability and effectiveness. The goal of the developed framework is to provide a set of procedures for V&V of future models, as well as identify problems with current simulation models. 1.1 Statement of the Problem Verification and Validation: From what was found, there were no set frameworks for V&V in regard to the development of models for esmini, this could be seen in previous master thesis works [3], [4]. One of the theses [3] approaches validation of the developed simulation model by scenario testing, which states a number of given scenarios and discusses the behavior of the model in these scenarios. While the other thesis [4] briefly mentions validation of the model, no steps are given. To ensure that the future development of simulation models for esmini works as intended, a framework for V&V would be beneficial. The found V&V frameworks and methodologies can not be directly applied to esmini as they are too general and loosely defined. Therefore, there is a need for a frame- 1 1. Introduction work in the field of V&V that fits esmini and can ensure that proper models are developed. Simulation modeling: The area of computer simulation is related to modeling [5, p. 920], where a mathematical model is used by a simulation. Another definition for computer simulation is that executed computer code provides solutions to the equations of a model [5, p. 39]. Simulations enable computers to mimic the dynam- ical behavior of a real system under specified conditions. Sargent [6] emphasizes that a simulation model should have a distinct purpose in order for validation to be performed with regard to that purpose. Without a properly defined purpose of the simulation, the correctness and reliability of the targeted system will not be applicable for V&V. Furthermore, the absence of V&V in a simulation will result in insignificant outcomes achieved from the simulation models. The importance of simulation is becoming more prominent and has become a com- mon part of the testing process for various technical fields [7]. Simulation models need to be verified and validated in order to find model errors and reach the desired correctness and reliability. Traffic simulation: Simulation of traffic scenarios require V&V to resemble real- life scenarios as close as possible or to test simplified scenarios such as those used by Euro NCAP [8]. Certification agencies and government authorities work together on developing maneuver descriptions to be used for testing, validation, and certification of vehicle safety systems and autonomous vehicles [9]. The simulation model’s rep- resentativeness of real traffic scenarios is essential in order to apply results captured by the simulation in a similar traffic scenario with real environments and vehicles. Moreover, the importance of effective and efficient traffic simulation is connected to the safety of vehicles and pedestrians. An adequate V&V process used to evaluate a traffic scenario can benefit the safety of all actors involved. The Swedish National Road and Transport Research Institute (VTI) conducts research on human behavior in transport systems [10]. They use an in-house custom simulator environment to recreate realistic driving experiences, which makes it possible to perform traffic ex- periments and study various factors and their effects. Demonstrably, simulators are tools to improve traffic safety with respect to technologies, vehicle characteristics, road design and traffic environments among other things. 1.2 Purpose and Significance of Study The purpose of this study is to create a framework for V&V for simulation modeling in esmini, in order to create a more accurate and reliable V&V process. In this case, the simulation models define road maneuver behavior inside the simulator. The framework is intended to benefit both researchers and practitioners that are working on traffic simulation modeling. The thesis aims to improve existing modeling and simulation V&V by introduc- ing a V&V framework for simulation model development in the simulator esmini. 2 1. Introduction The goal is that the developed framework can be used as a foundation for future V&V process development for esmini. It should also help improve simulation models and therefore, as mentioned in the statement of problems, help improve public safety through the increase of accurate simulation models. It will provide new aspects to the research on simulation modeling V&V. 1.3 Research Questions The connections and steps between the research questions are visualized in a flowchart diagram in Figure 1.1. RQ1: Develop and propose an optimized Validation & Verification framework for simulation models with the inclusion of lane changing scenarios for the esmini sim- ulator. • What steps are needed to create a V&V framework? • What procedures are used in the framework? Use knowledge from existing V&V frameworks and knowledge gathered from current simulation models in esmini to find and create procedures for the new framework. The developed V&V framework will consist of a set of systematic procedures and rule-based decisions, integrated to fulfill the functions of the new V&V framework. RQ2: How can the created framework be applied to a simulation model containing road maneuver scenarios from the standard ASAM OpenSCENARIO? The applica- tion will be evaluated based on the terms of usefulness and effectiveness. Document the application of the new framework on a developed simulation model containing road maneuver scenarios, and evaluate the framework based on a set of metrics measured during the application. The framework will be evaluated by per- forming simulated experiments, in the form of fault injections and removal of parts of the framework. Faults will be injected into the simulation model to verify that the faults are detected by some step of the framework. By removing parts of the framework, the usefulness and impact of the excluded steps will be presented. 1.4 Limitations and delimitations The main limitations of the thesis are the time limit of approximately 5 months and that the effort needed to be adapted to what is possible for two people to complete during the duration of the thesis. Another important limitation of the framework that is to be developed is that it can not be considered as a guarantee for developing flawless simulation models, however it will improve the developed models with the improved V&V process. To delimit the ASAM standards that were used in this thesis, only the most current versions of OpenSCENARIO (1.1.1) and OpenDRIVE (1.7.0) were considered. This 3 1. Introduction Figure 1.1: Flowchart of research questions. delimitation was added to reduce the confusion and conflicting information that could occur if several versions were used, but also to limit the complexity of the development of both the framework and the model. Conduction of a complete liter- ature review of known simulation model V&V methodologies and frameworks was not deemed feasible as it would take too much time, resulting in that the process of creating the framework and the application of it and model development might not have been completed in time. The choice of applying the framework on a singular model was as a delimitation due to the time limit of the thesis. 1.5 Thesis outline The thesis is organized as follows: Chapter 2 provides an overview of the theoretical background and related works. In Chapter 3, the methods used in the thesis are presented and motivated. The developed framework is described in Chapter 4, while Chapter 5 provides an example of how the developed framework can be used on a model in esmini. Chapter 6 presents the results in regard to the research questions. The discussion is presented in Chapter 7 and conclusion can be seen in Chapter 8. 4 2 Theory This chapter will present the related works, and introduce the needed theoretical background. Sections 2.1 and 2.2 present and summarize the related works, while the succeeding sections introduce important theory and concepts for the understanding of the thesis and framework. 2.1 Verification and validation in regard to simu- lation models V&V is applied to the software process to ensure realistic and reasonable solutions [1]. This is increasingly important for simulation models, as users and individuals affected by the model need to know that the results are correct [6]. To validate a model, there should be a specific purpose defined so that the model can be vali- dated with respect to the purpose. Riedmaier et al. [7] state that verification and validation must be performed on simulation models to assess the models’ inherent errors and accuracy. Accurate models of reality have become the decisive factor for simulation credibility and trustworthiness. Sargent [6] discusses and proposes different approaches for V&V of simulation mod- els, where the different approaches have takes on who and how the V&V process should be performed. The first approach that Sargent discusses is that the devel- opment team responsible for the model gets to decide if the model has reached a valid state, through testing and evaluations made during the developing process. However, Sargent states that this approach usually should be avoided if possible. Instead, Sargent proposes that if the team size is small, it is better to involve the model users in the validation of a model. This approach moves the focus of deter- mining validity from the developer to the user, which helps with model credibility. The third approach is the use of independent verification and validation, where a third party is used to decide if the model is valid. This approach can be done during different stages, which Sargent discusses, where the two main stages are concurrent with the development or after the development. This approach is also supported by the International Software Testing Qualifications Board (ISTQB) [11], as they state that testing should be performed at multiple levels where a part of those are performed by independent testers or test organizations. The last approach Sargent presents is through scoring, where a model is given scores based on the outcome of the validation steps performed, at the end of the validation the model’s score deter- mines if the model should be seen as valid. Sargent states that he does not believe 5 2. Theory that this is an approach that should be used due to several reasons, one being that a model could still pass even if it is flawed in some way. Sargent [6] also discusses four concepts that influence different phases through the validation process: data validity, conceptual model validity, computerized model verification and operational validity. Data validity is often not considered to be a part of model validation, as it is often difficult and time-consuming to obtain ap- propriate and accurate data. However, the data is needed for three main purposes, which are building the conceptual model, validating the model, and performing ex- periments. According to Sargent [6], building and validating the model are usually the reasons that data validity is an important part of model validation. Conceptual model validation is done on the conceptual model, where the theories and assump- tions made for the model are determined to be represented correctly. The model’s structure, logic, mathematics and causal relationship are evaluated based on if they are reasonable for the purpose of the model. Computerized model verification is done to ensure that the programming and implementation of the model are cor- rect, the main purpose of this step is to ensure that the implementation is tested and deemed error-free. The last phase, operational validity, determines the validity of the simulation model’s output. The output is evaluated based on the accuracy required for the intended purpose in the domain where the model is applied. It is during this phase that most of the validation and evaluation of the model takes place [6]. MITRE [2] discusses how the V&V process can be extended within certain safety critical fields, into the VV&A process. The extension is called accreditation, which is the process of giving simulation models an official certification that shows if the simulation models’ data can be regarded as acceptable for use within a specific purpose. 2.2 Existing verification and validation method- ologies Sargent [6] presents several V&V techniques that can be used to validate simula- tion models, together with a recommended minimum procedure for validation of a simulation model. The procedure is described in eight steps, but due to that these steps should be seen as the bare minimum according to Sargent, the procedure can be deemed too generalizable to be seen as a useful framework. However, some of the validation techniques that Sargent [6] discusses can be of interest for the goal of this thesis and these techniques are listed below: • Animation • Comparison to other models • Event validity • Face validity • Extreme Condition Tests • Traces 6 2. Theory A proposed framework for optimizing V&V of simulation models has been created by Roungas et al. [12]. They present a framework under the assumptions that sim- ulations, real-world systems, methods, and techniques are dependent on the circum- stances of the study performed. Furthermore, a table of suggested V&V methods are provided, with descriptions of each method’s suitability concerning source code accessibility, availability of real data, game V&V study suitability, types of require- ments and purpose of the study. The framework incorporates the four validation process phases introduced by Sargent [6]. In addition, the framework contains a list of suitable statistical techniques to validate or verify various kinds of datasets. The V&V method selection methodology proposed by Roungas et al. [12] utilizes the aforementioned model properties and characteristics to determine appropriate V&V methods in regard to the validation and verification purposes. Barceló [13] introduces validation on models as an iterative process that is used to calibrate the model parameters and compare the model to actual system behav- ior. The seen differences are used to gain insight on how to improve the model until an acceptable accuracy has been reached. The calibration process performed during the validation has the objective to find the values for parameters that produce a valid model. Therefore, according to Barceló, validation is concerned about if the simulation model is an accurate representation of a given system. He also discusses a basic methodological approach for validation of simulation models, assuming that the input data can be properly modeled and that the measured data that the simu- lation output is compared to is assumed to be error-free. He continues on to discuss how the validation process consists of collecting simulated data and comparing it to measured data, to determine if the simulated model is valid or not. If it is consid- ered non-valid, the model needs to be revised and revalidated, thus implying that validation is an iterative process that may need to be performed several times as the model changes. To help with the validation process, Barceló presents and discusses a four-step pro- cess, that can be used iteratively after the first step has been satisfied. Steps 2-4 can be used iterative until an acceptance accuracy of the model is met[13]. The four steps Barceló discusses are: 1. Error checking 2. Capacity calibration 3. Route choice calibration 4. Performance validation The four steps can be explained in the following terms. Error checking is the pro- cess to eliminate as many errors as possible in the model, to ensure that the model does not create inconsistencies which could create problems that should not exist. Capacity calibration is the process of determining the most appropriate and valid parameters for core functions in the model. Route choice calibration is the process to calibrate a model’s routes if routing exists in the model. The last step, performance validation, is performed on the entire model, where performance measurements such as travel time are compared to existing measures. 7 2. Theory Barceló [14] highlights that the validation of simulation starts at the verification level. He brings up the microscopic traffic simulator called SUMO, which is a sim- ulator similar to esmini, as an example of a computer application that verifies its models at different levels. The mentioned levels are: unit tests, acceptance tests and model tests. Unit tests are the smallest test entity to verify smaller, individual functions of a software. Each function’s expected behavior and output is compared to the actual results given a certain set of input parameters. Acceptance tests com- pare the output of the entire simulation model to the expected results. They are more difficult to implement as the testing scenarios can become complex, but they are more suitable for verifying the overall behavior of the model. Model tests on the other hand are performed by comparing a model to other similar models. Law [15] introduces a tutorial that demonstrates techniques for how to build valid and credible simulation models. He states that a model can be considered valid if it can be used in decision-making scenarios resembling those that would occur if experimenting with the real system was sensible and cost-effective to an equal degree. Since a simulation model is an approximation of a real system, the validity of a model can never be absolute and depends on the time and money investments during the model development. In addition, a model should be developed for a spe- cific purpose. Moreover, the credibility of a model depends on a number of factors; including what Law phrases as the decision-maker, their understanding of the model and involvement in the project, the reputation of the model developers and effective and convincing animation. Credibility is not synonymous with validity, a model can be considered credible without being valid. For example, a model can be visually convincing but not have the needed accuracy to be considered valid. A seven-step approach is presented in Law’s tutorial [15] on how to create, im- plement and evaluate valid simulations: 1. Formulate the problem 2. Collect data and construct assumptions 3. Verify that the assumptions are valid 4. Program the model 5. Verify the validity of the programmed model 6. Design, conduct and analyze experiments 7. Document and present the results Communication errors are highlighted as a major reason for invalid assumptions of simulation models, according to Law [15]. In order to mitigate these errors, a com- prehensible assumptions document should be constructed to act as the main source of documentation for various stakeholders. Law [15] emphasizes that the document should primarily describe the particular issues a simulation model addresses, rather than providing an extensive description of the entire system. The assumptions doc- ument should contain enough information to act as the foundation for the developed simulation computer program. A couple of relevant points to include in the assump- tions document are: 8 2. Theory • Project goals • Process-flow / System-layout diagram • Subsystem descriptions and their interactions • Simplifying assumptions • Model limitations • Model input data summary • Important sources of information In addition to the assumptions document, Law [15] pinpoints that for the assump- tions document can be validated by performing a structured walk-through. More- over, he specifies that this validation technique of the assumptions document might be one of the most important validation techniques for nonexistent systems. The structured walk-through is performed by having a meeting with experts and stake- holders, and go through each point of the document and only proceed to the next point if every meeting participant is satisfied. The assumptions document should be sent to each participant prior to the meeting to allow for early reviews and com- ments. The main benefit of this validation technique is to ensure that simulation analysts receive a comprehensive and correct understanding of the system to model, increasing both the validity and credibility of the simulation model. Law [15] proposes a couple of techniques to verify and validate simulation mod- els. One proposed technique is to validate the output of a model by comparing it to observed data from the modeled, or real, system. The simulation model would then be considered valid if the model output data is close to the output data of the real system, by some threshold. This method is sometimes referred to as results validation. Statistical tests can be used to compare the data, though the common assumptions of independent and identically distributed data are not directly appli- cable. This is due to the nature of most real-world and simulation systems being non-stationary and auto-correlated [15]. A similar suggested statistical method is a Turing test, where system experts examine different sets of data without knowing their origin. Another proposed technique is animations. The animations can be beneficial in displaying the invalidity of a simulation model, and they can also be used to improve its credibility. Additionally, it can also be applicable for verification purposes of a simulation computer program. David et al. [16] discusses the V&V process on simulation model and proposes several methods and techniques that can be used during the different stages of the process. They discuss that the verifiability of a simulation model is influenced by the procedure used to develop the simulation and breaks it down to two main pro- cedures: • The model is definable as a set of input/output pairs in a specific range and thus the corresponding model is verified for the range considered. • The model is defined according to a researcher/stakeholder’s intention in a specified range and thus the corresponding model is verified for the range considered until it reaches the researcher/stakeholder’s expectation. 9 2. Theory David et al. [16] mentions that both of these procedures create a clearly defined parameter range to be verified. To help with the verification, they discuss how techniques such as object-oriented design can simplify testing and debugging of the model and how the use of defensive programming methodologies such as asser- tion tests and unit test are well suited for verifying simulation models. They also specifically mention two verification methods that according to them are the most important ones to complement the techniques mentioned, these methods are traces and structured walk-through. Raunak and Olsen [17] investigate how validation can be performed on discrete- event simulation (DES) models, from the perspective that this type of simulation is often deemed as non-testable software. They discuss that performing validation on non-testable software does not have methods for identifying and establishing the coverage of validation in regard to simulation models. When validating a simula- tion model, one needs to identify all elements that need validation. They propose that this can be done by organizing them into high-level aspect groups, where each aspect contains elements types. The elements can be divided into two categories, input data and observable emerging information (OEI). OEI can be any observable or measured behavior or data during the execution of the simulation [17]. For DES models, Raunak and Olsen [17] propose that the aspects groups are: Re- sources, Request Characteristics and Workflow, these are defined after the three main groups of elements found in a DES model. Elements in each aspect should be validated to ensure that the entire model is valid. However, they do not specifically propose any validation techniques, rather they mention that one should use the most common techniques that fit each element, such as animation, comparison to other models and extreme condition tests which are presented by Sargent [6] and Law [15]. Raunak and Olsen’s [17] main contribution is the implementation of weights to the validation process to be used to calculate a coverage rate for the validation process , where different techniques are given different weights based on the level of confi- dence the technique has. Each element is also given a weighted score based on the importance of validation of each element, the higher importance the higher weight. To then calculate the coverage of the entire model, three sets of information are needed. • Elements that should be validated • Validation techniques that can be applied for each element • Validation techniques that were applied on each element For each element, the coverage degree is determined by combining the weighted score of each validation technique used divided by the combined weighted score of all possible validation techniques. The coverage degree is then multiplied with the weighted score of the element and after this is done on all elements, the coverage degrees are combined and lastly divided by the total combined weight of all elements to get the coverage degree on the entire DES model. Another framework for V&V is proposed by Wang and Lehmann [18]. The frame- 10 2. Theory work focuses on three concepts of a simulation study: an associated role concept, a process and product concept, and a quality assurance concept. The role concept establishes an organizational structure to the model development by introducing responsibility roles according to each role’s capabilities, such as sponsors, model designers, software/hardware developers and users. The process and product con- cept is targeted towards the intermediate products developed during the different development cycles of a simulation model. The intermediate steps here refer to the products created from each role, for example conceptual models from a model designer or executable models from a developer. In the framework, the simulation model development process consists of decision points [18]. Each decision point is associated to a progress stage of the development and a collection of V&V activities to perform in each respective stage. The intermediary V&V activities are used to evaluate the intermediate products. However, the framework description does not specify any specific suitable V&V techniques to apply, but mentions that the activ- ities should be defined along with a requirement document. The aforementioned frameworks or methodologies fall short of the area of this thesis, since the intention of the proposed framework in this thesis is to be more focused on simulation models based on the standards OpenDRIVE and OpenSCENARIO. Sargent’s [6] proposed framework can be seen as more focused on mathematical or physics simulation models, where his recommended procedure can be hard to apply to traffic and road scenarios, where the models simulate more than just physics. Roungas et al.’s [12] proposed framework can be seen as too general for the applica- tion on esmini, where some parts could possibly be extracted and used on esmini, but the entire framework is not directly compatible. Barceló’s [13] idea of an itera- tive process could be applied to esmini, but would require modifications to be more appropriate for the models developed in esmini. Law’s [15] proposed assumptions document could be directly applicable to esmini, as it would improve the under- standing of the model to develop, however it would not improve the V&V process alone. The idea of adding weights to the V&V process which Raunak and Olsen [17] propose does not in itself solve the issue of V&V, as their approach is more focused on improving existing processes to increase the trustworthiness. 2.3 Modeling and Simulation techniques Modeling and simulation techniques are research methods that can be applied to organizational and operational systems in order to improve the V&V process of model design and development [19]. They are utilized to increase the understanding of real-world system behavior. 2.3.1 Simulation techniques In operational management systems, there are two commonly used simulation meth- ods, agent-based simulations and discrete-event simulations. Agent-based simulation (ABS) is a bottom-up architecture simulation method for 11 2. Theory simulating industrial processes and complex scientific systems [19]. This form of method aims to model individual agents (entities) and their interactions in a decen- tralized manner, where each agent control themselves. The simulation model inputs are therefore usually based on theory and data with respect to the behavior of the agents. The individual agents’ decisions affect the outcome of the overall system. Consequently, ABS is becoming a more prominent method to model and simulate systems where behavior is heavily dependent on the behavior of smaller system en- tities. On the contrary, discrete-event simulation is a top-down architecture simulation method where the focus is put on modeling time-based behavior systems on a higher level in a centralized manner [19]. The model input data is frequently based on ob- jective data collected from the system. Each entity in the simulation is processed in sets of related steps dependently, where the simulation behavior is determined by the relationship of these steps. 2.3.2 Traffic modeling techniques The purpose of traffic modeling is to reproduce real-life observed traffic. Traffic modeling contributes to the planning and management of traffic in road networks [20]. In general, the traffic simulation models target three main areas of traffic dilemmas: traffic flows, road network elements, and time and cost estimation of travels. Traffic simulation models can be divided into three categories of modeling architectures and applications, these are: microscopic, macroscopic and mesoscopic modeling. Microscopic modeling is based around modeling attributes of vehicles and their movement in traffic. Various parameters such as traffic flow, vehicle speed and stops are taken into account when collecting the model data. Azlan and Rohani [20] provides brief descriptions of car following, lane changing and gap acceptance mod- els as examples of microscopic modeling. In each of the models, certain parameters are observed to allow individual vehicles to control their own behavior and perform suitable actions. Macroscopic modeling, models the traffic stream from mainly speed, flow, and den- sity characteristics in mathematical descriptions [20]. These forms of models are considered continuous simulations. Macroscopic models apply the same treatment to every vehicle, and thus cannot make independent actions for each individual ve- hicle [21]. This form of modeling is useful for models that do not require a high level of detail, for example highway networks. The main benefit of macroscopic modeling is its potential to provide useful and accurate information while still enabling a fast simulation. Mesoscopic modeling can be seen as a middle ground between macroscopic and microscopic modeling [22]. It provides analysis of the different entities in smaller groups [20]. The level of detail in the modeling of individual traffic entities is higher 12 2. Theory than macroscopic, but not as high as microscopic. There exists a fourth category, which is nanoscopic or submicroscopic modeling. An extension of the microscopic modeling approach, where details regarding inter- nal functions of a vehicle is taken into consideration. This type of modeling is often used for autonomous driving simulation, where vehicle vision or details like gear shifts are simulated [22]. 2.4 ASAM Standards Association for Standardization of Automation and Measuring Systems (ASAM) is a non-profit organization that hosts standards within the field of automotive de- velopment and testing, founded in 1998 after an initiative by several German car manufacturers, such as AUDI and BMW [23],[24]. As of now, ASAM has more than 350 companies that are members in the organization. The standards that ASAM hosts are developed in collaboration between the members. ASAM Standards defines data models, communication APIs, file formats, software components APIs and communication protocols for data exchanged to help research, development, and validation [25]. The aim of ASAM standardization is to allow for the choice of the best tools based on capabilities, efficiency and support. 2.4.1 OpenDRIVE Open Dynamic Road Information for Vehicle Environment (OpenDRIVE) is one of the standards that is created by ASAM members, where the goal is to create a common base for describing road networks for driving or traffic simulators [26]. The OpenDRIVE standard version 1.7.0 defines a format for creating road networks using extensible markup language (XML) syntax and the file extension xodr. The main purpose of OpenDRIVE is to allow for exchanges between simulators, this so that the same description can be used without the need of first translating it, thus reducing cost and spent time for development in the industry. OpenDRIVE can be used to describe geometry of roads, lanes, and objects such as road markers, it also allows for defining features such as signals. Roads in Open- DRIVE are defined based on a reference line, which has its own coordination system (s/t) which differ from the absolute coordination system (x/y). This allows for features such as road markers to be attached to a road’s reference line with (s/t) coordinates. Lanes are attached on either side of the reference line, given +/- IDs based on which side of the reference line the lane is [27], an example of this can be seen in Figure 2.1. 13 2. Theory Figure 2.1: Example of OpenDRIVE reference line [26] © ASAM e.V.(Used with permission) Roads can be connected to form junctions, and roads and junctions combined can form road networks. Junctions can be defined in three different ways according to OpenDRIVE, these are: common, direct or virtual junctions [27]. Common junctions contain several overlapping roads that are drivable, which can represent a normal four way intersection, seen in Figure 2.2. Direct junctions cannot have overlapping lanes and can be used to represent highway entries and exits, seen in Figure 2.3. Virtual junctions are not implemented in the current version of esmini, hence they are irrelevant for this thesis. Figure 2.2: Example of OpenDRIVE common junction [27] © ASAM e.V. (Used with permission) 14 2. Theory Figure 2.3: Example of OpenDRIVE direct junction [27] © ASAM e.V. (Used with permission) 2.4.2 OpenSCENARIO OpenSCENARIO is another standard created by ASAM members, it defines a file format using XML syntax to describe dynamic content of traffic and driving sce- narios. The OpenSCENARIO version referenced during this thesis is 1.1.1 [28]. OpenSCENARIO’s primary use-case is to describe complex maneuvers that involve several entities, where a maneuver is described as driver actions or trajectories that are performed synchronously. However, OpenSCENARIO is not limited to this, as it also can contain the description of vehicles, drivers, pedestrians, traffic and envi- ronmental conditions [9]. A maneuver is described in a storyboard, which is divided into three parts, sto- ries, acts, and maneuvers groups. A story describes a specific driving maneuver, for a single vehicle, or specifies the dynamic behavior of several entities. Each story can then be divided into several acts, which can be set to trigger at specific conditions. Each act contains a maneuver group, which contains maneuvers on multiple vehicles [9]. These maneuvers are described in detail by the use of events and actions, where events describe when something is supposed to happen and actions describes what should happen. There exists three types of actions in OpenSCENARIO, private, global or user-defined. A private action describes what should happen for a single entity, such as a vehicle, and could be a lane change. A global action relates to the simulation and can be used to modify non-entity elements, like time of day and traffic signals. A user-defined action however is an action that is not part of the standard OpenSCENARIO, but something added between the simulator and the scenario designer[28]. 15 2. Theory 2.5 Environment Simulator Minimalistic (esmini) Environment Simulator Minimalistic is a simulator developed during a Swedish col- laborative research project called Simulation Scenarios [29], the development of which has continued based on user needs and the continued development of Open- SCENARIO and OpenDRIVE [9], [30]. It is mainly written in the C++ program- ming language. Currently, Volvo Cars is one of the main developers of esmini. The two most important modules used in esmini are RoadManager and Scenario- Engine. RoadManager provides the interface to OpenDRIVE, while ScenarioEngine provides an interface to OpenSCENARIO. The purpose of esmini is to provide a cross-platform development tool for working with the OpenSCENARIO data for- mat [30]. Traffic scenarios in esmini are simulated from road network information stored in OpenDRIVE and OpenSCENARIO formats. The modeling techniques DES and microscopic modeling are used in esmini, and these concepts are explained in Section 2.3. Since esmini a deterministic simulator with little stochastic models, it can be used as a foundation for building other simulators that implement more stochastic fea- tures as it can act as the "ground truth" for elements of the simulation that are not as important for the objective of the simulation. The esmini simulator is therefore useful when only the barebone of a simulation is necessary. 2.5.1 Structure There are six main modules in esmini, these are RoadManager, ScenarioEngine, Con- trollers, ViewerBase, PlayerBase and CommonMini [31]. As mentioned previously, the RoadManager module is responsible for the implementation of the OpenDRIVE format. ScenarioEngine handles parsing of OpenSCENARIO files and constructs entities, triggers and actions which are then consequently executed. The Scenari- oEngine also accompanies a ScenarioGateway, whose purpose is to asynchronously provide information about entity states, for instance through the UDP protocol. Controllers are a concept present in OpenSCENARIO that enables custom behav- ior of individual entities in addition to the functionality provided by the default controller within ScenarioEngine. Controllers follow an API defined by class inher- itance, where inherited classes can override and define custom behavior for certain entities in the scenario. The ViewerBase module is a 3D viewer that provides visu- alizations for the road network and features, as well as 3D models for the scenario entities. The PlayerBase module connects the ScenarioEngine and ViewerBase to provide a higher level API for handling scenarios in custom player applications. Lastly, the CommonMini module is a collection of useful functions that can be used by other modules. Some examples of the functionality provided in collection are timers, math operations and loggers [31]. 16 2. Theory Figure 2.4: UML diagram of acts and all related classes in esmini. 2.5.2 Event and Action Events and actions in esmini are implemented based on the definitions from the OpenSCENARIO standard [28]. As previously mentioned, OpenSCENARIO defines three different actions, however only two of them are implemented in esmini as of spring 2022. The two actions that are implemented can be seen in Figure 2.4, which shows how all the classes related to acts are linked in esmini. In the UML diagram, it can be seen that events can have one or more actions and a trigger that states the start conditions of the event. It can also be seen that all links between these classes are unidirectional, meaning that e.g, actions can not see what events they are a part of. Global and private actions work on different parts of the simulation, global on the simulation and private on an entity. The implemented global actions are: • Set parameter • Traffic swarm • Environmental / infrastructure • Add entity The private actions implemented in esmini are: • Longitudinal speed • Longitudinal distance • Lateral lane change • Lateral lane offset • Assign controller • Activate controller • Teleport entity • Assign route • Follow trajectory • Synchronize Most of the private actions in esmini are quite self-explanatory of the functionality 17 2. Theory based on the name of the action. The actions have different parameters that change their behavior, one of these parameters are if the actions has a relative or absolute target. E.g, in lateral lane changes, this changes the behavior to either change to a relative lane based on the location of the entity or an absolute lane that the entity should change to. 2.5.3 Controllers Controllers, which is a core element of the OpenSCENARIO simulation [28], is im- plemented in esmini. The purpose of a controller is to customize the behavior of simulation agents during runtime. A motivating factor for the usage of controllers is that they provide a flexible way of extending functionality in esmini while also conforming to the OpenSCENARIO standard [32]. The standard does not define controller implementations, and the implementation present in esmini is hence an interpretation of the controller concept. However, the standard does specify that controllers are deactivated by default. Furthermore, esmini provides a set of embed- ded controllers that can be useful for simulating certain scenarios. For instance, the SloppyDriverController is a simulation model where the simulation entity performs slight speed and turning deviations in order to mimic a sloppy driver [32]. Controllers are activated for a domain: longitudinal, lateral or both [32]. A do- main represents the direction of motion of a simulation entity. The longitudinal domain corresponds to motion along a reference line, for example, increasing the speed of the vehicle. The lateral domain corresponds to motion perpendicular to a reference line, such as lane changes. OpenSCENARIO specifies two controller types: the default controller and user- defined controllers. The default controller must enforce so-called control strategies precisely, while user-defined controllers may customize these strategies. Control strategies define how entities are assigned behaviors [28]. The default controller follow control strategies after how they are defined in the OpenSCENARIO file. 2.5.4 Similar simulators There are two other traffic simulators that are similar to esmini in the terms of being capable of using the ASAM standard OpenDRIVE. These two simulators are CARLA [33] and SUMO [34]. CARLA is a open-source simulator developed for autonomous driving development and uses high fidelity graphics and physics enabled by the use of Unreal Engine 4 to ensure that it is possible to train sensors with realistic data. This means that CARLA is quite heavy to run compared to esmini. CARLA have some support for OpenDRIVE and OpenSCENARIO, but does not currently support the most recent version of OpenSCENARIO. SUMO is a more lightweight traffic simulator and focuses more on traffic flows and traffic management, which differs from esmini that is more useful for simulating smaller traffic cases. The main difference between esmini and the simulators CARLA and SUMO is that esmini was purposefully developed to support the ASAM standards 18 2. Theory Figure 2.5: Example of a general QAS for the quality attribute availability. OpenDRIVE and OpenSCENARIO, while the other simulators has support for these standards by the use of additional components but it is not one of the main features. 2.6 Quality Attribute Scenarios Quality attributes are used in software engineering to describe a measurable or testable property of a system [35]. The ISO 25010:2011 Standard [36] defines a set of software quality attributes to use together with requirements and quality evaluations. The attributes defined in the standard are the ones referred to in this thesis. A quality attribute is used to indicate how well a system reaches its requirements from stakeholders [35]. These are often retrieved from descriptions such as functional requirements of a system, which means that they often specify attributes, like modifiability or availability, that can be hard to measure without further explanation or requirements. To solve this, one can use quality attribute scenarios (QAS), which contains more concrete information about what is expected of a system. A QAS is divided into six parts, each describing one section of the requirement [35]: • Stimulus: An event that arrives to the software. • Stimulus source: Where the stimulus came from. • Response: How the system should respond to the stimulus. • Response measure: Measure of the response, if it reached the required degree. • Environment: The condition of a system when the stimulus happens. • Artifact: The portion of the system that is applicable for the requirement. QAS allows for making quality attribute requirements testable and can be divided into two subgroups, general or concrete, depending on how the QAS is defined. General QAS are quality attributes scenarios that are not bound to a specific system [35]. These are defined with a collection of definitions for each part of the QAS which contains the characteristics for a specific quality attribute. A general QAS can be modified to be system dependent by just having the relevant definitions for a quality attribute for the given system. An example of a general QAS for the quality attribute availability can be seen in Figure 2.5. Concrete QAS (CQAS) are however specific to a system [35]. These are used when testing if a system reached the wanted degree of a certain quality attribute. The 19 2. Theory Figure 2.6: Example of a concrete QAS for the quality attribute availability. difference between a general and a concrete QAS is that the CQAS only have one definition for each part, which allows it to create a testable definition of how a system should react to an event, as well as making it measurable. An example of a CQAS for the quality attribute availability can be seen in Figure 2.6. 2.7 Acceptance Test-Driven Development Acceptance Test-Driven Development (ATDD) is a development methodology that is similar to regular test-driven development, but focuses on acceptance testing to help improve software quality and ensure that customer needs are fulfilled [37]. ATDD helps to provide a software development structure that closely follows and implements the required functionality. Furthermore, the productivity of a team and the quality of the code are increased by applying ATDD in a team [38]. The main difference between a typical development workflow and an ATDD workflow can be seen in Figure 2.7 and Figure 2.8. This section aims to describe the concept of ATDD and how it can be beneficial in a V&V context. Figure 2.7: Flowchart of a typical development workflow Figure 2.8: Flowchart of acceptance test-driven development workflow The main building blocks of ATDD processes are user stories [37]. Briefly described, 20 2. Theory Figure 2.9: The ATDD cycle visualized as a graph a user story is a format for describing requirements that expresses desired behavior by specifying who does what and why. In other words, the who can be seen as a role or entity, the what can represent some functionality and the why describes the benefit of the functionality. User stories are convenient for specifying requirements in a format that most project participants can comprehend. Acceptance tests are high-level descriptions of some desired system behavior or func- tionality [37]. These tests are designed in relation to a user story and establishes the behavior of the system in regard to some set of conditions as well as input and output data. Acceptance tests focus mainly on what functionality to implement and not on how it is achieved. An important aspect of acceptance tests is that they are usually written together with different project roles, where the customer, developer, and tester are all contributing to the design of the test. The ATDD process is performed cyclically, and the cycle consists of four steps. A basic visual representation of the cycle can be seen in Figure 2.9. A more detailed visualization can be seen in Figure 2.10. 1. Pick a story 2. Write tests 3. Automate tests 4. Implement functionality This process is performed for every user story. First, a user story is picked, and the choice of story can depend on a priority list of user stories to consider. Secondly, cre- ate the acceptance tests together with the customer, for example by spontaneously coming up with relevant scenarios to test and sketching them down in any shape or form. Thirdly, the tests should be automated so that they can be executed auto- matically, making sure that the tests are identically executed every time and that they provide simple yes-or-no answers. The fourth and final step in the cycle is to implement the functionality in order to pass the previously created tests. When all 21 2. Theory Figure 2.10: The ATDD cycle graph in a more detailed visualization the tests are passed, by implementing the corresponding functionality, a new user story is picked and the cycle is repeated accordingly [37]. One of the main benefits of an ATDD workflow is the distinct definition of done it provides. Koskela [37] highlights two key issues of project management, knowing where a team is in a project timeline and knowing when to stop. ATDD provides a simple answer to these questions, which is that a requirement is fulfilled if all the tests for the corresponding user story pass. If not, there is remaining functionality to be implemented before proceeding to a new user story. 2.8 Test automation Test automation is usually associated with the automation of test execution, but has in more recent years been expanded to other test activities such as the design, scripting, evaluation and reporting of tests [39]. Several tools and software libraries already exist to provide effective help for automation in different testing activities. The automation of testing activities aim to reduce the effort and cost required to develop tests, which subsequently promotes development of higher quality software. An example of a testing approach to automate both test design and generation is model-based testing (MBT) [40]. Instead of letting a developer manually design a test, MBT uses modeling languages such as UML to express the behavior of a system. Various model elements, such as states and transitions, are traced from the 22 2. Theory requirements to describe the system flow and its decisions. For most MBT tools, tests can then be automatically generated from the model, with input parameters and expected output parameters. The generated test describe sequences of actions and events that a developer would normally have to perform manually. The essential modeling process of MBT creates a visual representation of the model, usually in the form of a graph. This promotes understanding of the model by not only develop- ers, but also testers and other stakeholders. Furthermore, the MBT test generation criteria can be altered to limit the number of generated tests [40]. For instance, they can focus on requirements, allowing more tests that prioritize requirements to be generated to verify that all requirements are fulfilled. This encourages close de- velopment with specified requirements, which ensures a more reliable V&V process and ultimately higher quality software. Another example of a testing approach to automate test generation is property- based testing (PBT) [41]. PBT generates random tests through description of valid inputs and expected properties to hold for all valid inputs. Therefore, testers only have to specify valid inputs and properties instead of manually verifying inputs and expected outputs for many test cases. Writing tests with PBT is fast, more concise, and generates more test cases than a human would be able to do manually [41]. Gambi et al. [42] propose a search-based testing technique, together with proce- dural generation, to automatically test self-driving cars. In summary, their idea is to create challenging virtual scenarios to test a virtual vehicle in, by generating new road networks procedurally. Challenges introduced in a scenario could be factors such as weather conditions or complex road shapes. In Gambi et al.’s case, this testing technique is used to ensure self-driving cars perform lane keeping correctly in a virtual environment. 23 2. Theory 24 3 Methods This chapter will describe and motivate the choices made during the development of the framework and the model. It will also present the research and workflow techniques used to achieve the result. 3.1 Design Science Research Process The design method chosen for this thesis is the Design Science Research Process (DSRP), as it has several advantages for our study. Mainly, DSRP allows us to develop artifacts (a framework for V&V). DSRP is a six stage process and uses an iterative process, seen in Figure 3.1, where the fifth and sixth step checks the current system and allows for improvements if deemed necessary [43][44]. DSRP will also allow us to solve a real problem, which in our case is to improve the V&V process for development in esmini. First stage: The first stage is problem identification and motivation. The problem to be investigated in this thesis is to improve the V&V process for development in esmini using the ASAM standards OpenDRIVE and OpenSCENARIO. The re- search methods applied here are field study and desk research. These methods will be applied in order to gain knowledge on existing research as well as the systems involved. In this stage, the modeling techniques used for lane changing behavior in esmini will be identified. The identified techniques will create the foundation for a new artifact that is to improve V&V process. Model-based performance analysis can be leveraged to explore performance properties of esmini models. Moreover, this analysis would help us understand the modeling of parametric dependencies for better performance prediction [45]. Second stage: The second stage is the definition of the objectives of a solution. The objective is to create a framework artifact for V&V on simulation models for the esmini simulator. Other similar proposed frameworks will be examined to un- derstand the potential steps that are feasible in the framework artifact. The artifact should primarily focus on lane changing models, but the desire is for the framework to be generalizable. Third stage: The third stage is design and development. During this step, the creation of the artifact (a framework in the form of an informal process containing a set of systematic procedures applied in given steps) takes place (RQ1). Defining 25 3. Methods Figure 3.1: The six steps of the Design Science Research Process. desired properties, planning of the artifact’s architecture and the creation will take place during this step. The artifact will be designed based on information gathered from esmini as well as literature of existing similar frameworks. During this stage, we will get support from Volvo Cars in the form of details of simulation models in use and documentation on algorithms and access to experts in the field (esmini). Therefore, all of these resources combined should help us define the details of our framework development. Fourth stage: The fourth stage is demonstration. In this step, the artifact will be deployed on a simulation model (RQ2). The artifact will be used to drive the V&V process on the specific model, and relevant data will be collected regarding the artifact’s usefulness and effectiveness. Fifth stage: The fifth stage is evaluation. In this stage, data collected during the previous stage, the demonstration of the artifact (the framework), will be used to determine the artifact’s usefulness and effectiveness. Meaning, how much help the framework provides during the process in terms of ensuring that the stated re- quirements and users needs are met. At the end of this stage there will be a decision determining if the artifact has reached an acceptable level, if it is deemed to not have reached the sought after efficiency, the processes can revert to the second stage for additional adjustments. However, if deemed acceptable, the process can be moved to the sixth and last step. (RQ2) Sixth stage: The sixth and last stage is communication. During this stage, the artifact together with the problem is communicated to relevant audiences, as for ex- ample researchers, professionals, and practitioners. In our case, the outcome will be communicated through a thesis that will present the framework and how it improves V&V. 26 3. Methods 3.2 Research methods The research methods chosen for this thesis are field study and desk research. Field study refers to research conducted in a specific, real world setting to study a specific software engineering phenomenon [46], which for this thesis are V&V of a specific simulation software (esmini). This research method was chosen as it would allow for information gathering about the existing system in regard to modeling tech- niques and V&V techniques. A base knowledge needed to be established for the V&V Framework development. Therefore, the plan was to investigate the modeling techniques that were used in the simulator esmini, especially with regard to Volvo Cars’ lane changing road scenarios. Desk research is a secondary study and aims to summarize or synthesize existing research presented in primary studies [46]. The reason for choosing desk research as one of the methods is that to gather knowledge for RQ1 & RQ2, we need to study and summarize already existing frameworks in regard to the V&V process for simulation models. 3.3 The creation of the framework In this section, the process of creating the framework will be described, as well as the motivation behind the choices made during the development. It will also present the metrics used to determine the frameworks usefulness and effectiveness. 3.3.1 Development process In order to create the framework, a structured process that could help guide the di- rection of development was needed. This chosen structure was DSRP, as described previously in Section 3.1. As described in the DSRP process, the first step for devel- oping the framework was to identify the problems of esmini and V&V. To identify the problems, more knowledge of existing V&V techniques and methods was needed, so the first step was to gather more knowledge of V&V and the structure of esmini. After the knowledge was acquired, the next step was to begin to define the definition of a solution to the V&V process in esmini. During this step, the most relevant tech- niques and ideas were extracted and combined with the ideas of QAS and ATDD to form the beginning of a solution for V&V in esmini. After the definition of a solution is defined, the next step was the design and de- velopment iterations, step three to six in Figure 3.1. The first iteration began with combining the ideas of Barceló [13],[14], Law [15], QAS [35] and ATDD [37], [38] to create the first draft of the framework. After the first draft was made, the next step was to use in a concrete case, to make it more understandable and to see what was missing in terms of verification and validation. The concrete case used for this is the lane independent routing model, since the framework creation is further than the development of the model, some data, and ideas used in the test application of the first draft were fictional. This process repeated itself over a number of times before the idea of the framework was finalized. The application of the framework, on the concrete case, can be seen in Chapter 5. 27 3. Methods 3.3.2 Motivation for chosen procedures The assumptions document, based on Law’s [15] idea, is important in relation to esmini mainly because of how esmini is developed with regard to the OpenDRIVE and OpenSCENARIO standards. The standards do not always specify exactly how some functionality is implemented, therefore requiring interpretations of how it could be achieved specifically in esmini. Along with the interpretations, assumptions will be needed to create a compliant implementation. Due to how esmini is implemented, there are some limitations to what can be done and with the help of assumptions the limitations in esmini can be circumvented. Additionally, the idea of using an assumptions document, or requirement document, as part of the V&V process is backed up by Wang and Lehmann [18]. Requirements are vital to the testing process of all software, as they document the desired functionality. To verify the stated requirements, different testing tech- niques can be used, one being acceptance tests. The creation of acceptance tests has an increased difficulty compared to other testing levels, as it can be hard to break down the requirements into testable components. The use of CQAS allows for a clearer connection to acceptance tests, where the focus of CQAS is on creating requirements based on quality attributes such as correctness or availability. Conse- quently, the acceptance tests are easier to define as they target a specific attribute. Additionally, CQAS reduces the complexity of behavior and functional testing by providing an example that makes the behavior or functionality description easier to comprehend. The use of requirements is also supported by David et al.’s [16] idea that models are verifiable after the stakeholder’s intentions, and should therefore be verified until it reaches the stakeholder’s expectations. ATDD is a development structure that helps to closely follow and implement the desired functionality. The idea of implementing the ATDD methodology into the framework is motivated by the fact that validation becomes more prioritized and a part of the development process. Since esmini does not currently have a designated validation process, ATDD creates a structure to build the validation process upon. Acceptance tests provide additional definitions of done, allowing a team to know where they are on the project timeline and knowing when a requirement is fulfilled. The combination of requirements, CQAS and acceptance tests through ATDD en- ables a substantial and iterative V&V process, where new requirements can be added throughout the development of the model. This promotes the robustness, reliability, and correctness of the simulation model. Moreover, each stage of the iterative pro- cess can be seen as what Wang and Lehmann [18] define as a decision point. Specific V&V techniques are applied during each stage in order to evaluate the intermediary artifacts created during the model development. In addition, the different develop- ment stages follow the role concept introduced by Wang and Lehmann [18], where each stage is connected to a set of roles that are responsible for that part of the model development process. The idea of performing V&V iteratively comes from Barceló [13], as he states 28 3. Methods that validation can be performed iteratively until an acceptable accuracy has been achieved. The iterative process allows for simulation models to be calibrated and optimized based on the requirements and acceptance criteria that has been created in earlier stages of the process. This is useful for esmini, where accurate models are especially important due to esmini being used as a test tool for safety critical software for vehicles. The final validation step in the iterative process relates to the model tests men- tioned by Barceló [14], where models are compared to each other or expected data. Here, methods such as statistical comparison, empirical analysis, and visual inspec- tion can be used to calculate the model’s accuracy. Other methods for statistical and visual validation can be obtained from Roungas et al.’s paper [12]. The addition of the final validation step enhances the validation process by including other ways of validating requirements and acceptance criteria. Using a diverse set of validation methods increases the probability of developing a highly accurate model. If the model fulfills the acceptance criteria it can be seen as validated, if not, revert to the assumption document to find why the model does not behave as expected. 3.3.3 The application of the framework The framework will be applied to the simulation model developed as a part of the thesis, the application of the framework on the model will be shown in Chapter 5. The application will then be evaluated based on the metrics: usefulness and effec- tiveness to answer RQ2. The evaluation of the framework will be based upon the example of how the frame- work can be used, found in Chapter 5. The evaluation will consist of two main methods, where one is the injection of faults into the model to see how effective the framework is on detecting the faults. The other is user interviews, where developers of esmini will be asked to answer a series of questions to determine if they find the framework useful or effective for esmini based on the example provided. Based on the result of the injection of faults and the feedback from developers, a degree of use- fulness and effectiveness should be possible to determine. However, a study, where the framework is used during development of models by esmini developers and then evaluated based on their feedback, would have provided a stronger foundation for evaluation. But due to time and resource limitation, it was deemed not possible for the thesis. 3.4 The development of a lane independent rout- ing model (LIRM) This section will present the methods and steps followed in order to develop the lane independent routing model (LIRM), based on the case provided by Volvo Cars, seen in Appendix A. 29 3. Methods 3.4.1 Model criteria The simulation model developed simultaneously as the framework is a simplified model of a vehicle navigator that should be able to find and calculate a path to a target anywhere on the simulated road network. A detailed case description can be seen in Appendix A. The current implementation of path finding in esmini is lane dependent, meaning that the path finding can not find paths that are not directly connected to the starting position’s lane, which is how the default control strategy is defined in OpenSCENARIO. The developed model, the LIRM, will be in the form of a controller capable of overriding the default behavior of path finding in esmini. The model will be able to find paths independent of the starting lane. The model should also implement a functionality for changing lanes when it is necessary, to be able to follow the path. 3.4.2 Model design The development of LIRM started with research into existing path finding algo- rithms that could be adopted to fit the model’s purpose. A few alternatives were considered, such as Dijkstra’s algorithm and A* search algorithm [47]. The choice fell on Dijkstra’s as the path finding algorithm for the implemented model, due to the lesser complexity and easier implementation. After the choice of algorithm was made, the work began to find out and write down all assumptions made, this included the assumption around the algorithm as well as around the implementation of the entire model, such as the simulated road network in OpenDRIVE. These assumptions can be seen in the example case for the framework, in Section 5.1. This was done both as a part of V&V process of the model, but also for documentation on why some functionality of the implementation works as it does. 3.4.3 Model development The implementation of LIRM into esmini started as a new controller that will con- tain both the modeled path finding and the logic for modeled lane changes. The code for the model, LIRM, can be found on GitHub1. The implementation can be found in files ControllerFollowRoute and LaneIndependentRouter. The newly added controller enables scenarios to use it dynamically, adding to the behavior of the default controller and provide the intended pathfinding and waypoint following functionality. Lane independent path finding algorithm The new functionality of lane independent path finding needed an algorithm capable of finding the path through an OpenDRIVE road network. As mentioned, the choice fell on an implementation of Dijkstra’s algorithm which needed some modifications to work for the given purpose. To implement Dijkstra’s algorithm, there was a 1https://github.com/esmini/esmini/tree/follow_routes 30 3. Methods need for defining what were nodes and edges in the OpenDRIVE network. Our implementation was that roads became edges and the connections between roads became the nodes. This meant that the algorithm would find all connections between two roads the vehicle needs to pass through to reach the goal. To make this approach lane independent, the implementation added a node for the connection of each lane in the driving direction on a road. The lane independent path finding supported three different routing strategies, that are also specified in OpenSCENARIO [28], and these were: shortest, fastest and least number of intersections (junctions). A simplified description in pseudocode of the implemented path finding algorithm can be seen in Algorithm 1. priorityqueue pq; list visited; Create startNode; Add startNode to pq; while pq not empty do Get first node (n) in pq; if n is in visited then Continue; end Add n to visited; if n == target and target is in driving direction then create targetNode; return backTraceTargetToStart(targetNode); end for each road in connection n do for each lane in driving direction on road do Get connecting road for lane; Create newNode; Calculate weight for newNode; Add newNode to pq; end end end return empty list; Algorithm 1: The lane independent path finding algorithm in pseudocode. Lane change controller A lane change controller was developed to allow the modeled vehicle to follow routes that included lane changes to reach the target, the logic of the controller can be seen in the flowchart in Figure 3.2. This was needed as the default controller in esmini does not have any functionality of changing lanes. However, since the normal driving and route following could be handled by the default controller, it was decided to let it control the vehicle until a lane change was necessary. This design choice was made as creating a controller that handled all driving on its own would require duplicated 31 3. Methods Figure 3.2: A flowchart of the logic in the lane change controller. code and functionality that the default controller already had, which would have reduced the maintainability of esmini and should be avoided if possible. When a lane change is necessary, the controller creates a new lateral lane change action, which overrides the default controller and takes control over the modeled vehicle while the lane change is performed. After the lane change is complete, the control is returned to the default controller. To get the default controller to follow the waypoints created by the lane indepen- dent pathfinder, it was necessary to override the default waypoints created by the default pathfinder. This was done by continuously updating the list of waypoints that the default controller uses after each waypoint has been passed. When the final waypoint has been passed, the controller is deactivated and the default controller behavior takes over the driving of the vehicle. 32 4 Framework This chapter will present the developed framework. An illustrative example of how the framework can be used to validate a model in esmini is seen in Chapter 5. The developed framework for verification and validation for models in esmini is described in the section below. It will provide a description of how the methods and process presented in the framework can be applied to a model to improve the verification and validation. Displayed in Figure 4.1 is a flowchart of the framework. The flowchart visualizes the connections between the different steps of the frame- work and shows how the framework deploys the ATDD and CQAS methodologies through an iterative process. In the current framework, all steps and processes that are a part of the framework need to be performed manually except for unit tests and some acceptance tests that can be automatically executed if wanted. The primary reason for including mostly manual steps in the framework is the time constraint of the thesis, as automating certain test activities would still require manual devising and preparation, which would have been too time-consuming. 4.1 Assumptions Document Communication errors across stakeholders are highlighted as a major reason for in- valid assumptions of simulation models [15]. In order to mitigate these errors, a comprehensible assumptions document should be constructed to act as the main source of documentation for various stakeholders. The document should primarily describe the particular issues a simulation model addresses, rather than providing an extensive description of the entire system. The assumptions document should contain enough information to act as the foundation for the developed simulation computer program. The document can be validated by performing a structured walk-through, where a meeting is held with stakeholders to ensure they are all sat- isfied with the stated information and that they understand the fundamentals of the modeled system. The term model in the assumption document refers to the simulation model that should be developed, in other words, the software or code that the model consists of. The assumptions document can be seen as quite similar to a requirement document, that is often used in software engineering, as both explains the wanted functionality 33 4. Framework Figure 4.1: Flowchart of the V&V framework 34 4. Framework of the software that is to be developed. However, a requirement document repre- sents the functionality that a system should have by stating requirements that the developed system should fulfill. The definition of an assumptions document pre- sented in this thesis aims to represent both the general functionality of a model and all possible assumptions and constraints which could affect the model development. The assumptions document should then be used during the creation of the actual requirements to ensure that the stated requirements consider the assumptions and constraints that could affect the adaptation of the real system to a model. 4.1.1 Model functionality Provide a description of the functionality that is desired with the model usage. For example, write down a list of features, explain the model’s behavior output depending on inputs, or show a figure that contributes to the overall comprehension of the model’s behavior and purpose. The functionality described here does not need to be excessively formal, as the more formal descriptions are created during a later requirement stage of the framework application. 4.1.2 Model flowchart Create a flowchart of the model’s functionality. It should contain the most important functionality for the model and depict the different paths within the functionality. This should be done to improve the understanding of the model both for stakeholders and the model designers, to minimize the risk of misunderstandings of the model’s functionality. 4.1.3 Model subsystems Provide a more detailed description of the model’s main functionalities. Define what each subsystem is responsible for and how they interact with each other, in terms of what type of data may be exchanged between them. 4.1.4 Simplifying assumptions Here, all assumptions regarding simplification of the real system should be stated. Every assumption on how the simulation model simplifies the real system should be explained in detail of how it differs and why the assumption is needed. The documentation of assumptions is necessary for the creation of requirements and validation, as it provides important information about how the developed model is expected to behave in relation to reality. 4.1.5 Model limitations Describe the possible limitations of the model that is to be developed. Any underly- ing limitations in algorithms, software, or data used should be considered and noted down. The limitations are essential in order to understand what cannot be achieved with the model or how to possibly circumvent certain constraints. 35 4. Framework 4.1.6 Model input data summary Summarize the model input data to create a collection of data that can more easily be comprehended by different types of stakeholders. The summary acts as an overview of what input data the model is provided. It should also include the different types of data, for instance text, numbers, boolean values, files etc. Complex data should be explained and simplified to increase the level of comprehension and make it easier for stakeholders with less specific knowledge to gain an understanding of the data. 4.1.7 Important sources of information Document the sources used to gather the information that is being used to establish the assumptions and limitations of the model. For each source, provide information of what type of information the source provides. This should be done to facilitate development in future iterations of the model, as it will eliminate the need to find where more detailed information regarding the model can be found. 4.1.8 Acceptance criteria Acceptance criteria are used to define the definition of done for the model. The criteria should define how the model will behave related to the real system, and to what degree the model can differ while still being acceptable. These criteria are used during the V&V process to know if the model has reached the wanted degree of accuracy to be deemed validated. These are also important for the requirement and acceptance test creation for the model, in order to reconnect to the criteria and check if the model is considered verified and validated. 4.1.9 Assumption validation Validate the assumptions document before proceeding to the next step. A suggested validation technique to use here is a structured walk-through [15]. A structured walk-through is performed by having a meeting with experts and stakeholders, and together address each point of the document and only proceed to the next point if every participant in the meeting is satisfied. If any point is deemed inappropri- ate, that part of the assumptions document should be reconsidered and improved accordingly. The assumptions document should be sent to each participant prior to the meeting to allow for early reviews and comments. The main benefit of this validation technique is to ensure that simulation analysts receive a comprehensive and correct understanding of the system to model, increasing both the validity and credibility of the simulation model. 4.2 Requirements, Concrete QAS and Acceptance Tests After the assumptions document has been created and validated, the next step in the framework is to create the requirements, CQAS and acceptance tests for the 36 4. Framework model. 4.2.1 Requirements Requirements are used to ensure that the system fulfills a specific functionality or behavior. The same applies with models, where requirements are used to state im- portant functions. These are then used to create CQAS and acceptance tests for the model, which helps improve the V&V process. An effective way of formulating the requirements is to create user stories. User stories follow the form of who wants what and why, most commonly “as a I want because ”. They provide a compact way of describing the desired functionality, who benefits from it and its purpose. 4.2.2 Concrete Quality Attribute Scenario Quality attributes are used in software engineering to describe a measurable or testable property of a system [35]. A quality attribute is used to indicate how well a system fulfills its requirements from stakeholders. CQASs are used for each requirement to further explain where the source of the data comes from, how it interacts with the system and what is expected from the interaction. This allows for even more specific information about what is expected from a requirement than just having it as a user story. It also allows for validation of models, where real- life/system data may not exist, as CQASs can be used to compare to expectations obtained from the earlier stated assumptions. A CQAS is divided into six parts, each describing one section of the requirement, these can be seen in Table 4.1. Stimulus An event that arrives to the model Source of stimulus Where the stimulus came from Response How the model is expected to respond to the stimulus Response measure Measure of the response Artifact The part of the system that the requirement applies to Environment The condition of the model when the stimulus happen Table 4.1: Table for defining a Concrete Quality Attribute Scenario. 4.2.3 Acceptance test Acceptance tests are high-level descriptions of some desired system behavior or func- tionality. These tests are designed in relation to a user story (a requirement) to es- tablish the behavior of the system in regard to conditions, input, and output data. The main purpose of using acceptance tests is to provide a way of determining whether a requirement is satisfied. The tests focus mainly on what functionality to implement, but not how it is achieved. They are usually designed in coordination between different project roles (e.g., customer, developer, tester etc.) to include contributions from different stakeholders. A simplified version of the acceptance test can be provided in pseudocode to allow 37 4. Framework for a coherent overview of what is required for the test. In addition, a description of the acceptance test should be provided to clarify any ambiguities. During the creation of the acceptance tests, an OpenSCENARIO file or an OpenDRIVE file should be created or specified, so that it can be used for the later testing. 4.3 Model development In this step, the model is developed and the functionality described by the previously elicited requirements is implemented. Unit tests are added to test smaller individual parts of the model. Additionally, the execution of both unit tests and acceptance tests should be automated in order to enable efficient and iterative confirmation on whether tests are passed. This step is part of the ATDD process of implementing functionality that passes the acceptance tests defined in step 2 in the flowchart in Figure 4.1. 4.4 Calibration, Verific