Adoption of Robot Framework for Software Download testing Development of tool for Testing Software Downloads and Diagnostics in Automotive Electronic Control Units Degree Project Report in Computer Engineering Henning Bjersander DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING CHALMERS UNIVERSITY OF TECHNOLOGY Gothenburg, Sweden 2024 www.chalmers.se www.chalmers.se Degree project report 2024 Adoption of Robot Framework for Software Download testing Henning Bjersander Computer Science and Engineering Chalmers University of Technology Gothenburg, Sweden 2024 Adoption of Robot Framework for Software Download testing Henning Bjersander © Henning Bjersander, 2024. Supervisor: Robert Krook, Computer Science and Engineering Examiner: Nicholas Smallbone, Computer Science and Engineering Degree project report 2024 Department of Computer Science and Engineering Chalmers University of Technology SE-412 96 Gothenburg Sweden Telephone +46 31 772 1000 Typeset in LATEX, template by Kyriaki Antoniadou-Plytaria Gothenburg, Sweden 2024 iii Abstract This degree project had the goal to improve the test reports and test case creation in the software download pod, which is a test environment designed to be able to execute various diagnostic tests without requiring expensive or complex hardware. One of the primary objectives was to allow for the creation of html-formatted reports for better visualization in the development flow. This was done by moving the test setup from a hard-coded python script to using Robot Framework. The project, also called the Robot Controller, was completed to allow for easy triggering by the build server and two test cases covering the main abilities of the environment were created. Keywords: Robot Framework, Testing Contents 1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Technical Background 4 2.1 Software download Pod . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Robot Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Continuous Integration Test Suite . . . . . . . . . . . . . . . . . . . . 5 3 Methodology 7 3.1 Setting up environment and reading the code base . . . . . . . . . . . 7 3.1.1 SWDL-Wrapper . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.1.2 Robot-Controller . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Coordinate work with stakeholders . . . . . . . . . . . . . . . . . . . 8 3.3 Configure SWDL-wrapper script . . . . . . . . . . . . . . . . . . . . . 9 3.4 Finalize Robot Controller . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5 Parse SWDL log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.6 Create Robot Framework test cases . . . . . . . . . . . . . . . . . . . 11 3.6.1 SWDL test case . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.7 Diagnosics test case . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.8 Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4 Result and Discussion 13 5 Conclusion 15 A Appendix I A.1 Appendix 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I v 1 Introduction 1.1 Background In the last years, many car companies have adopted a Star network architecture when it comes to their electrical engineering[1]. The star topology network topology is a network architecture that is structured like a tree with a single often powerful central computer to which most data is sent. Volvo is one of these companies and started its transition over to a Star network topology with its new SPA2 (Scalable Product Architecture 2) architecture. In Volvo’s case, there are a number of smaller ECUs (Electronic Control Units) on the edge of the (network both physically and topologically) which in turn are connected to larger ECUs designed specifically to forward the data to the central VCU (Vehicle control unit) by taking in data from slower CAN and LIN networks and converting it to Ethernet traffic. In the process of developing this new architecture, Volvo has also put a focus on doing a larger portion of their development in-house as opposed to outsourcing it. This has resulted in a large need for new test equipment and other environ- ments for testing/development. Recently there has been a focus on CI (Continuous Integration) which is the practice of frequently integrating to a main branch and automatically testing builds for a project to create faster development loops and allow for earlier identification of issues. The use of CI in software development is a widespread trend which comes with many upsides such as simpler merges and bet- ter reliability, but also several difficulties tied to the complicated test environment required [2]. At Volvo, this push for CI and more automated tests has led to teams coming up with different solutions to get more test resources faster and cheaper. HILs (Hardware In the Loop) are test environments that are heavily used in the automotive industry. They consist of one or more devices under testing, often re- ferred to as DUTs, along with hardware and software to be able to simulate or control the DUTs. This environment is often the place where new software builds are first run on their target hardware through the use of automated regression tests. Fur- thermore, HILs often offer a variety of powerful testing tools such as fault injection and the ability to simulate all data from other ECUs [3]. These HILs however are expensive to use since they require significant hardware that is capable of measur- ing on hundreds of physical pins and network busses. They also require experienced testers to maintain them and can take a long time before they’re operational. For this reason, many teams at Volvo have been moving to add smaller, simpler, and 1 1. Introduction cheaper systems that can perform some of the testing to free up more time in the more advanced test environments. These systems have also been used in the CI as feasibility tests to evaluate if it’s worth performing longer automated tests. In the development of the SPA2 platform the teams in charge of the LPC (Low power controller) and FIOC (Front input-output controller) platforms have, likely due to the increased in-house development of the base software layer, had some issues with downloading software to their ECUs. This has in turn led to issues in the automated regression tests being run in the HILs where in some cases software builds would lead to the HIL ending up in an unrecoverable state, requiring someone to manually solve it. Instead of buying more hardware and expanding the capabilities of the HILs, the team instead decided to create a gating test that the software had to pass before being passed on to the main suite of tests in the HIL. This would not only act as a gate for merging the software into the main branch, but also as an exit criterion for the testing which is a good way to avoid testing too much [4]. These test environments came to be called SWDL-Pods (Software download pods). This was beneficial both by blocking unviable softwares from taking up any testing time in the HILs and also by preventing buggy softwares from causing problems in the testing environment. The way these SWDL-Pods were implemented by the team was by using a very simple environment containing only the ECU being tested, a power supply, a headless computer, and the hardware to simulate the main network performing diagnostics. A script was added that triggered on finished builds from the build server which downloaded the software into the unit twice and reported back whether the software download tests were successful or not. Seeing the success of this simple but useful test environment, the team decided to expand the ability to create test cases and also create proper test reports. For this task, the Python framework Robot Framework was chosen since it supports the creation of html-formatted test reports and also uses Python which was used for the initial script. 1.2 Objective The objective for this project was to complete the development of the SWDL-pods using Robot Framework, and allow for the creation of HTML-formatted test reports as well as creating two initial test cases for the FIOC ECU. These test cases are primarily designed as examples to allow further implementation of more test cases in the SWDL-Pod. One of the test cases should be a diagnostic test case while the other one should be a SWDL (Software download) test case. It should be done in such a way that it is easy for other developers or testers to create new tests for either SWDL or diagnostics. 2 1. Introduction 1.3 Delimitations These delimitations were chosen for this degree project to scale the project down to suite the allocated time. 1. The scope is limited to the SWDL-Pod test environment; the test cases and surrounding code only need to apply to that environment. 2. Only the FIOC is supported by the implementation and test cases even if support for other nodes may be easily added. 3. Testing of the full test system will only be performed manually due to time constraints and the increased complexity. 3 2 Technical Background 2.1 Software download Pod The SWDL-Pod is a purposefully simple test environment that only has the goal of verifying that a software build can be properly downloaded to an ECU. It’s a rack-based environment that allows the stacking of several SWDL-Pods in a single cabinet. The SWDL-Pod is consists of a computer running the test scripts, processes and interfacing with power supply and hardware for network communication, as seen in figure 2.1. The computer is headless and can be accessed remotely, but is generally not designed to be controlled manually. For communication with the ECU different hardware will be necessary depending on the network used for diagnostic data. For the FIOC, Vector hardware is used to communicate on the CAN network and allows for logging of the networks or passing the data to other applications through Vectors API (Application Programming Interface), which allows the user to read and output messages on the CAN bus. Parallell Diag is a locally developed tool which is used for testing of diagnostics and software download. It is used to perform the SWDL and other diagnostic tasks; it is controlled through an API. Since the FIOC isn’t connected directly to the diagnostic port, ECU sim is used instead which is a locally developed tool to format the doip messages to follow the syntax and addressing of the upstream ECU instead of from the diagnostic tool. The value of this environment is its simplicity and low price. This allows it to be quickly deployed at a low cost. It was designed to create a set of tests that can stop any builds with poorly functioning software download functionality before letting testing move on to the HILs which are often in high demand due to their versatility. 2.2 Robot Framework Robot Framework is a Python-based test framework that can, through the use of libraries from a variety of languages, test implementations ranging from embedded systems to web applications. One of the main features of Robot Framework is that it uses keywords in a tabular format to create the test cases[5]. These keywords represent different actions that can range from more simple things such as assertions, starting an IF statement, or throwing an error to complex actions like running functions from imported libraries. This makes the test cases easily readable which gives the users a better overview of the test cases and allows for better cooperation between team members with different technical knowledge. 4 2. Technical Background Figure 2.1: Original SWDL-Pod structure Robot Framework also allows for the creation of detailed and informative test reports which can be easily viewed and understood. There’s also the option to customize the reports in various formats such as HTML and XML. The reports can be detailed using Robot Framework’s flexible options for logging which allows for further customization of the test reports. The organization of test cases is quite easy thanks to the modularity of the frame- work which allows the test cases to be simply organized into test suites; or, if there’s a need for more involved test case management, then the Robot Framework API can be used to allow another program to put together a test suite based on tags or other qualities. 2.3 Continuous Integration Test Suite For the testing of software for the LPC and FIOC, there’s a continuous integration test suite that is there to help automate the testing and also allow developers to faster get feedback on their commits [2]. When a new software has been built, the build server triggers a set of automated test sets in different test environments. On a new trigger, the test sets are added to a queue where they will wait until the required hardware is available to perform the tests. The types of test sets depend on the type of build triggering the suite, where builds in smaller development branches may only trigger a few very fast test sets while builds meant for release may trigger more test sets. There are also several tools that can be used by the teams to control the queue, such as the ability to cancel or prioritize certain builds. Every software build will always start triggering the software download tests that are run in the SWDL-Pod. These tests exist mainly as a way to stop any builds 5 2. Technical Background which have a risk of putting the hardware into an unrecoverable state in the later tests and to stop softwares with issues in the software download process. This is the only test set that will stop further testing of the build if any of the test cases fail. After the software download test has passed, the Smoketest is run; it tests basic functionality such as making sure communication works on the required networks and other important functionality such as diagnostics and IOs (Inputs/outputs) works in the most basic cases. The Smoketest and all the further tests are performed in whichever HIL is the first one that is compatible with the tests and available. The test suites after the Smoketest are requirement-based and each tests one or several specific requirements imposed on the ECU. These test sets are divided into different groups based on how long the sets should take to run and other factors such as requiring specific hardware to run or the tests being owned by other teams that may have their own test schedules. 6 3 Methodology The project was started by gathering information to define an object and lay out the different steps in the project. After meetings with the developers of the SWDL-Pod, a general plan was laid out. For structuring the project an agile approach was used with weekly increments. The program Notion was used to create tasks and organize them on a weekly basis [6]. In the beginning, only the larger overarching parts of the project were added but as each week started they were split into smaller tasks. 3.1 Setting up environment and reading the code base Visual Studio Code was used as the IDE (Integrated development environment) and Pythons package manager Pip was used to resolve all dependencies [7]. After getting the development tools ready, the next and significantly larger task was to understand the existing code. Two main areas of code were directly related to the SWDL-Pod: the SWDL-wrapper (Software download wrapper) Python script, and the new Robot Controller Python scripts. The SWDL-wrapper script was the part that was driving the current functionality of the SWDL-pod, whereas the robot controller scripts were meant to take over the functionality of the SWDL-wrapper to be able to run several different usermade testcases than the specific testcase that the SWDL-wrapper script was designed for. These two scripts were written by different developers and with very different styles. The SWDL Wrapper was designed more to get the job done with very little flexibility, where the functions took in large dictionaries that weren’t necessarily left unchanged, while the Robot Controller was designed in a very structured manner where the functions had very narrow and specific roles. This difference in style made combining the two scripts more difficult due to not being able to fully adhere to the style of either script and many of the functions would change the input variables which made them difficult to use. Aside from these two scripts, there was also code for triggering the job from the build as well as functions to return the results to the rest of the test chain. These did not need any significant change to implement the complete robot controller however and were left out of the scope. 7 3. Methodology 3.1.1 SWDL-Wrapper The SWDL-Wrapper script held all the functionality to start the prerequisite pro- grams and perform the SWDL and diagnostic communication with the ECU. This script did things like kill Windows processes and restart them to get to some form of default state. These Windows processes were in charge of controlling the power sup- ply, performing diagnostics communication, and other networking tasks were then controlled by the SWDL-wrapper using their APIs (application programming inter- face). The SWDL-wrapper was written with a scripting style, using only a single file with several thousand lines of code. It also worked mainly by passing around large dictionaries to its functions; often these dictionaries could be edited in the functions. Approximately half of the parameters in these dictionaries were paths to the location where the programs that it controls were located on the disk and the remaining contained more concrete information of the test. This made it hard to isolate specific functions without taking them apart completely. 3.1.2 Robot-Controller The Robot Controller is the main driver of the tests; it gathers the test cases based on the tags supplied by the trigger. It also handles taking in the other relevant arguments which might differ depending on the ECU being tested and whether to upload the results to the CI board or just send a mail with the result. It was also designed in a way to allow for further expansion in the future. It had been a work in progress and had never been fully tested. Everything was written in Python and in a very object-oriented manner, following styles often seen in object-oriented languages such as C# or Java with many small files containing a single object or similar structure. Even considering it wasn’t quite complete this made it simple to follow and take in. Even if the full system was untested there were several unit tests for almost every module. Many of the planned modules were also either empty or not yet created. 3.2 Coordinate work with stakeholders While getting situated with the code, a number of meetings were created with the two main stakeholders in the project to help define specific more specific goals for the project and support with answers to questions. For the first few weeks, these meetings were more similar to demos where the developers of the existing scripts would go over the current functionality and explain the rationality that went into their solutions. Later in the project, more discussion-style meetings took over where the discussion was focused on deciding more in detail how they envisioned the system looking in the future. This brought up questions on how to make it easy to add different features to the test solution in the future, how to make it robust, and how to make it easy to create test cases using this solution. 8 3. Methodology 3.3 Configure SWDL-wrapper script After testing and verifying that the SWDL-wrapper script worked as intended, a discussion was initiated with the stakeholders on how to proceed with the script. It was more complex than was necessary and many of the functions were designed in a way to take significantly more inputs than was needed for the Robot Controller tests. This was because the SWDL-Wrapper script was originally designed to be stand- alone and roughly 15 input arguments were required to trigger a test depending on the ECU. The two main alternatives were to either write a new script to interact with the relevant processes or to use it as a library and just pass in dummy inputs in cases where the functions were more complex than necessary. Rewriting it would have required a significant amount of time to understand the different processes and APIs being used to control the SWDL-Pod. It was ultimately decided that the SWDL-wrapper should be used as a library. The primary reason for this choice was the inflexible nature of the script which would have made it difficult to reuse any significant parts, causing the script to require a full rewrite of the code which would have taken too long for the scope of the project. Instead, an intermediary layer was created which imported the SWDL- wrapper script and created Robot Framework Keywords that could later be used in the test cases. In some cases, it wasn’t enough to simply call the functions using dummy inputs, since some of them did too many things which might be undesirable. In these cases, their functionality was simply copied out to the intermediary layer where new functions were created. One of the functions that had to be rewritten in the intermediary layer was the one responsible for triggering the diagnostic tool Parallell Diag to download software, because in the SWDL-wrapper it was hard- coded to cause several downloads instead of the expected one. This middle layer was created using Python and the Robot Framework decorator to create Keywords. Keywords for validating input data, clearing the FIOC, per- forming downloads, and parsing the Parallell Diag log were created. These were in large part created by assembling the required functionality based on the SWDL- Wrapper script. The new structure containing the Robot Controller and the middle layer with the rest of the environment can be seen in this figure 3.1. 3.4 Finalize Robot Controller The first step of this process was to spend some time in an empty Robot Framework project to create simple extra test cases. After that, some time was spent getting the existing parts of the Robot Controller to work for some very simple test cases. This revealed several different bugs and other unintended functionality in the existing code base. These had to be resolved before any further work could be done. Some of the bugs were related to how the Robot Controller recursively looked for test case 9 3. Methodology Figure 3.1: New SWDL-Pod structure files, as well as a bug in how the Robot Controller initiated Robot Framework to run the test cases. These issues were resolved in pair programming sessions with the original developer. Among the modules that were missing and in need of implementation was way to download and extract the software artifacts. To achieve this, an Artifactory Client was created which utilized the Artifactory Python library to download the software artifacts from the JFrog Artifactory repository [8][9]. It then used the default tarfile Python library to extract only the relevant download files. It was also designed to delete the files after every download to ensure there is only one set of files at a time to both save disc space and reduce clutter. A way to start the Robot Controller was also needed. For this, the Python library Argparse was used to collect the arguments needed to know which tests to run and on what software build [10]. This allowed the user to start the Robot Controller by inputting the software build number, ECU type, and test case tags. The tags are added to the test cases and the only test cases that will be run when the Robot Controller is triggered correspond to the chosen tags. After collecting the input arguments, the Robot Controllers Test Case Investigator is triggered to find any test cases that correspond with the chosen tags in any file in the location designated for the test cases. 3.5 Parse SWDL log After the SWDL test case had performed its download, a way was needed to some- how determine a verdict. Previously the verdict was simply checked by opening the log provided by Parallell Diag and checking that the fault count is 0 and just re- turning a pass or a fail. Here it was requested that the report contain more detailed information about the SWDL attempt, so a log parser was written. Since the log file has a very rigid structure regex’s were the primary parsing method [11]. The data was split into two parts: the download settings and the results. The download settings could be shown as a simple dictionary and were simply added to the start of 10 3. Methodology the test case’s report section using the Python Logging library. For the results part, the result of each download was parsed and packed into a Python data class using the data class decorator. When writing the result to the test report, each failed download would have all its log data added as errors using the Logging library. The successful downloads log data would be written to the test report as info hiding and preventing clutter of the test report. 3.6 Create Robot Framework test cases To create the two test cases for diagnostics and software download several, Robot Framework Keywords were first created to support the test cases. Due to the fact that the SWDL-Wrapper script was used as a library some of the functions took in large Python dictionaries containing all the parameters that would normally be passed as arguments when starting the script. Most of these parameters were related to paths of different executables or similar and would not need to be placed in the test cases themselves. These parameters were created as global variables as a way to prominently display them at the top of the test file and make them easily available to all test cases. A Keyword was then created to return the input dictionary for the SWDL-Wrapper using these global variables. A Keyword was also created to validate the input dictionary. It worked by calling a function from the SWDL- Wrapper designed for the reason of validating all the paths and testing the data types and values of the other parameters. 3.6.1 SWDL test case The test case for SWDL was created by starting with the Keyword to create the input dictionary and setting the number of downloads to 1. The Keyword to validate the input dictionary was then called, and if the result was false the, test would log a Fatal error and abort the test case. This was done to prevent users from being able to input invalid inputs and causing a crash in the system. The input dictionary would be sent to the SWDL Keyword which executes function from the SWDL wrapper to to perform the software downloads but with further flexibility. After the requested SWDLs had been completed, the log parser would then be called to output the download logs generated by Parallell Diag to the test report. The SWDL Keyword would then return both a result and an error code that used to assert whether the test case is a pass or fail. As a whole, the SWDL test case was quite similar to how it functioned when the SWDL-Wrapper performed the test itself, with the main difference being that the new test would generate a useful test report. This new test structure also allowed for more flexibility when testing SWDL, such as being able to run diagnostic requests conveniently between each SWDL. 3.7 Diagnosics test case Creating a test that performs one or several diagnostic requests requires creating a Parallell Diag sequence file. This is a file that contains a set of diagnostic requests 11 3. Methodology and ECU addresses that can be loaded by Parallell Diag which in turn executes those commands and generates a log file based on the responses. For the previous SWDL-Pod tests, there were only two sets of diagnostic requests that ran, depending on which ECU was being tested so there was no functionality to generate these sequence files. This meant that the first course of action was to create a Python script that took in a list of diagnostic requests and an ECU address. These were validated to be possible requests using a regex and then written out to an XML file in the correct format using Pythons XML library’s Element Tree functionality. This was then imported in the middle layer between the Robot Framework test cases and the SWDL-Wrapper, and a Keyword was created that takes in a list of Diagnostic requests and expected response values. These response values were given in the form of Regex patterns so that future test cases could be created by either specifying the specific expected response or more complicated patterns that could be used to accept many different responses in case an analog sensor is tested for example. The Keyword then parsed the generated log and wrote the details to the test report in the same way as the SWDL test case. To turn this into a complete test, some input validation was added to stop any invalid tests from running on the hardware. Then as seen in listing A.1 the Robot Keyword takes in a list of tuples containing the diagnostic request and regex pattern to validate it against. 3.8 Verification To verify the test setup, two different tests we performed for the SWDL test case and the Diagnostic test case. The SWDL test case was verified by running a passing test and a failing test. The passing test simply worked as expected with the test report showing pass and all of the SWDL logs being hidden. The failing test was performed by rerunning the same test but disconnecting the CAN network during the download process. This was done by stopping the ECU sim simulation which in practice is similar to disconnecting the CAN wires. Doing this caused the test case to fail and caused the test report to show fail and also show the error code for aborted download. SWDL fails were also tested by manually cancelling the download in the middle of the process which also yielded a fail with the errror code corresponding to user cancellation in the report. The code to test the Diagnostic commands was verified by altering the test case to expect the incorrect result from the response which caused the test to fail with no specific code but with the response highlighted in red. Verification of the diagnostic test case was done in a similar manner with one passing test and one failing test. For the diagnostics, the failing test was created by inputting a request together with an incorrect response. This failed the test case and wrote the expected as well as the actual response in the test report. Finally, a code review was held to lower the risk of unforeseen problems with the solution. A large reason for this was that creating unit tests for a significant part of the project would have been needlessly complicated for the value gained. 12 4 Result and Discussion The primary objective of the project was to complete the implementation of the Robot Controller test solution for the FIOC ECU. Furthermore, there were also the objectives of creating a pair of example test cases for diagnostics and SWDL using Robot Framework. The project successfully achieved these objectives. If imple- mented in the live build chain, this solution will improve the ability for developers to more easily debug faulty builds relating to software download as well as easily implement different diagnostic test cases. The example test cases also allow users to very easily create a test run that suits their needs, since the test case can very easily be copied and simply have the number of downloads changed. This would make the process of running a weekly SWDL robustness test consisting of 1000 downloads or more as simple as creating a test case and triggering it. Similarly the diagnostics example test case allows users who want to create some test cases to simply copy the test case and change the input diagnostic request and response strings which is good when not all users are expected to know Robot Framework. Robot Framework is a relatively simple to use and well-documented test frame- work, but it comes with some significant downsides such that choosing another test framework could be discussed. The reasons for creating this new test setup were pri- marily to create better and more detailed test reports in an HTML format that could be uploaded to the CI along with the test reports from other environments. The secondary reason was to enable simpler test case creation for anyone who would want to implement diagnostic tests or other applicable tests. For this secondary reason, the Robot Framework may not have been the best option, since it adds a requirement of knowledge in its relatively unique syntax. The syntax for writing test cases with Robot Framework is relatively simple but does still require learning something new which always takes some amount of time and reading documenta- tion. So this would present an unnecessary barrier for anyone looking to create a simple test case for a simple environment. Many different testing frameworks for Python could have been used in its place, but to allow for easier test case creation a framework such as PyTest might have been more appropriate. PyTest uses Python to write new test cases which can be seen as a positive since Python is used heavily in all environments at Volvo Cars, so a user is likely to be familiar with it already. PyTest can also generate reports in HTML format using the PyTest-html plugin which covers the primary reason for this project[12]. When it comes to the use of the SWDL-Wrapper, a thing that could have been considered would have been to rewrite it completely instead of using it as a library. 13 4. Result and Discussion Doing this was discussed but was deemed to take too much time for the scope of this project. There would however have been some significant upsides to just per- forming this work straight away. The most apparent upside is that it would be removing needless complexity in the system as a whole largely by removing parts made redundant by the new code interfacing with Robot Framework. This complex- ity reduction would also allow for the implementation of new features which may be required if new ECUs were to be added or some used process or hardware needs to be changed. Rewriting the SWDL-Wrapper would also effectively serve as a refac- toring which brings many benefits such as improving the code, removing potential bugs, and increasing the overall quality [13]. Finally, it could also achieve several speed improvements in the tests since the SWDL-Wrapper’s functions perform sev- eral things that may not be needed for the Robot Framework test case. This was discussed again after the project was completed and due to these benefits, this will likely be rewritten sometime in the future. Aside from the ability to create test cases, the option more easily add support for other ECUs was also a significant reason behind the project. This was not part of the objective however as it would require involving another team to support with supplying hardware, software and all details regarding how to communicate with it and perform other diagnostic actions. A large part of the functionality required to test a different ECU has been implemented however. The Robot Controller is versatile enough to create test cases and tags for different ECUs. This means that, by creating a Robot Framework Keyword for performing SWDL and diagnostic communication other teams can also use this system for their basic testing. 14 5 Conclusion This project achieved its goal of improving the test reports from the SWDL-Pod as well as allowing users to create new test cases with relative ease. This transition from using a hard-coded script with only a pass or fail response as a test into a more flexible test environment will likely lead to increased interest in using the SWDL-Pods and may also lead to further investigations on how these cheap test environments can be expanded further. To create a stable base to build on, however, it will likely be worth it, in the long run, to rewrite the SWDL-Wrapper script to create a uniform and easily expandable code base. This could prove especially useful if the team plans on increasing the number of SWDL-Pods in which case it is likely good if the code is easily understood. Furthermore, the completion of the Robot Controller allows for its use in new test environments for other specialized use cases. The completion of the Robot Controller could also be used in the testing of other ECUs since it’s supported through the use of tags even though some Robot Framework Keywords may have to be created based on which network is primarily used for diagnostics. 15 Bibliography [1] J. Schäuffele and T. Zurawka, 2.5.3.1 Star Topology. SAE International, 2016. [2] O. Elazhary, C. Werner, Z. S. Li, D. Lowlind, N. A. Ernst, and M.-A. Storey, “Uncovering the benefits and challenges of continuous integration practices,” IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2570–2583, 2022. [3] P. King and D. Copp, “Hardware in the loop for automotive vehicle control systems development,” in 2004 Mini Symposia UKACC Control, pp. 75–78, 2004. [4] T. Toroi, A. Raninen, and L. Väätäinen, “Identifying process improvement targets in test processes: A case study,” in 2013 IEEE International Conference on Software Maintenance, pp. 11–19, 2013. [5] R. F. Foundation, “Builtin,” 2023. https://robotframework.org/ robotframework/latest/libraries/BuiltIn.html. [6] I. Notion Labs, “Notion,” 2024. https://www.notion.so/. [7] Microsoft, “Visual studio code,” 2024. https://code.visualstudio.com/. [8] K. Nazarov, “A python to artifactory interface,” 2016. https://pypi.org/ project/artifactory/. [9] J. Ltd, “Jfrog artifactory,” 2024. https://jfrog.com/artifactory/. [10] T. Waldmann, “Python command-line parsing library,” 2015. https://pypi.org/ project/artifactory/. [11] T. Stubblebine, Regular expression pocket reference. [electronic resource]. O’Reilly, 2003. [12] D. Hunt, “Pytest-html,” 2020. https://pytest-html.readthedocs.io/en/latest/ index.html. [13] R. P. S. P. Veerraju, A. S. Rao, and G. Murali, “Refactoring and its benefits.,” AIP Conference Proceedings, vol. 1298, no. 1, pp. 645 – 650, 2010. 16 https://robotframework.org/robotframework/latest/libraries/BuiltIn.html https://robotframework.org/robotframework/latest/libraries/BuiltIn.html https://www.notion.so/ https://code.visualstudio.com/ https://pypi.org/project/artifactory/ https://pypi.org/project/artifactory/ https://jfrog.com/artifactory/ https://pypi.org/project/artifactory/ https://pypi.org/project/artifactory/ https://pytest-html.readthedocs.io/en/latest/index.html https://pytest-html.readthedocs.io/en/latest/index.html A Appendix A.1 Appendix 1 Listing A.1: Diagnostics test case Current Primary Boot loader [ Tags ] FIOC ${diagnostic_commands}= Create L i s t . . . #Check Normal mode . . . ${{ ( ’22 F1 86 ’ , ’ 62 F1 86 01 ’ ) }} . . . #Check PBL . . . ${{ ( ’22 F1 21 ’ , ’ 62 F1 21 32 28 76 77 20 41 46 ’ ) }} ${ user_input_dict } Create User Dict ${ user_input_correct } Check User Input ${ user_input_dict } IF ${ user_input_correct } #Send Diagnos t i c commands ${ r e s u l t }= Test D iagnos t i c s . . . user_input_dict=${ user_input_correct } . . . diagnostic_command=${diagnostic_commands} Should Be Equal ${ r e s u l t } ${True} ELSE Fatal Error User input i s i n c o r r e c t . END I Introduction Background Objective Delimitations Technical Background Software download Pod Robot Framework Continuous Integration Test Suite Methodology Setting up environment and reading the code base SWDL-Wrapper Robot-Controller Coordinate work with stakeholders Configure SWDL-wrapper script Finalize Robot Controller Parse SWDL log Create Robot Framework test cases SWDL test case Diagnosics test case Verification Result and Discussion Conclusion Appendix Appendix 1