Leveraging AI for Automated Requirement to Test Alignment in Automotive Embedded Systems - A Design Science Research in the Automotive Industry

Abstract

The rapid increase in the complexity of automotive software has made the manual verification of software traceability labour-intensive. Consequently, practitioners can encounter “alignment gaps”, where a formal traceability link exists between a requirement and a test case, but the test case fails to exercise the requirement’s intended functionality. To address this challenge, this thesis develops a diagnostic artificial intelligence (AI) artefact. The artefact is aimed to automate the assessment of requirement-to-test alignment within the automotive embedded systems domain. The study employs a three-cycle Design Science Research methodology. In the first cycle, a literature review and practitioner interviews were conducted to identify design attributes for the proposed artefact. Four design attributes were determined: semantic similarity, contextual information, structural consistency, and hierarchical dependencies. During the second cycle, a system was developed utilizing Large Language Models (LLMs) integrated with a Retrieval-Augmented Generation (RAG) architecture. In the final cycle, the artefact was evaluated through a mutant injection analysis to measure its fault detection capabilities, alongside a case study involving industry professionals to assess its practical utility. The mutant injection analysis demonstrated that the RAG architecture improved its analytical consistency and its ability to detect injected logical defects. Furthermore, the case study, which utilized the NASA-TLX questionnaire and post-task interviews, indicated that the artefact reduced the general workload associated with evaluating test case alignment. How ever, due to the small sample size and sampling limitations, these findings should be interpreted as suggestive. Ultimately, this thesis concludes that combining RAG architectures with LLMs represents a promising proof-of-concept for transparent decision support in maintaining requirement-to-test alignment in safety-critical automotive environments.