Supporting the generation of engineering analysis reports with Large Language Models

Abstract

At GKN Aerospace Sweden, engineers work hard to conduct simulations and analyses to verify the performance of their products. Being an aerospace company, the main product portfolio consists of aero-engine components, space systems and upgrades of military aircraft engines. The outcome of the analysis performed on these types of products needs to be documented in technical analysis reports. To go from the generation of a report to the final stamp of approval, 3 roles are involved: Author, reviewer and approver. The process often has long lead times and suffers from many unnecessary loops and iterations back and fourth between the roles. With the rapid rise of AI in the form of Large Language Models, there is an interest in evaluating how this technology can be used to aid the engineers in this process and reduce lead time. This is the main aim of this study. An important factor is that due to confidentiality, data from the company cannot be transferred to unauthorized LLM providers. This study conducts a thorough interview study with engineers responsible for the different parts of the process. A qualitative thematic analysis of the interview transcripts then resulted in a comprehensive process problem analysis clearly mapping the issues within. Literature research was carried out providing a wide overview of the recent advancements in the field of Large Language Models and inspiring implementations. Using the problem analysis and literature, several design criteria were defined and explored based on how the process can be supported using LLMs. Software was developed using the Langchain framework in Python, resulting in 12 different concepts to support the process. The outcome shows that LLM has the potential to generate reviews of the engineering analysis reports, producing promising results in five test cases created regarding the topics language, content, references and consistency. However, the study finds that in the cases where the concepts found a high proportion of report errors, there were also problems with the production of a lot of false or untruthful feedback. Further demonstrations of generating or retrieving parts of a report were developed and are presented as proofof- concept. Lastly, a prototype of a user interface visualises how GKN Aerospace engineers might interact with the developed functionality. The study strongly indicates that GKN Aerospace can benefit to a great extent from such solutions. All code implementations can be found at https://github.com/DaniSode/Master_Thesis.