Enhancing feasibility analysis through LLMs: An Action Research Approach

Abstract

Requirement elicitation is a critical part of Requirements engineering (RE). However, it is heavily reliant on the knowledge of domain experts, is prone to human-errors, and remains a time-consuming process. This study investigates how Large Language Models can assist domain experts in conducting feasibility analyses of customer demands. Specifically, the study investigates two different solutions as an LLM; the Retrieval-Augmented Generation (RAG) architecture, and the finetuned pre-trained LLM. The two models were developed using the action research methodology in collaboration with Siemens Energy AB. All the data used for training the models were related to historical gas turbine projects and were provided by Siemens Energy. The findings showed that the finetuned LLM struggled with ambiguous requirements and was prone to hallucinate for a certain type of customer demands, in particular demands that were covered by Siemens Energy’s standard. In contrast, the RAG demonstrated a higher accuracy and relevance in its outputs. The two models were evaluated through a survey, which was answered by the domain experts at Siemens Energy. The surveys revealed that both of the models have potential as a decisionsupport tool, but that the RAG was preferred since it outperformed the finetuned LLM in all of the metrics. Lastly, a finetuned embedding model was developed as part of the RAG solution. This embedding model was quantitatively evaluated and compared to state-of-the-art models. This evaluation showed that the fine-tuned model outperforms state-of-the-art benchmarks on the intended task and in the environment of Siemens Energy.