LLM-based Log Analysis for Fault Localization in the Automotive Industry

Abstract

This thesis investigates the application of large language models (LLMs) to aid practitioners of log analysis for fault localization in the automotive industry. An existing LLM-based log summarization tool is extended and evaluated, focusing on the cognitive load of practitioners and how satisfied they are with the tool. The effect of LLM-based log summarization on productivity of practitioners in the automotive industry is investigated through a case study at a company within the automotive industry. Think-aloud sessions and semi-structured interviews are carried out to asses the impact of the tool on the fault localization flow of study participants. Results suggest that LLM-generated log summaries can aid practitioners by giving them a first glance of the issue, thereby potentially reducing manual effort and improving productivity. However, the results also suggest that the context of the issue, domain knowledge, and interactivity of the tool plays a major role for success. A lack of context and means for the practitioner to guide the tool could result in a less effective workflow with higher cognitive load. The thesis provides insights on the integration of LLM-based log analysis tools within fault localization workflows in the industry, highlighting both the benefits and challenges of deploying LLMs in real-world fault analysis scenarios