Applying and Evaluating Large Language Models for Triage at a Paediatric Emergency Department in a Swedish Hospital

Abstract

This thesis aims to explore and evaluate a software system using an LLM at the Paediatric Emergency Department (PED) at Sahlgrenska University Hospital. Approximately 60,000 patients visit the PED annually, while reports of decreasing staff availability and increases in burnout are observed. LLMs have shown potential in medical tasks, however, there is limited knowledge on how they would perform in a real setting. This thesis explored LLMs for streamlining the triage process to address this problem. Design Science Research was applied through three iterations involving interviews, prompt engineering, system-level simulations and human evaluations with nurses and voluntary patients.

15 functional and 10 non-functional requirements covering aspects such as accuracy, relevancy, usability and regulatory compliance were elicited from the stakeholders: nurses, the head of section, data scientists, and infrastructure providers. These were translated into a prototype using four instances of Llama 3.3 70B Instruct with Retrieval-Augmented Generation (RAG), each handling tasks such as generating follow-up questions, suggesting clinical controls and tests, or summarising information. The prototype demonstrated potential to support the triage process in 80% of the cases, showed particularly promising results in terms of accuracy when suggesting controls and generating relevant questions. However, it also exhibited certain limitations. Implementing LLM systems in a PED requires further research, especially on validating information completeness and how the RAG document structure and content affect accuracy.