Bridging Trust and Design of a Multi-Agent LLM-Based HR Chatbot: For the Times They Are A-Changin’

Abstract

Introduction: The integration of Large Language Models (LLMs) into workplace systems presents significant opportunities, particularly in the domain of human resources (HR), where repetitive tasks—such as providing information that employees could retrieve themselves—are common and could potentially be replaced by an LLM-based solution. However, a lack of user trust remains a major barrier to the adoption of LLM-based systems.

Objective: This thesis investigates what trust factors exist in LLM-based systems and how they can be addressed by system design, with a specific focus on a multiagent HR chatbot.

Method: Using a Design Science Research methodology, the study was conducted in two iterative cycles. Cycle I identified trust factors through literature review and interviews with six employees at a multinational company. It also included a workshop with five AI experts to discuss and validate design choices. Cycle II involved implementing, and evaluating an artifact, a multi-agent chatbot tailored to HR queries.

Findings: Thematic analysis revealed external trust factors: transparency, organisational measures, and external security and internal trust factors: internal security, model differences, risk of bias and reliability, which emerged as the most critical trust factor. The artifact was evaluated through interviews and metrics such as answer relevancy, faithfulness, and robustness, showing consistently strong performance and broad user acceptance.

Conclusion: The multi-agent HR chatbot effectively addressed key trust concerns and was positively received by most interviewees, demonstrating its potential for real-world application. These findings suggest that trust factors can be meaningfully addressed through thoughtful design and should be treated as a core consideration throughout the development process of LLM-based systems.