Enhancing the User Experience for AIdriven Complex Knowledge Systems with Natural Language Interfaces

Abstract

While user experience (UX) in AI systems and natural language interfaces has been explored in previous research and frameworks, few studies specifically focus on the business-to-business (B2B) context. Existing guidelines and design implications have primarily been shaped by leisure focused applications where they have stemmed from business-to-consumer (B2C) contexts.

In the context of designing for professional users, there is currently a need for UX design guidelines specifically targeting work related contexts. This thesis aims to address this gap by exploring what factors should be considered when seeking to improve AI-driven complex knowledge systems with natural language interfaces. It focuses on the needs of diverse professional users in B2B contexts and examines how UX design can address these factors. The study is guided by the following research questions: (1) What factors should be taken into consideration when seeking to improve AI-driven complex knowledge systems with natural language interfaces in B2B contexts? (2) What role can UX design play in addressing factors influencing the improvement of AI-driven complex knowledge systems with natural language interfaces in B2B contexts?

This study uses a mixed-method approach where qualitative data are collected through data logs, interviews, and a survey to draw insights from users in real-world B2B scenarios working with an AI- driven system provided by a threat intelligence company.

As a result of this research, eight factors were identified to affect the user experience of natural language interfaces for information retrieval in a B2B context. Eleven design guidelines are proposed to provide UX designers guidance in designing natural language interfaces that support user control and customization, trust and transparency, and communication of the AI’s abilities and limitations. The identified factors and proposed guidelines offer a foundation for future research in UX design for AI-driven systems in professional environments. This study invites for further exploration through validation across different B2B sectors and the identification of additional context specific design factors. This thesis contributes to bridge the UX research gap for AI-driven natural language interfaces in professional contexts by offering practical insights to support trust, effective information retrieval, and user friendly interactions.