An Exploration of Explainability for Internal Stakeholders: A Qualitative Study

Abstract

AI applications are becoming increasingly prevalent across various domains and industries. However, the challenge of comprehending the inner workings of ML/AI systems extends beyond end users and significantly impacts AI/ML developers and testers. This research investigates Explainable Artificial Intelligence (XAI) aspects and challenges concerning internal stakeholders. We conducted a qualitative interview study involving experts and researchers specializing in explainability and engineering AI-based systems. Our findings emphasize the importance of explainability exclusively for internal stakeholders, an aspect that has received limited attention in previous research. We identified research gaps in the following areas: the lack of exploration into how explainability can enhance AI/ML model testing, the need for further investigation into existing XAI evaluation metrics that align with diverse internal stakeholder needs, and the knowledge gap surrounding the concept of XAI and its integration into existing processes. Additionally, we present the challenges that internal stakeholders encounter when incorporating explainability features.

The key results show that explainability positively impacts testability, as it can serve as a tool for guiding the test process. There are several noticeable benefits of explainability methods (XAI) for both developers and testers such as explainability aids in debugging the model output, which is an essential aspect of error analysis, and in detecting potential biases in the data. Other benefits are discussed in this paper. Furthermore, there is a need for an accepted set of standardized metrics to assess the trustworthiness of explainability, which would evaluate the effectiveness of the explanations themselves.

Our study offers foundational work for future research and underscores critical research gaps. The ability to design explainability for internal stakeholders holds the potential to facilitate the development of complex, AI/ML systems.