Evaluating the Impact of a Multi-Agent AI Assistant on Human Creativity in Exploratory Testing

Abstract

Exploratory Testing (ET) is an important approach for uncovering complex software defects, however its success is frequently impacted by human factors such as cognitive bias, mental fatigue, and rigid thinking patterns. This research investigates the impact of AI assistance on the creative performance of human testers. Instead of treating AI as a passive assistant, this study explores how creative stimuli can help testers break through cognitive bottlenecks and explore non-obvious failure scenarios. Using the WICKED multi-agent framework as an experimental platform, we conducted a human-centered user study involving participants with software engineering backgrounds. The research analyzes how real-time AI prompts influence the generation of test scenarios across multiple Software Requirement Specifications (SRS). We assess the interventions along two questions: their effect on the creativity of test artifacts (Quantity, Quality, Novelty), and their usability (Perceived Usefulness and Perceived Ease of Use). We find that WICKED did not reliably increase creativity. Particularly, (i) it did not raise the number of tests created, (ii) its effect on novelty was inconclusive, and (iii) quality was improved only on the more complex system, where it surfaced non-obvious faults such as a concurrency bug missed by manual testers. In terms of usability, participants found WICKED easy to learn and useful in principle, but its slow and unclear interaction limited its perceived usefulness. These results suggest that creativity chatbots are best targeted at complex, uncertain testing tasks and must protect early user trust through speed and relevance, indicating that successful adoption depends as much on the testers themselves as on the strength of the model.