Automated Alignment of Software Requirements and Test Cases

Abstract

Software testing can be expensive, but there are numerous tools available to enhance its efficiency. Nevertheless, certain areas still lack adequate technology. To address this, we conducted a qualitative study to identify shortcomings in existing tools and determine what features are needed in new ones. Through Thematic Analysis of the interviews conducted, we discovered that system-level testing stands out as an area in need of improvement, with a particular emphasis on the importance of well-defined requirements for effective testing. In light of this, we propose the development of a tool that assists testers in aligning their tests with requirements and generates suggestions for new tests using a powerful Large Language Model (LLM). The tool demonstrated great efficacy in terms of accuracy and recall, achieving an average of 86.394% each. It also provided valuable suggestions for testing approaches related to requirements. However, we observed limitations in its ability to achieve perfect alignment between requirements and test cases, as it showed a tendency for false positives. This led to the tool having an average precision of 45.582%. Moreover, the tool exhibited high efficiency when handling smaller input sizes. However, as the input size increased, we noticed a more than linear growth in analysis time, suggesting potential scalability challenges.