Automated GUI Testing: A Comparison Study With A Maintenance Focus

Abstract

Automated GUI (Graphical User Interface) tests can alleviate work from testers, making it beneficial to convert manual test cases into automated GUI tests. However, automated GUI tests come with costs and drawbacks not found in manual tests. These limitations can differ between automated GUI testing tools. Two such tools are Selenium and EyeAutomate. The tools differ in their ways of locating GUI components, with Selenium utilising underlying information about a web page and EyeAutomate relying on image recognition. For a practitioner deciding to adopt either tool, or similar ones, it is a benefit to know the strengths and weaknesses of them. This study has investigated general differences, implementation cost, maintenance cost, return on investment, and the defect-finding capabilities of Selenium and EyeAutomate. These properties were examined by subjecting tests written in each tool to system changes using version control history. Additional capabilities were determined by using manual fault injection. Qualitative data concerning the tools and automated GUI testing were collected using interviews. Results indicate that while EyeAutomate tests are quicker to implement than Selenium tests, they require more time to maintain. Both tools have a similar return on investment, being able to reach it within one year compared to running a manual test suite weekly. The tools are comparable when finding defects during system development, with EyeAutomate being able to find more purely graphical related defects.