Automated Usage Tracing and Analysis: a comparison with web survey

Abstract

A challenge in taking decisions on how to improve a software product is to gain knowledge on how end-users interact with it. One way of getting this knowledge is by asking them through a web survey. Another approach is based on tracing what the users do and then run an analysis on that gathered data. This paper presents an approach called Automatic Usage Tracing and Analysis (AUTA) to automatically gather usage data for a software system with Aspect-Oriented Programming (AOP) and analyzes the gathered data through data mining. A brainstorming workshop with the developers of the software system was used to define a set of questions that the data mining should answer. The questions were implemented in AUTA and a web survey was conducted to compare the two methods. The comparison showed that there is a resemblance between AUTA and a conducted web survey. However, the resemblance is not strong enough to conclude with certainty that AUTA can replace the use of web surveys. It was also discovered that some questions identified in the workshop were not possible to be answered with a web survey but could be answered with AUTA. The recommendation is therefore that AUTA and web surveys are used as complement to each other.