Noise Handling for improving anomaly detection in application logs

Abstract

Application logs can often play a vital role in maintaining information systems. However, many of the applications in production environments are generating more and more logs. This can make it particularly difficult for users to locate the needed information in a short time. As anomalies can appear in the application logs and the symptoms of the problems may lead to some serious consequences such as huge economic loss, it is crucial to detect them effectively. The cases that successfully apply machine learning to solve problems in the software engineering area are gradually increasing, for example, the end-to-end pipelines became more automated after applying some artificial intelligence algorithms. Machine learning can also be advantageous in detecting anomalies in application logs. In this thesis study, we aim to investigate whether attribute noise removal has an impact on improving the learning performance of anomaly detection. We achieve our goal by applying an existing machine learning algorithm, log clustering, to identify anomalies. Also, an existing attribute noise removal method, PANDA, is used to study the effect of attribute noise removal on the learner performance of anomaly detection. Then we evaluate whether attribute noise removal can help improve the anomaly detection process by comparing the training results of different experimental groups. Overall, as the percentage of attribute noise removal increases, the values of precision, F1-Score, and MCC increase slightly. We conclude that attribute noise removal has the potential to be beneficial to the anomaly detection process, such as increasing the number of detected false cases.