Real-time anomaly detection in computer networks using machine learning

Abstract

This thesis explains how to employ machine learning methods for anomaly detection in real-time on a computer network. While using machine learning for this task is not a novel concept, little literature is on the subject of doing it in real time. Most machine learning research in computer network anomaly detection is based on the KDD ’99 data set and aims to prove the efficiency of the algorithms presented. The focus on this data set has caused a shortage of scientific papers explaining how to gather network data, extract features and train algorithms for use in real time networks. It has been argued that using the KDD ’99 data set for anomaly discovery is not applicable to real time networks. This thesis proposes how the data gathering process can be done using a dummy network and compares the results of k-means clustering, one class SVM and LSTM neural networks with reported results of the same algorithms on the KDD ’99 data set. The results show that algorithms trained using the KDD data set have worse accuracy, but that this can be linked to the lack of complexity in the gathered data.