A Study on Isolation Forest for Anomaly Detection in Cloud-Based Systems

Abstract

The need for effective monitoring solutions increases as more organizations migrate to a cloud infrastructure. The leading cloud providers offer their own monitoring services, but they lack customizability and are black-box, making it hard to understand and control their behavior. This thesis investigates developing and deploying a custom monitoring service to the cloud. This will be done by applying the Isolation Forest (iForest) algorithm as an anomaly detection tool for monitoring cloud services within Amazon Web Services (AWS). While iForest is a powerful unsupervised learning algorithm, it is made for static analysis. Applying it to streaming data introduces some challenges, such as concept drift and seasonal changes in the data. Our research addresses these challenges by tailoring iForest to cloud monitoring. As many of you are aware, more and more companies have been migrating their operations to the cloud in order to enhance flexibility, scalability, and efficiency in their operations. In this thesis, we studied iForest for detecting anomalies in CloudWatch metrics data in the context of WirelessCar. The developed model demonstrated superior performance compared to state-of-the-art alternatives. Additionally, we deployed the model to the cloud through AWS where it was used to detect anomalies in live data from a data stream. We have summarized the findings and provided practical guidelines for anomaly detection development and cloud deployment. These guidelines aim to assist practitioners and researchers with integrating anomaly detection for cloud monitoring.