A Cluster Analysis Framework using DiMaxL - Clustering and data reduction in the presence of noise

Abstract

The goal of this project was to develop a framework which can be used to make accurate predictions on large, sampled, data sets where statistical outliers are present. A secondary aim was to develop a method to reduce the large amount of data sometimes available, but all of it not always useful when making a prediction. The framework was developed using the DiMaxL algorithms and was tested on data sets taken from biology. These data sets are protein measurements where a large amount of statistical outliers are present. The results indicate that the method can accurately detect patterns even in presence of large amount of noise without any excessive overfitting. In the case of data reduction, the accuracy of the method is more sensitive to the amount of available data, and a semi-automatic procedure is recommended. In conclusion, the framework developed, is able to effectively remove noise while detecting the underlying pattern present, even in complex correlations.