Modeling spatiotemporal information with convolutional gated networks

Abstract

In this thesis, a recently proposed bilinear model for predicting spatiotemporal data has been implemented and extended. The model was trained in an unsupervised manner and uses spatiotemporal synchrony to encode transformations between inputs of a sequence up to a time t, in order to predict the next input at t + 1. A convolutional version of the model was developed in order to reduce the number of parameters and improve the predictive capabilities. The original and the convolutional models were tested and compared on a dataset containing videos of bouncing balls and both versions are able to predict the motion of the balls. The developed convolutional version halved the 4-step prediction loss while reducing the number of parameters by a factor of 159 compared to the original model. Some important differences between the models are discussed in the thesis and suggestions for further improvements of the convolutional model are identified and presented.