End-to-End Object Tracking on Simulated Microscopy Data

Abstract

Object tracking using neural networks has become a pivotal technology in digital microscopy, enabling automated analysis and interpretation of dynamic visual data. Typically, this tracking is performed in two steps, often utilizing two separate neural network models. First, images are segmented to detect individual objects within each time frame and to extract their centroids and other features using one neural network model. Next, a selected set of features from these objects is used as input to a second neural network, which creates temporal trajectories by linking the objects across a sequence of frames. This work proposes a novel method that combines the object detection and linking steps, theoretically leading to better temporal links. Instead of relying on a fixed set of properties for each object, the combined model has access to the entire image and to temporal information, enabling it to autonomously select the most relevant features for optimal tracking. Two different architectures for a combined model were tested: a supervised model based on a graph neural network (GNN) and an unsupervised model based on a variational autoencoder (VAE). The supervised GNN-based model did not succeed in predicting the position of the centroids, but it showed promise in linking the centroids between frames. Therefore, the VAE-based model was developed that uses the same approach for linking. The VAE-based model showed promising results with a mean absolute error of under 0.002 on its detection placement, a detection miss-rate of 2.69%, and an F1-score of 81.2% when tracking simulated data.