Diffusion based interacting particle system for optimization

Abstract

The ability to minimize a function is crucial across various fields, from training neural networks by minimizing loss functions to reducing costs in production lines. This minimization is typically achieved using optimization algorithms where the choice of algorithm impacts both the overall result and the computational cost. In this thesis, we propose new optimization algorithms based upon particle interactions driven by stochastic differential equations (SDEs). Specifically, we aim to adapt first and second-order ensemble Langevin sampling methods to use as optimization algorithms. Additionally, we explore a variant of consensus-based optimization (CBO) that incorporates repulsive forces. Our results demonstrate the effectiveness of the sampling methods with annealing as optimizers and highlight the benefits of repulsion for CBO. Furthermore, we test ensemble Langevin dynamics as an optimizer for training neural networks. We approximate gradient using Kalman approximation that allows for training without the need for back-propagation. The results indicate performance similar to stochastic gradient descent (SGD).