Mean-Field Analysis of Loss Landscapes in Overparameterized Two-Layer ReLU Neural Networks

Abstract

We study the loss landscape of high-dimensional two-layer ReLU neural networks in the teacher-student setting through a macroscopic description in terms of summary statistics, where the training dynamics reduce to a closed system of ordinary differential equations for the order parameters. When the student and teacher networks have the same size, we identify several structured families of spurious local min ima organized into discrete loss levels. These families exhibit block-structured order parameters, characterized by aligned and anti-aligned groups of student units. Motivated by this structure, we introduce a reduced ansatz and compute theoretical fixed points whose loss values and order-parameter patterns agree with large-scale ODE simulations. When the student network is slightly overparameterized by adding a single extra neuron, the loss landscape changes qualitatively: the lowest-order spurious minima family is destabilized, the basin of attraction of the global minimum expands, and the remaining high-loss families become less frequent. Using the string method as an exploratory diagnostic, we find that representative fixed points separated by visible barriers in the equal-size setting become less isolated under overparameterization, with paths often attracted toward the global-minimum basin. These results suggest a mechanism by which overparameterization improves optimization in this model.