Special CMT Forum: Pierfrancesco Urbani

Separation of timescales controls feature learning and overfitting in large neural networks

Understanding how overparameterized networks can extract meaningful features from structured data while remaining expressive enough to perfectly interpolate noise is a central conceptual challenge in machine learning. To address this, we analyze the out-of-equilibrium, non-linear, high-dimensional training dynamics of overparameterized two-layer neural networks using dynamical mean field theory.

Our findings reveal a separation of timescales in training, with a slow timescale associated with the growth of network complexity. When the initial network complexity is sufficiently small, the system exhibits an inductive bias toward low-complexity solutions. This separation also leads to a dynamical decoupling between feature learning and overfitting phases, as well as a non-monotonic trend in test error, marked by a "feature unlearning" regime in the later stages of training.

I will conclude by constructing an intriguing analogy with the dynamics of strongly correlated quantum systems.