Ludovic Stephan
Speaker: Ludovic Stephan (EPFL)
https://scholar.google.com/citations?user=mEd3WCsAAAAJ&hl=en
Title: Feature learning in two-layer neural networks with large gradient steps
Abstract: Feature learning is an important mechanism of neural networks, and an integral part of their advantages over simpler (e.g. kernel) learning methods. In this talk, I will present how this phenomenon occurs in two-layer networks trained with large gradient steps, in which both the batch size and the learning rate grow polynomially with the dimension. In particular, we uncover an occurence of the so-called "staircase" property of learning, where important directions are learned sequentially at each new step.