Skip to content. | Skip to navigation

Personal tools

Sections

UMR 5672

logo de l'ENS de Lyon
logo du CNRS
You are here: Home / Seminars / Machine Learning and Signal Processing / Deep learning theory through the lens of diagonal linear networks

Deep learning theory through the lens of diagonal linear networks

Scott Pesme (post-doc in the Thoth team at Inria Grenoble)
When Dec 10, 2024
from 12:45 to 01:45
Attendees Scott Pesme
Add event to calendar vCal
iCal

Scott Pesme 

Title: Deep learning theory through the lens of diagonal linear networks

Abstract:  Surprisingly, many optimisation phenomena observed in complex neural networks also appear in so-called 2-layer diagonal linear networks. This rudimentary architecture—a two-layer feedforward linear network with a diagonal inner weight matrix—has the advantage of revealing key training characteristics while keeping the theoretical analysis clean and insightful. 

In this talk, I’ll provide an overview of various theoretical results for this architecture, while drawing connections to experimental observations from practical neural networks. Specifically, we’ll examine how hyperparameters such as the initialisation scale, step size, and batch size impact the optimisation trajectory and influence the generalisation performances of the recovered solution.
 
 

Website: https://scottpesme.github.io/

In Room M7 101, 1st floor, Monod campus, ENSL.