[IMS Seminar] Embedding Principle of Loss Landscape of Deep Neural Networks

ON2022-05-18TAG: ShanghaiTech UniversityCATEGORY: Lecture

Time:Friday, May 20th, 2022, 10:00-11:00

Location:Tecent Meeting


Room Number: 778-1386-4492

Speaker:  Yaoyu ZhangShanghai Jiao Tong University

Abstract: Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously important. In this talk, I will present the embedding principle that the loss landscape of a DNN contains all the critical points of all the narrower DNNs. Empirically, we find that a wide DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides a new perspective to study the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training, by which a more exact and comprehensive understanding can be anticipated in the near future.