Communications and Signal Processing Seminar

Uncovering the Law of Data Separation in Deep Learning

Weijie SuAssociate Professor of Statistics and Data ScienceUniversity of PennsylvaniaAssociate Professor of Computer and Information Science (secondary appointment)The Wharton School, University of PennsylvaniaCo-Director, Penn Research in Machine LearningUniversity of Pennsylvania
WHERE:
1690 Beyster BuildingMap
WHEN:
Thursday, April 13, 2023 @ 9:15 am - 10:15 am
This event is free and open to the publicAdd to Google Calendar
SHARE:

Abstract: In this talk, we document a law of data separation that is a pervasive empirical phenomenon in the terminal phase of deep learning training. This law describes how data are separated according to their class membership from the bottom to the top layer in a well-trained neural network. We will show that, through extensive computational experiments, neural networks improve data separation through layers in a simple exponential manner. This law leads to roughly equal ratios of separation that a single layer is able to improve, thereby showing that all layers are created equal. We will further discuss the implications of this law on the interpretation, robustness, and generalization of deep learning, as well as on the inadequacy of some existing approaches toward demystifying deep learning. The presentation will conclude by introducing an infinitesimal variant of the data separation law. This is based on joint work with Hangfeng He and Cheng Shi.

Bio: Weijie Su is an Associate Professor in the Wharton Statistics and Data Science Department and, by courtesy, in the Department of Computer and Information Science, at the University of Pennsylvania. He is a co-director of Penn Research in Machine Learning. Prior to joining Penn, he received his Ph.D. in statistics from Stanford University under the supervision of Emmanuel Candes in 2016 and his bachelor’s degree in mathematics from Peking University in 2011. His research interests span privacy-preserving data analysis, deep learning theory, optimization, mechanism design, and high-dimensional statistics. He is a recipient of the Stanford Theodore Anderson Dissertation Award in 2016, an NSF CAREER Award in 2019, an Alfred Sloan Research Fellowship in 2020, the SIAM Early Career Prize in Data Science in 2022, and the IMS Peter Gavin Hall Prize in 2022.

***Event will take place in a hybrid format. The location for in-person attendance will be room 1690 Beyster Building.   Attendance will also be available via Zoom.

Join Zoom Meeting  https://umich.zoom.us/j/91414297851

Meeting ID: 914 1429 7851

Passcode: XXXXXX (Will be sent via e-mail to attendees)

Zoom Passcode information is also available upon request to Sher Nickrand (sherylb@umich.edu).

Faculty Host

Qing QuAssistant Professor, Electrical Engineering and Computer ScienceUniversity of Michigan