Blessing of Nonconvexity in Factorized Models

Abstract: Factorized models, from low-rank matrix recovery to deep neural networks, play a central role in many modern machine learning problems. Despite their widespread applications, problems based on factorized models are deemed difficult to solve in their worst case due to their inherent nonconvexity. Our talk is inspired by the recent observations in the optimization and machine learning communities that many realistic and practical instances of factorized models are far from their worst case scenarios.

We study a natural nonconvex and nonsmooth formulation of two prototypical factorized models, namely low-rank matrix factorization and deep linear regression, where the goal is to recover a low-dimensional model from a limited number of measurements, a subset of which may be grossly corrupted with noise. On the negative side, we show that this problem does not have a benign landscape: with high probability, there always exists a true solution that is not a global minimum of the loss function. However, on the positive side, we show that a simple subgradient method with small initialization is oblivious to such “problematic” solutions; instead, it converges to a balanced solution that is not only close to the ground truth but also enjoys a flat local landscape. Lastly, we empirically verify that the desirable optimization landscape of factorized models extends to other robust learning tasks, including deep matrix recovery and deep ReLU networks with L1-loss.

Bio:  I am an assistant professor in the Department of Industrial and Operations Engineering at the University of Michigan. I am also affiliated with Michigan Institute for Computational Discovery and Engineering (MICDE), Michigan Institute for Data Science (MIDAS), Michigan Center for Applied and Interdisciplinary Mathematics (MCAIM), and Controls Group.

My research focuses on developing large-scale computational and statistical methods for structured problems. In particular, I develop new techniques in optimization and machine learning to solve massive-scale and data-driven problems.

I received my M.Sc. and Ph.D. degrees in Industrial Engineering and Operations Research from University of California, Berkeley. I also received a M.Sc. degree in Electrical Engineering from Columbia University, and a B.Sc. degree in Electrical Engineering from Sharif University of Technology.

***Event will take place in hybrid format. The location for in-person attendance will be room 1008 EECS.   Attendance will also be available via Zoom.

Join Zoom Meeting https: https://umich.zoom.us/j/91414297851

Meeting ID: 914 1429 7851

Passcode: XXXXXX (Will be sent via e-mail to attendees)

Zoom Passcode information is also available upon request to Michele Feldkamp ([email protected]).