Theoretical Characterization of Forgetting and Generalization of Continual Learning

Abstract: Continual learning (CL), which aims to learn a sequence of tasks in a lifelong manner, has attracted significant attention recently. However, most work has focused on the experimental performance of CL, and theoretical studies of CL are very limited. In particular, there is a lack of understanding of what factors are important and how they affect “catastrophic forgetting” and generalization performance. In this talk, I will present our results along this direction. I will first introduce our theoretical analysis under overparameterized linear models, which provide the first-known explicit form of the expected forgetting and generalization error. I will then provide a number of theoretical explanations about how overparameterization, task similarity, and task ordering affect both forgetting and generalization errors of CL. I will further talk about our experiments on real datasets using deep neural networks (DNNs), which illustrate that our insights can be carried over to practical setups and further motivate better practical algorithm designs for CL. I will conclude my talk with comments on a few future directions.

The work was done jointly with Sen Lin (U. Houston), Peizhong Ju (OSU), and Ness Shroff (OSU).

Bio: Dr. Yingbin Liang is currently a Professor at the Department of Electrical and Computer Engineering at the Ohio State University (OSU) and a core faculty of the Ohio State Translational Data Analytics Institute (TDAI). She also serves as the Deputy Director of the AI-EDGE Institute at OSU. Dr. Liang received the Ph.D. degree in Electrical Engineering from the University of Illinois at Urbana-Champaign in 2005 and served on the faculty of University of Hawaii and Syracuse University before she joined OSU. Dr. Liang’s research interests include machine learning, optimization, information theory, and statistical signal processing. Dr. Liang received the National Science Foundation CAREER Award and the State of Hawaii Governor Innovation Award in 2009.  She also received the EURASIP Best Paper Award in 2014. She is an IEEE fellow.

*** This event will take place in a hybrid format. The location for in-person attendance will be room 3427 EECS.   Attendance will also be available via Zoom.

Join Zoom Meeting https: https://umich.zoom.us/j/99102451525

Meeting ID:  991 0245 1525

Passcode: XXXXXX (Will be sent via e-mail to attendees)

Zoom Passcode information is also available upon request to Sher Nickrand ([email protected]).

This seminar will be recorded and posted to the CSP Seminar website.