Towards a Theoretical Understanding of Parameter-Efficient Fine-Tuning (and Beyond)

Abstract: As pretrained models grow in complexity and size, traditional transfer learning methods, which adjust all the model parameters, are becoming increasingly expensive. In response, parameter-efficient fine-tuning, a new adaptation method that updates only a tiny fraction of the model parameters, leaving the remainder unchanged, has gained popularity. Despite the growing importance of this emerging paradigm, its theoretical foundations remain largely unexplored. Current algorithms and design choices substantially rely on heuristics, signaling a significant opportunity for a comprehensive, theoretical exploration of this novel paradigm. I will introduce two recent results in this new area: the expressive power of low-rank adaptation (also known as LoRA), and the memorization capacity of side-tuning. I will conclude the talk by briefly discussing prompting and looping, a completely different approach to adaptation that does not involve any parameter updates.

Bio: Kangwook Lee is an Assistant Professor at the Electrical and Computer Engineering Department and the Computer Sciences Department (by courtesy) at the University of Wisconsin-Madison. Previously, he was a Research Assistant Professor at the Information and Electronics Research Institute of KAIST and was a postdoctoral scholar at the same institute. He received his PhD in 2016 from the Electrical Engineering and Computer Science department at UC Berkeley. He is the recipient of The IEEE Joint Communications Society/Information Theory Society Paper Award (2020) and the KSEA Young Investigator Grants Award (2022).

*** The event will take place in a hybrid format. The location for in-person attendance will be room 1303 EECS.  Attendance will also be available via Zoom.

Zoom Passcode information is available upon request to Sher Nickrand ([email protected]).

This seminar will be recorded and posted to the CSP Seminar website