CANCELED: Communication-Efficient and Straggler-Resilient Distributed Learning

Abstract:  In many large-scale machine learning applications, data is acquired and processed at the edge nodes of the network such as mobile devices, users’ devices, and IoT sensors. While distributed learning at the edge can enable a variety of new applications, it faces major systems bottlenecks that severely limit its reliability and scalability including communications bottleneck and network heterogeneity (straggler) bottleneck. In this talk, we first focus on federated learning which is a new distributed machine learning approach, where a model is trained over a set of devices such as mobile phones, while keeping data localized. We present FedPAQ, a novel communication-efficient and scalable Federated learning method with Periodic Averaging and Quantization. FedPAQ is provably near-optimal in the following sense. Under the problem setup of expected risk minimization with independent and identically distributed data points, when the loss function is strongly convex the proposed method converges to the optimal solution with near-optimal rate, and when the loss function is non-convex it finds a first-order stationary point with near-optimal rate. In the second part of the talk, we consider a decentralized stochastic learning problem, where a set of computing nodes aim at solving an optimization problem collaboratively in a master-less topology. We propose a decentralized and gradient-based optimization algorithm named QuanTimed-DSGD, and provide exact convergence guarantees for the proposed method for both convex and non-convex settings.

Biography: Ramtin Pedarsani is an assistant professor in the ECE department at UCSB. He obtained his Ph.D. in Electrical Engineering and Computer Sciences from UC Berkeley in 2015. He received his M.Sc. degree at EPFL in 2011 and his B.Sc. degree at the University of Tehran in 2009. His research interests include machine learning, optimization, information and coding theory, and stochastic networks. He is the recipient of the best paper award in the IEEE International Conference on Communications (ICC) in 2014