Ultra Scalable Messaging Systems

Rapid dissemination of data to multiple receivers is critical for trade in the stock exchange, message delivery over Facebook and Twitter feeds and to enable emerging technologies such as content distribution, Internet television (IPTV) and virtual reality games. Despite two decades of work on creating a usable mechanism for these communication patterns, current mechanisms are broken in a number of ways. The Achilles' heel of these protocols is scalability, and those which scale lack either efficiency or generality.

I will show how to overcome these deficiencies with two highly scalable distributed systems: Dr. Multicast and Kevlar. These address rather different problems but they share a core issue. At first glance, efficient scalability seems unlikely because each involves solving NP-complete optimization problems. My work uses ideas from the fields of social networks and approximation algorithms to find high quality solutions. In the talk I'll focus on the core insights, which involve recognizing the theoretical structure of this important class of problems and then finding ways to exploit the strongly correlated use patterns seen in real-world data to make the needed optimization algorithms exceptionally scalable and fast.

I'll touch on some ongoing and future projects that combine networked systems, algorithms and data analysis, offering opportunities to enlarge the theory even as we use it to solve other kinds of real-world problems.
Dr. Ymir Vigfusson is a researcher at IBM Research in Haifa and an adjunct professor at Reykjavik University. Ymir received his Ph.D. in Computer Science from Cornell University in August 2009, where he researched ways to exploit group similarity and improve scalability in distributed systems. Cornell nominated his dissertation for the
ACM Doctoral Dissertation Award. Ymir's research projects include creating and optimizing systems and algorithms for distributed settings, and getting multicast to work in a variety of environments. His work has been partially supported by a Fulbright Scholarship and a Yahoo! Research grant.