MIDAS Seminar

Geometries of Word Embeddings

Pramod ViswanathProfessor, Electrical Engineering and Computer ScienceUniveristy of Illinois at Urbana-Champaign

For more information on MIDAS or the Seminar Series, please contact [email protected]. MIDAS gratefully acknowledges Northrop Grumman Corporation for its generous support of the MIDAS Seminar Series.
Real-valued word vectors have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities via simple geometrical operations. In this talk, we demonstrate further striking geometrical properties of the word vectors. First, we show that a very simple, and yet counter-intuitive, postprocessing technique, which makes the vectors "more isotropic", renders off-the-shelf vectors even stronger. Second, we show that a sentence containing a target word is well represented by a low-rank subspace; subspaces associated with a particular sense of the target word tend to intersect with a line (one-dimensional subspace). We harness this Grassmannian geometry to disambiguate (in an unsupervised way) multiple senses of words, specifically so on the most promiscuously polysemous of all words: prepositions. A surprising finding is that rare senses, including idiomatic/sarcastic/metaphorical usages, are efficiently captured. Our algorithms are all unsupervised and rely on no linguistic resources; we validate them by presenting new state-of-the-art results on a variety of multilingual benchmark datasets.


Geometry of Compositionality, AAAI '17, https://arxiv.org/abs/1611.09799
Geometry of Polysemy, ICLR, '17, https://arxiv.org/abs/1610.07569
Representing Sentences as Low-rank subspaces, ACL '17, https://arxiv.org/abs/1704.05358
Prepositions in Context, preprint, https://arxiv.org/abs/1702.01466

Pramod Viswanath received the Ph.D. degree in electrical engineering and computer science from University of California at Berkeley in 2000. From 2000 to 2001, he was a member of research staff at Flarion technologies, NJ. Since 2001, he is on the faculty at University of Illinois at Urbana Champaign in Electrical and Computer Engineering, where he currently is a professor.

He received the Eliahu Jury Award from the EECS department of UC Berkeley in 2000, the Bernard Friedman Prize from the mathematics department of UC Berkeley in 2000, a NSF CAREER award in 2002, the Xerox faculty research award from the college of engineering of UIUC in 2010 and the Best Paper Award at the Sigmetrics conference in 2015. He is a coauthor, with David Tse, of the text Fundamentals of Wireless Communication, which has been used in over 60 institutions around the world. He is coinventor of the opportunistic beamforming method and codesigner of Flash-OFDM communication algorithms used in all fourth-generation cellular systems.

He was an Associate Editor of the IEEE Transactions on Information Theory from 2006 to 2008, the Technical Program co-chair of the Information Theory Workshops in 2010 and 2015 and the Technical Program co-chair of the Allerton conference in 2007 and 2008.

His current research interests are in machine learning and natural language processing. In the past he has worked on, and continues to hold interest in, information theory and its applications in various fields, including wireless communication and data privacy.

Sponsored by


Faculty Host

Al Hero