Object Detection and Grounding Referring Expressions Using Deep Networks

Consider the situation where a human is interacting with a visual surveillance system and wants the surveillance system to track a specific entity moving through the field of view of a camera. While this can be done using a graphical interface to identify the object of interest, it might be more natural for the human to provide the surveillance system with a short, verbal description of the object "“ for example, the "tall man following a child." Such short verbal descriptions intended to identify a specific instance of an object in an image are called "referring expressions." The typical pipeline for grounding referring expressions is to first identify instances of the objects named in the expression in an image, and then select the instance(s) that best satisfy the referring expression. I will describe recent research on the two basic problems "“ object detection and grounding referring expressions "“ in this talk. For detection, I will present two deep learning architectures for detection and a recurrent network for applying context to improve detection. Then, I will explain how LSTM's can be used to ground referring expressions over the set of object classes known by the detectors. If time permits, I will also present a brief overview of recent research on relighting faces with CNN's.

"g-CNN: an Iterative Grid Based Object Detector" , Mahyar Najibi, Mohammad Rastegari, and Larry S. Davis, CVPR 2016.

"Modeling Context Between Objects for Referring Expression Understanding," Varun K. Nagaraja, Vlad I. Morariu, Larry S. Davis European Conference on Computer Vision (ECCV) 2016.

Larry S. Davis received his B.A. from Colgate University in 1970 and his M. S. and Ph. D. in Computer Science from the University of Maryland in 1974 and 1976 respectively. From 1977-1981 he was an Assistant Professor in the Department of Computer Science at the University of Texas, Austin. He returned to the University of Maryland as an Associate Professor in 1981. From 1985-1994 he was the Director of the University of Maryland Institute for Advanced Computer Studies. He was Chair of the Department of Computer Science from 1999-2012. He is currently a Professor in the Institute and the Computer Science Department, as well as Director of the Center for Automation Research. He was named a Fellow of the IEEE in 1997 and of the ACM in 2013.