An Informative-theoretic perspective on multi-modal data processing

Combined with nonparametric statistics, information-theoretic approaches to computer vision, sensing, data processing, and machine learning have increased in popularity in recent years. The formalism is attractive in that one need not make strong modeling assumptions while retaining the capacity to model complex statistical relationships. The realization is not novel in that many of the links were pointed out in the work of Kullback, Jaynes and others over the last 50 years. Of particular interest in the work presented in this talk is how to apply the formalism to complex high-dimensional data. I will present methods and empirical results for estimation of dependency structures across multi-modal data types. Such inference tasks can formulated as hypothesis tests over graphical structure and have a natural information-theoretic interpretation with interesting structure. They are applicable in a variety of settings including multi-modal data association, link analysis of moving objects, and image segmentation, examples of which will be presented. Static versions of such tests can be extended to the dynamic setting, where we consider a problem in which we are presented with multiple (possibly high-dimensional) data streams and are interested in the nature of their interaction as it evolves over time. Here, evolving interaction is equated to changing graphical structures, i.e., the presence or absence of edges between data streams, but for which the parameters (or more generally the parameterization) is not available. I will present empirical results for an audio-video association task on a standard publicly available data set. In contrast to competing methods, the approach makes no use of training data and yet achieves the best known results to date. Further analysis of the results provides insights into both the complexity of the task and pointers to unmodeled phenomenology.

John Fisher is a Principal Research Scientist in the Computer Science and Artificial Intelligence Laboratory and affiliated with the Laboratory for Information and Decision Systems, both at the
Massachusetts Institute of Technology. Prior to joining the Massachusetts Institute of Technology he was affiliated with the Electronic Communications Laboratory at the University of Florida from 1987 to 1997, during which time he conducted research in the areas of ultra-wideband radar for ground and foliage penetration applications, radar signal processing, and automatic target recognition algorithms. His current area of research focus includes information theoretic approaches to signal processing, multi-modal data fusion, machine learning and computer vision, and sensor management.