Perceiving Action in Space-Time: Computational and Human Perspectives

Humans are highly articulated, which leads to complex and
idiosyncratic actions in space-time. This complexity has challenged
computational models of human action for some time now, and yet humans
themselves are highly adept at parsing action. In this talk, I will
motivate the challenge of interpreting human action from spatial,
temporal, and spatiotemporal points of view. Then, I will present
both computational and human perspectives on modeling action. First,
I will describe how video can be decomposed into a multilevel semantic
scale-space using a Markov approximation framework. Within this
semantic scale-space, we have conducted a visual psychophysical study
of how humans perceive action, and I will report our findings in that
study.

Second, I will present a computational model for human action. The
method, called Action Bank, creates a high-level action space that is
spanned by individual space-time actions. Query videos are projected
into this action space and non-linear classifiers are learned for
recognition. Experiments demonstrate how space-time, action-specific
modeling can outperform conventional feature-based methods that do not
leverage space-time continuity. Third, I will bring these two
perspectives together in a "full-circle" experiment leveraging ideas
from both camps. Time-permitting, I will relate these findings to
other work in my group in computational neuroscience and cognitive
robotics.