Coupled Hidden markov models for complex action recognition

Nuria Oliver, Matthew Brand and Alex Pentland
Vision and Modeling Group, MIT Media Lab
Cambridge, MA 02139-1130


We present algorithms for coupling and training hidden Markov models (HMMs) to model interacting processes, and demonstrate their superiority to conventional HMMs in a vision task classifying two-handed actions. HMMs are perhaps the most successful framework in perceptual computing for modeling and classifying dynamic behaviors, because they offer dynamic time warping, a learning algorithm, and a clear Bayesian semantics. However, the Markovian framework makes strong restrictive assumptions about the system generating the signal---that it is a single process having a small number of states and an extremely limited state memory. The single-process model is often inappropriate for vision (and speech) applications, resulting in low ceilings on model performance. Coupled HMMs provide an efficient way to resolve many of these problems, and offer superior training speeds, model likelihoods, and robustness to initial conditions.
Examples of gestures robustly classified by CHMMs:
And here is the data (taichiData.tar.gz) that I used for the experimental results: "Taichi Data"
Last revised 27nov96

Nuria Oliver/ Microsoft Research

Matthew Brand / MERL /