In this talk, I will discuss learning partially observable Markov decision process (POMDP)
models of human interactive tasks, in which a decision-making agent must interact with a human
to achieve its goals. Central to our work is the modeling of human behaviors within this
context using computer vision. The functions we seek to learn are mappings from spatially
and temporally extended observations (video sequences) to actions which are optimal with
respect to some utility function (i.e. a policy). Given the large observation space of the
video sequences, this task is one of learning spatio-temporal abstractions of the
observations that can enable optimal action choices at the level of goals and utility.
One of the most significant advantages of this type of learning is that it does not require
labeled data from expert knowledge about which behaviors are significant in a particular
interaction. Rather, the learning process discovers
the significant behaviors, leading
to a transportable system. A key idea is that only those behaviors which are "important"
(in the sense that they enable or help the system to complete its goals) need to be distinguished
perceptually. That is, we assume some observation subspace that spans the perceptual distinctions
that need to be made in order to achieve goals. We find this subspace by learning dynamic Bayesian
network models of human behaviors. The parameters of these models are learned from training data
using an a-posteriori constrained optimization technique
based on the expectation-maximization algorithm.
I will briefly overview POMDPs and multi-agent POMDPs, and expose some of the difficulties
inherent in learning to act in a multi-agent situation. I will then describe the computer
vision observation functions which we have experimented with. I will show how to learn the
parameters of these models, and show how they can be applied to a gestural robot control
experiment and a collaborative card game. Finally, I will describe my current work towards
building these models for an assisted living task, and will discuss some of the outstanding issues.