One disadvantage of the discrete representation of
the state is that it is an inefficient representation
in comparison to a distributed representation with multiple
state variables.
When the state variable can take n values, only
bits of information about the past of
the observed sequence are carried by its
value. For example, if instead n binary variables
were used, exponentially more values would be
available. In general such models would be very
expensive to maintain, but so-called factorial HMMs [&make_named_href('',
"node31.html#Zoubin-nips8","[125]")]
have been proposed with such properties. On the
other hand, models with a continuous-valued state
have been typically restricted to a linear-Gaussian model,
again for reasons of computational tractability.
Some particular exceptions with nonlinear and non-Gaussian
models are also explored in [&make_named_href('',
"node31.html#kitagawa87","[126]"), &make_named_href('',
"node31.html#kitagawa96","[127]"), &make_named_href('',
"node31.html#kitagawa+gersch96","[47]")].
For example, the above models avoid the Gaussian assumptions
with non-parametric (e.g., piecewise-linear) density
functions [&make_named_href('',
"node31.html#kitagawa87","[126]")],
or with a Monte-Carlo approximation [&make_named_href('',
"node31.html#kitagawa96","[127]")].
One objective is to model both the abrupt and gradual changes in time series.
With a similar objective,
several researchers have in fact proposed hybrids
of state space models and discrete-state
HMMs (or IOHMMs), also known as state space models
with switching, or jump-linear systems. See [&make_named_href('',
"node31.html#Bar-Shalom93","[122]")]
and [&make_named_href('',
"node31.html#Zoubin96b","[14]")] for a review of such models. The
hybrid state can be represented with a distinct vector
variable for each value of the discrete variable.
Many early models assume that some of the parameters of the distribution are known a-priori, and others [&make_named_href('', "node31.html#Shumway91","[13]")] approximate the EM algorithm with a heuristic, because the E-step would require exponential computations. Others [&make_named_href('', "node31.html#Carter94","[128]"), &make_named_href('', "node31.html#Athaide95","[129]")] used expensive Monte-Carlo simulations to address this problem. Instead, in [&make_named_href('', "node31.html#Zoubin96b","[14]")], a function that is a lower bound on the log likelihood is maximized with a tractable algorithm. This paper uses the idea of variational approximation that has already been proposed in [&make_named_href('', "node31.html#Saul96","[130]")] for other intractable models. A simpler version of this idea used in physics is the mean-field approximation [&make_named_href('', "node31.html#Parisi88","[131]")] for statistical mechanics systems.