One problem that is faced in classical applications of HMMs is that emission
probabilities dominate in the choice of the state sequence and in the value of
. One consequence of this problem is a poor modeling of
durations in HMMs for speech and other sequence recognition
applications. Another consequence is that the influence of context on
recognition decisions may be too weak. For example, in context-dependent
speech recognition HMMs, the influence of phonetic context is represented by
transition probabilities between context-dependent phonetic units. Similarly,
the language model may be represented by transition probabilities from one
word model to the word models that can follow.
The reason for this imbalance is the following: on a logarithmic scale (which is
appropriate to compare the probabilities that are multiplied to obtain
), the range of values that emission probabilities
(e.g., between
and
)
can take is much larger from that of transition probabilities
(e.g., between around
and
).
The dynamic range of transition probabilities (on a
logarithmic scale) is smaller than that of emission probabilities because
typically there are only a few allowed transitions from a state to the next
state, whereas the space of observations is very large (e.g., continuous).
As a consequence, the path chosen by the Viterbi (or other search) algorithm,
is mostly influenced by the emission probabilities. When comparing two paths with equation 9, what makes the most difference is whether the emissions are well modeled by the sequence of states that is compatible with the topology of the HMM, i.e., with the existence or non-existence of transitions. In the extreme case, if the numerical value of non-zero transition probabilities are completely ignored, the Viterbi algorithm only does a ``dynamic time-warping'' match [&make_named_href('', "node31.html#Sakoe78","[72]")] between the observed sequence and probabilistic prototypes associated (through the emission distributions) with a sequence of state values in the HMM. Some operational speech recognition models actually ignore transition probabilities altogether, because they have very little impact on the recognition result. This fact suggests a poor use of phoneme duration statistics in current speech HMMs.
The distribution of durations associated with each speech unit can in principle be represented by multiple states that share the same emission model, with a left-to-right structure and appropriate transition probabilities between them. However, because of the imbalance problem, the only constraint on durations that is really effective is the one obtained from the topology of the HMM, i.e., by duplicating some states and by forcing some transition probabilities between these states to zero. Note that learning algorithms for parametric models, such as the EM algorithm, cannot be used to learn such discrete structure: instead the topology of the HMM is often decided a priori. Ignoring the value of non-zero transition probability corresponds to assigning a uniform probability for the duration within certain intervals and zero outside these intervals.
In section 5 we discuss the recently proposed asynchronous Input-Output HMMs, which could significantly alleviate this problem. Other solutions are heuristics in which the logarithms of transition probabilities and emission probabilities are linearly weighed differently in order to correct this problem. This was used for example in [&make_named_href('', "node31.html#Bengio_LeCun95","[30]")].