Artificial Neural Networks (ANNs) or connectionist models have been successfully used in several pattern recognition and sequential data processing problems. Multi-layered ANNs [&make_named_href('', "node31.html#Rumelhart86c","[73]")] can represent a non-linear regression or classification model. Several researchers have proposed ways to combine ANNs with HMMs, in particular for automatic speech recognition. The proposed advantages of such systems include more discriminant training, the ability to represent the data with richer, non-linear models (in comparison to linear/Gaussian or discrete models) and the improvement in representing context (by using in their input vector some transformations of lagged values of the input variables). Some models (such as the Input-Output HMMs described in the next section) are also designed to learn long-term dependencies better and to reduce the problem of imbalance between emission and transition probabilities, therefore yielding more effective models of duration. Some ANN/HMM hybrids attempt to address some of the modeling weaknesses in HMMs as they are used for speech recognition, such as the incorrectness of the two Markov assumptions (with respect to the interpretation of state values that is made in these models), the poor modeling of phoneme duration (as discussed in section 3.6), and the poor use of some of the contextual information (including both short-term acoustic context and long-term context such as prosody).
Figure 9: Most combinations of ANNs and HMMs
use the ANN to compute emission probabilities
(for each state), given the observed vector
.
The dotted arrows represent error information
(such as, gradients) propagated from the HMM
into the ANN, for training.
As illustrated in Figure 9, most combinations of ANNs and HMMs use the ANNs to compute emission probabilities, i.e., to represent local structure, whereas the HMM represents the sequential structure of the data.
A left-to-right HMM can be seen as a flexible template and the Viterbi algorithm as a sophisticated way to align that template with the observed speech. Since ANNs were successful at classifying individual phonemes, initial research focused on using a dynamic programming alignment similar to the one done in HMMs with the Viterbi algorithm in order to go from the recognition of individual phonemes (or other local classification) to the recognition of whole sequences [&make_named_href('', "node31.html#Lippmann87b","[74]"), &make_named_href('', "node31.html#Bourlard-pami90","[75]"), &make_named_href('', "node31.html#Morgan90","[76]"), &make_named_href('', "node31.html#Bourlard-cspla89","[77]"), &make_named_href('', "node31.html#Franzini90","[78]"), &make_named_href('', "node31.html#Robinson+Fallside90","[79]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Tebelskis91","[82]"), &make_named_href('', "node31.html#Bengio_book96","[2]")]. In some cases [&make_named_href('', "node31.html#Bottou90","[83]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Bengio_book96","[2]"), &make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Tebelskis91","[82]")], the ANN outputs are not interpreted as probabilities, but are rather used as scores and generally combined with a dynamic programming algorithm akin to the Viterbi algorithm to perform the alignment and segmentation. In some cases the dynamic programming algorithm is embedded in the ANN itself [&make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Levin92","[84]")]. Alternatively, the ANN can be used to re-score the N-best hypotheses of phoneme segmentation produced with an HMM [&make_named_href('', "node31.html#Zavaliagkos93","[85]")], by assigning posterior probabilities to the phonemes for each of the phonetic segments hypothesized with the HMM. An HMM can also be viewed as a particular kind of recurrent [&make_named_href('', "node31.html#Rumelhart86c","[73]")] ANN [&make_named_href('', "node31.html#Bridle90","[86]"), &make_named_href('', "node31.html#Bridle90b","[87]")].
Although the ANN and the HMM are
sometimes trained separately, most researchers have proposed schemes
in which both are trained together, or at least the ANN is trained in
a way that depends on the HMM.
The models proposed by
Bourlard et al. rely on a probabilistic interpretation of the ANN
outputs [&make_named_href('',
"node31.html#Bourlard-pami90","[75]"), &make_named_href('',
"node31.html#Morgan90","[76]"), &make_named_href('',
"node31.html#Bourlard93","[1]"), &make_named_href('',
"node31.html#Morgan95","[88]")]. The ANN
is trained to estimate
posterior probabilities of HMM states, given a context
of observation vectors,
,
centered on the current time step. By normalizing these posteriors
with state priors
, one obtains scaled emission probabilities.
These scaled emission probabilities
are used in the usual Viterbi algorithm for recognition. Training
of the ANN is based on the optimal
state sequence obtained from the constrained HMM
(with knowledge of the correct word sequence). For each
time step, the ANN is supervised with a target value of 1 for
the correct state and a target value of 0 for the other
states. This procedure has been found to converge and
yield speech recognition performance at the level of
state-of-the-art systems [&make_named_href('',
"node31.html#Bourlard93","[1]"), &make_named_href('',
"node31.html#Morgan95","[88]")]. Bourlard et al.
draw links between variants of this procedure and the EM algorithm.
Rather then directly optimizing a global criterion, training is based on the
local targets provided by the constrained Viterbi
alignment algorithm.
Another approach [&make_named_href('',
"node31.html#Bengio_book96","[2]"), &make_named_href('',
"node31.html#Bengio_trnn92","[89]"), &make_named_href('',
"node31.html#Bengio_LeCun95","[30]")]
uses the ANN to
transform the observed sequence into a form that is easier to model for
an HMM that has simple (but continuous) emission models (e.g., Gaussian
or Gaussian mixture). The ANN is used as a non-linear trainable
pre-processor or feature extractor for the HMM. In that case, the objective of
learning for the combined ANN/HMM system is given by
a single criterion defined at the level of the whole sequence,
rather than at the level of individual observations
or segments (e.g., for phonemes or characters). In some
applications of this idea, the ANN is viewed as an ``object
spotter'' (e.g., to spot a phoneme or a character, for speech or
handwriting recognition), and the HMM as a post-processor
that can align the sequence of outputs from the ANN with
a higher-level (e.g., linguistic and lexical) model of
the temporal structure of the observed sequences.
This model was introduced in [&make_named_href('',
"node31.html#Bengio_trnn92","[89]")] for
phoneme recognition. It is also described in [&make_named_href('',
"node31.html#Bengio_book96","[2]")], and
was extended to character recognition in [&make_named_href('',
"node31.html#Bengio_LeCun95","[30]")].
The ANN transforms an input sequence
into
an intermediate observation sequence
, with
a parametrized function
.
For example, this function may capture some
of the contextual influences, and transform the input in
a way that makes it more invariant with respect to
the classifications of interest.
A basic idea of the implementation of this model is that
the optimization criterion C used to estimate the
parameters of the HMM
is a continuous and differentiable function of the
intermediate observations
. As illustrated with
dotted arrows from the HMM into the ANN in Figure 9,
the gradients
can be used to train the parameters
of
the ANN: gradient descent using the chain rule for derivatives
(also called back-propagation [&make_named_href('',
"node31.html#Rumelhart86c","[73]")])
yields the parameter gradients
To train such hybrids, the most commonly used criterion is
the maximum mutual information criterion (or almost
equivalently, maximizing the likelihood of the correct
interpretation given the transformed features).
In both cases the derivatives
can be obtained from the state posteriors
which would have to be computed
for the EM algorithm (see equation 18)
In some cases of ANN/HMM hybrids, it is possible with an a priori idea of what the ANN should accomplish to train the ANN and the HMM separately. However, it has been shown experimentally with the above ANN/HMM hybrid how training the ANN jointly with the HMM improves performance on a speech recognition problem [&make_named_href('', "node31.html#Bengio_trnn92","[89]"), &make_named_href('', "node31.html#Bengio_book96","[2]")], bringing down the error rate on a plosive recognition task. It has later been shown how using joint training with respect to a discriminant criterion on a handwriting recognition problem [&make_named_href('', "node31.html#Bengio_LeCun95","[30]")] reduced the character error rate. The idea of training a set of modules together (rather than separately) with respect to a global criterion with gradient-based algorithms was proposed several years ago [&make_named_href('', "node31.html#Bottou_these91","[90]"), &make_named_href('', "node31.html#Bottou91","[91]"), &make_named_href('', "node31.html#Bengio91","[92]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Bengio_book96","[2]")]. This general theme (combining modules that compute a differentiable function of their input and their parameters) establishes a link between HMMs and much ANN litterature based on the idea of back-propagating gradients. This idea is also further expanded in section 6, which introduces transducers, and the idea that most of the probabilistic assumptions in a model such as an HMM can be avoided or pushed to a higher level (where the decisions are taken).
In many of the above ANN/HMM hybrids and their application to speech recognition, the probabilistic assumptions behind HMMs are not realistic. The HMM is rather used as a mechanism to impose constraints on the sequential structure of the data, and the ANN is used to recognize local patterns.
Another way to integrate ANNs with HMMs is based on the idea of Input-Output HMMs described in the next section.