next up previous
Next: Input-Output HMMs Up: Markovian Models for Sequential Previous: Unrealistic Probabilistic Assumptions in

Integrating Artificial Neural Networks and HMMs

 

Artificial Neural Networks (ANNs) or connectionist models have been successfully used in several pattern recognition and sequential data processing problems. Multi-layered ANNs [&make_named_href('', "node31.html#Rumelhart86c","[73]")] can represent a non-linear regression or classification model. Several researchers have proposed ways to combine ANNs with HMMs, in particular for automatic speech recognition. The proposed advantages of such systems include more discriminant training, the ability to represent the data with richer, non-linear models (in comparison to linear/Gaussian or discrete models) and the improvement in representing context (by using in their input vector some transformations of lagged values of the input variables). Some models (such as the Input-Output HMMs described in the next section) are also designed to learn long-term dependencies better and to reduce the problem of imbalance between emission and transition probabilities, therefore yielding more effective models of duration. Some ANN/HMM hybrids attempt to address some of the modeling weaknesses in HMMs as they are used for speech recognition, such as the incorrectness of the two Markov assumptions (with respect to the interpretation of state values that is made in these models), the poor modeling of phoneme duration (as discussed in section 3.6), and the poor use of some of the contextual information (including both short-term acoustic context and long-term context such as prosody).

  
Figure 9: Most combinations of ANNs and HMMs use the ANN to compute emission probabilities (for each state), given the observed vector . The dotted arrows represent error information (such as, gradients) propagated from the HMM into the ANN, for training.

As illustrated in Figure 9, most combinations of ANNs and HMMs use the ANNs to compute emission probabilities, i.e., to represent local structure, whereas the HMM represents the sequential structure of the data.

A left-to-right HMM can be seen as a flexible template and the Viterbi algorithm as a sophisticated way to align that template with the observed speech. Since ANNs were successful at classifying individual phonemes, initial research focused on using a dynamic programming alignment similar to the one done in HMMs with the Viterbi algorithm in order to go from the recognition of individual phonemes (or other local classification) to the recognition of whole sequences [&make_named_href('', "node31.html#Lippmann87b","[74]"), &make_named_href('', "node31.html#Bourlard-pami90","[75]"), &make_named_href('', "node31.html#Morgan90","[76]"), &make_named_href('', "node31.html#Bourlard-cspla89","[77]"), &make_named_href('', "node31.html#Franzini90","[78]"), &make_named_href('', "node31.html#Robinson+Fallside90","[79]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Tebelskis91","[82]"), &make_named_href('', "node31.html#Bengio_book96","[2]")]. In some cases [&make_named_href('', "node31.html#Bottou90","[83]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Bengio_book96","[2]"), &make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Tebelskis91","[82]")], the ANN outputs are not interpreted as probabilities, but are rather used as scores and generally combined with a dynamic programming algorithm akin to the Viterbi algorithm to perform the alignment and segmentation. In some cases the dynamic programming algorithm is embedded in the ANN itself [&make_named_href('', "node31.html#Haffner91","[81]"), &make_named_href('', "node31.html#Levin92","[84]")]. Alternatively, the ANN can be used to re-score the N-best hypotheses of phoneme segmentation produced with an HMM [&make_named_href('', "node31.html#Zavaliagkos93","[85]")], by assigning posterior probabilities to the phonemes for each of the phonetic segments hypothesized with the HMM. An HMM can also be viewed as a particular kind of recurrent [&make_named_href('', "node31.html#Rumelhart86c","[73]")] ANN [&make_named_href('', "node31.html#Bridle90","[86]"), &make_named_href('', "node31.html#Bridle90b","[87]")].

Although the ANN and the HMM are sometimes trained separately, most researchers have proposed schemes in which both are trained together, or at least the ANN is trained in a way that depends on the HMM. The models proposed by Bourlard et al. rely on a probabilistic interpretation of the ANN outputs [&make_named_href('', "node31.html#Bourlard-pami90","[75]"), &make_named_href('', "node31.html#Morgan90","[76]"), &make_named_href('', "node31.html#Bourlard93","[1]"), &make_named_href('', "node31.html#Morgan95","[88]")]. The ANN is trained to estimate posterior probabilities of HMM states, given a context of observation vectors, , centered on the current time step. By normalizing these posteriors with state priors , one obtains scaled emission probabilities. These scaled emission probabilities are used in the usual Viterbi algorithm for recognition. Training of the ANN is based on the optimal state sequence obtained from the constrained HMM (with knowledge of the correct word sequence). For each time step, the ANN is supervised with a target value of 1 for the correct state and a target value of 0 for the other states. This procedure has been found to converge and yield speech recognition performance at the level of state-of-the-art systems [&make_named_href('', "node31.html#Bourlard93","[1]"), &make_named_href('', "node31.html#Morgan95","[88]")]. Bourlard et al. draw links between variants of this procedure and the EM algorithm. Rather then directly optimizing a global criterion, training is based on the local targets provided by the constrained Viterbi alignment algorithm.

Another approach [&make_named_href('', "node31.html#Bengio_book96","[2]"), &make_named_href('', "node31.html#Bengio_trnn92","[89]"), &make_named_href('', "node31.html#Bengio_LeCun95","[30]")] uses the ANN to transform the observed sequence into a form that is easier to model for an HMM that has simple (but continuous) emission models (e.g., Gaussian or Gaussian mixture). The ANN is used as a non-linear trainable pre-processor or feature extractor for the HMM. In that case, the objective of learning for the combined ANN/HMM system is given by a single criterion defined at the level of the whole sequence, rather than at the level of individual observations or segments (e.g., for phonemes or characters). In some applications of this idea, the ANN is viewed as an ``object spotter'' (e.g., to spot a phoneme or a character, for speech or handwriting recognition), and the HMM as a post-processor that can align the sequence of outputs from the ANN with a higher-level (e.g., linguistic and lexical) model of the temporal structure of the observed sequences. This model was introduced in [&make_named_href('', "node31.html#Bengio_trnn92","[89]")] for phoneme recognition. It is also described in [&make_named_href('', "node31.html#Bengio_book96","[2]")], and was extended to character recognition in [&make_named_href('', "node31.html#Bengio_LeCun95","[30]")]. The ANN transforms an input sequence into an intermediate observation sequence , with a parametrized function . For example, this function may capture some of the contextual influences, and transform the input in a way that makes it more invariant with respect to the classifications of interest.

A basic idea of the implementation of this model is that the optimization criterion C used to estimate the parameters of the HMM is a continuous and differentiable function of the intermediate observations . As illustrated with dotted arrows from the HMM into the ANN in Figure 9, the gradients can be used to train the parameters of the ANN: gradient descent using the chain rule for derivatives (also called back-propagation [&make_named_href('', "node31.html#Rumelhart86c","[73]")]) yields the parameter gradients

 

To train such hybrids, the most commonly used criterion is the maximum mutual information criterion (or almost equivalently, maximizing the likelihood of the correct interpretation given the transformed features). In both cases the derivatives can be obtained from the state posteriors which would have to be computed for the EM algorithm (see equation 18)

In some cases of ANN/HMM hybrids, it is possible with an a priori idea of what the ANN should accomplish to train the ANN and the HMM separately. However, it has been shown experimentally with the above ANN/HMM hybrid how training the ANN jointly with the HMM improves performance on a speech recognition problem [&make_named_href('', "node31.html#Bengio_trnn92","[89]"), &make_named_href('', "node31.html#Bengio_book96","[2]")], bringing down the error rate on a plosive recognition task. It has later been shown how using joint training with respect to a discriminant criterion on a handwriting recognition problem [&make_named_href('', "node31.html#Bengio_LeCun95","[30]")] reduced the character error rate. The idea of training a set of modules together (rather than separately) with respect to a global criterion with gradient-based algorithms was proposed several years ago [&make_named_href('', "node31.html#Bottou_these91","[90]"), &make_named_href('', "node31.html#Bottou91","[91]"), &make_named_href('', "node31.html#Bengio91","[92]"), &make_named_href('', "node31.html#Driancourt91","[80]"), &make_named_href('', "node31.html#Bengio_book96","[2]")]. This general theme (combining modules that compute a differentiable function of their input and their parameters) establishes a link between HMMs and much ANN litterature based on the idea of back-propagating gradients. This idea is also further expanded in section 6, which introduces transducers, and the idea that most of the probabilistic assumptions in a model such as an HMM can be avoided or pushed to a higher level (where the decisions are taken).

In many of the above ANN/HMM hybrids and their application to speech recognition, the probabilistic assumptions behind HMMs are not realistic. The HMM is rather used as a mechanism to impose constraints on the sequential structure of the data, and the ANN is used to recognize local patterns.

Another way to integrate ANNs with HMMs is based on the idea of Input-Output HMMs described in the next section.


next up previous
Next: Input-Output HMMs Up: Markovian Models for Sequential Previous: Unrealistic Probabilistic Assumptions in

Yoshua Bengio
Tue Oct 7 08:34:36 EDT 1997