next up previous
Next: Learning Criteria Up: Speech Recognition with HMMs Previous: HMM Topology from A

Performance

The performance of speech recognition systems based on HMMs varies a lot depending on the difficulty of the task. Benchmarks to compare speech recognition systems have been set up by ARPA [&make_named_href('', "node31.html#ARP94","[65]")] in the U.S.A.. The difficulty increases with the size of the vocabulary, the variability of the speech among the speakers, and other factors. For example, on the ATIS benchmark (where the task is to provide airline information to users, and the vocabulary has around 2000 words), laboratory experiments yielded around 5% of incorrectly answered queries [&make_named_href('', "node31.html#Levinson96","[66]")]. This task involves not only recognition but also understanding. On a large vocabulary task set up by ARPA with around 60000 words (no understanding, only recognition), the word error rates reported are below 11%. However, performance of speech recognition systems is often much worse in the field than in the laboratories. Speech recognition is now used in commercial applications, as in the AT&T telephone network. This system looks for one of five keywords. It makes an error in less than 5% of the calls and processes around one billion calls per year [&make_named_href('', "node31.html#Levinson96","[66]")].



Yoshua Bengio
Tue Oct 7 08:34:36 EDT 1997