A Segmental HMM for Speech Waveforms

Kannan Achan, Sam Roweis and Brendan Frey

Processing of speech signals directly in the time domain is commonly regarded to be difficult and unstable, due to fact that perceptually very similar utterances exhibit very large variability in their raw waveforms. As a result, by far the most common preprocessing step for most speech systems is to convert the raw waveform into a time-frequency representation, using a variety of spectral analysis and filterbank techniques.

In this work, we pursue a purely time domain approach to speech processing in which we identify the samples at the boundaries between glottal pulse periods (in voiced speech) or at the boundaries between unvoiced segments. The computational challenge with this approach is in efficiently and robustly identifying the segment boundaries, across silence, unvoiced and voiced segments. We introduce a segmental Hidden Markov Model, defined on variable length sections of the time domain waveform, and show that performing inference in this model allows us to identify segment boundaries.  Once these segment boundaries are identified, we can perform a variety of important low level speech analysis operations directly and conveniently.  For example, timescale modification without pitch or format distortion can be achieved by stochastically eliminating or replicating segments in the time domain directly.