This first chapter reviews the classical and neural foundations on which contemporary speech recognition is built. It is organized in two main parts: the classical and hybrid paradigms, then the neural and self‑supervised paradigms. Each part develops three subsections in mathematical detail.
Automatic Speech Recognition is, in its modern formulation, the inversion of a noisy generative channel. Let \(X\) denote the observed acoustic feature sequence and \(W\) the underlying word sequence. The Bayesian decomposition
has organized the discipline since Jelinek and Bahl (1976) and remains, in spirit if not always in letter, the structuring principle of every paradigm reviewed below. The acoustic term \(P(X \mid W)\) describes how a word sequence generates a feature stream; the language‑model term \(P(W)\) describes the prior plausibility of a sequence; the marginal \(P(X)\) serves only as a normalizing constant in the argmax. ASR has evolved by progressively replacing each of the three terms by learned components while retaining the formal architecture of the decomposition.
Four broadly successive paradigms structure the field. The classical HMM‑GMM paradigm modelled both the acoustic emission and the temporal dynamics by hand‑crafted, generatively trained probabilistic components. The hybrid DNN‑HMM paradigm retained the temporal HMM scaffolding and replaced the Gaussian emission by a discriminatively trained neural network. The end‑to‑end neural paradigm collapsed the modular pipeline into a single network mapping acoustics to text. The self‑supervised representation paradigm shifted the bottleneck from supervision to representation by pre‑training large encoders on unlabeled audio. Each paradigm absorbed rather than discarded its predecessor: a contemporary system, in particular the systems on which the present thesis is built, exhibits traces of all four. The chapter that follows develops each paradigm in turn, with attention to the mathematics that bears on the methodology of Chapter III.
The classical and hybrid paradigms share the modular architecture inherited from Bahl, Jelinek, and Mercer: an acoustic model, a pronunciation lexicon, a language model, and a decoder. Their successive variants differ in how the acoustic model is parameterized and trained.
The hidden Markov model with Gaussian‑mixture emissions (HMM‑GMM) modelled speech as a hidden discrete Markov chain whose states emit, at each time step, observations drawn from a Gaussian mixture density over short‑frame features (MFCC, PLP). For a state sequence \(s = (s_1, \ldots, s_T)\) and a feature sequence \(X = (x_1, \ldots, x_T)\), the joint likelihood factorizes as
where \(a(\cdot,\cdot)\) are transition probabilities and \(b_s(\cdot)\) are state‑dependent emission densities, each itself a Gaussian mixture. Parameters were estimated by the Baum‑Welch algorithm, an instance of Expectation‑Maximization (EM) whose E‑step computes occupation probabilities via the forward‑backward recursion and whose M‑step re‑estimates means, covariances, and mixture weights in closed form.
Three features of HMM‑GMM deserve mention because they recur in subsequent paradigms. First, the noisy‑channel decomposition was operationalized through the explicit composition of acoustic, lexical, and language‑model components. Second, decoding was formalized as a shortest‑path search over a Weighted Finite‑State Transducer (WFST) obtained by composing the HMM transducer \(H\), the context‑dependency transducer \(C\), the lexicon transducer \(L\), and the language‑model transducer \(G\):
That composition remains, in form, the formal substrate of contemporary beam search. Third, EM as the parameter estimation procedure introduced a vocabulary — latent variables, posterior responsibilities, the contrast between observed and complete‑data likelihood — which the methodology of Chapter III inherits in its variational form.
Beginning with the work of Hinton, Deng, and Yu around 2009 and continuing through Kaldi‑era systems, deep neural networks replaced the Gaussian mixture as the emission model while the decoder, the lexicon, and the language model remained external. Architecturally, this consisted in training a neural network to predict the posterior \(P(s \mid x_t)\) of an HMM state given the acoustic frame, then converting the posterior to a scaled likelihood through Bayes' rule:
where \(P(s)\) is the state prior estimated from the training alignments. The training of the network required an initial alignment, typically produced by an HMM‑GMM system, and proceeded by minimization of the frame‑level cross‑entropy.
Three refinements have proven enduring. The time‑delay neural network (TDNN) introduced by Peddinti, Povey, and Khudanpur (2015) replaced the dense feed‑forward layers by sub‑sampled, dilated convolutions over time, giving access to long contextual windows at moderate cost. The factorized variant TDNN‑F, by inserting a low‑rank linear bottleneck between successive TDNN layers, controlled over‑fitting in low‑data regimes. Lattice‑free Maximum Mutual Information (LF‑MMI), introduced by Povey and colleagues (2016), replaced the frame cross‑entropy by a sequence‑discriminative criterion computed without lattices, on a denominator graph that approximates the language‑model marginal:
Hybrid systems remain competitive on small supervised sets, particularly when phonetic resources (pronunciation dictionaries, decision trees) are already available. They are engineering‑intensive: each stage of the pipeline must be tuned and the dependency chain is long. They do not, on their own, address the multilingual or low‑resource extension.
Both HMM‑GMM and DNN‑HMM rely on a single decoding apparatus: weighted finite‑state transducer search. The decoding graph D = H ° C ° L ° G is determinized and minimized offline; at run time, a beam‑pruned Viterbi search produces a lattice from which n‑best lists, confusion networks, and posterior word probabilities can be extracted. The pronunciation lexicon L is the only component of the chain that requires explicit linguistic knowledge; for low‑resource languages it is also typically the bottleneck. The methodology of Chapter III does not rely on a pronunciation lexicon, but inherits from the WFST tradition the habit of expressing language priors as composable weighted transducers.
The second part of the chapter addresses the two paradigms in which the modular pipeline of the classical and hybrid systems is replaced or relegated to a pre‑training stage. End‑to‑end models collapse the pipeline into a single trainable network; self‑supervised models shift the bottleneck from supervision to representation.
Three training objectives still structure the end‑to‑end family. They differ in how they relate the unaligned target sequence \(y = (y_1, \ldots, y_U)\) to the frame sequence \(X = (x_1, \ldots, x_T)\), with \(T \gg U\).
Connectionist Temporal Classification (CTC, Graves et al. 2006) introduces an additional blank symbol and marginalizes over all monotonic, many‑to‑one alignments \(\pi : \{1,\ldots,T\} \to \mathcal{Y} \cup \{\text{blank}\}\) that collapse to \(y\):
The forward‑backward recursion over the alignment lattice makes the gradient tractable. CTC converges rapidly, has few hyperparameters, and tolerates small supervised sets — properties that make it the dominant choice in low‑resource ASR pipelines.
Attention‑based encoder‑decoder models (Listen, Attend and Spell; Chan et al. 2016) drop the monotonic assumption. An encoder produces a hidden sequence \(h = (h_1, \ldots, h_T)\); an autoregressive decoder generates \(y\) token by token, attending at each step to a context \(c_u\):
The loss is the cross‑entropy of the target tokens. Attention models are more expressive than CTC but more data‑hungry; they require regularization mechanisms (label smoothing, scheduled sampling, joint CTC‑attention training) to remain stable on small supervised sets.
RNN Transducer (Graves 2012, Battenberg et al. 2017) re‑introduces a monotonic alignment through an internal predictor network. A joint network combines the encoder state and the predictor state to produce
The lattice over (t, u) admits an efficient forward‑backward recursion and a streaming‑compatible decoder. RNN‑T is the production choice for low‑latency recognition.
The encoder of choice in contemporary end‑to‑end systems is the Conformer (Gulati et al. 2020), which combines convolution and self‑attention in a single block. Each Conformer block applies, in order: a macaron feed‑forward sub‑layer; multi‑head self‑attention with relative positional encoding; a depthwise‑separable convolution module; a second macaron feed‑forward sub‑layer; layer normalization. The block‑level update may be written as
Self‑attention captures long‑range dependencies while the convolution module captures local patterns over a few tens of frames. The Conformer encoder underlies most of the foundation models reviewed in Chapter II and, in particular, the Wav2Vec2‑BERT model that serves as one of the three teachers in the methodology of Chapter III.
Self‑supervised pre‑training on unlabeled audio shifted the bottleneck from supervision to representation. The contrastive objective of Wav2Vec 2.0 (Baevski et al. 2020) masks a fraction of the convolutional features and asks the Transformer to identify, among a set of distractors \(q_k\), the quantized target \(q_t\) aligned with the masked position:
where \(c_t\) is the contextual representation at the masked position, \(\operatorname{sim}\) is cosine similarity, and \(\tau\) is a temperature.
HuBERT (Hsu et al. 2021) replaced the contrastive term by masked prediction over discretized targets obtained by iterative k‑means clustering of MFCCs (first iteration) or of the model's own intermediate representations (subsequent iterations). The objective becomes a standard cross‑entropy over a finite codebook:
WavLM (Chen et al. 2022) added a denoising pre‑training signal by overlapping the input with noise and asking the model to predict the targets of the clean stream. data2vec (Baevski et al. 2022) replaced both discrete and contrastive targets by self‑distilled latent representations of the unmasked input, computed by an exponential moving average of the student.
The practical value of these models for low‑resource ASR is that fine‑tuning a small CTC head on a few hours of supervised audio often matches the performance of a system trained from scratch on substantially larger supervised corpora. The four models listed above are nominally usable for Malagasy; Wav2Vec2‑BERT, the cross‑lingual variant pre‑trained on Wav2Vec 2.0 plus BERT‑style masked language modelling, serves as one of the three teachers retained in Chapter III.
The four paradigms reviewed in the present chapter may be summarized along the four dimensions on which they principally differ.
| Paradigm | Acoustic model | LM integration | Typical data | Key reference |
|---|---|---|---|---|
| Classical HMM-GMM | Gaussian mixture on MFCC | External WFST (G) | 10-500 h supervised, phonetic dict. | Bahl, Jelinek, Mercer 1983 |
| Hybrid DNN-HMM | TDNN-F + Bayes' rule | External WFST (G) | 100-2000 h supervised | Hinton et al. 2012; Povey et al. 2016 |
| End-to-end (CTC/Attn/RNN-T) | Conformer encoder | Shallow / deep fusion | 1000-50000 h supervised | Graves 2006; Chan 2016; Gulati 2020 |
| Self-supervised + fine-tune | Transformer pre-trained on unlabeled audio | Optional shallow fusion | 10000-1M h unlabeled + small sup. | Baevski 2020; Hsu 2021 |
The four paradigms reviewed in the present chapter share, in spite of their architectural differences, a single inferential skeleton: the inversion of a noisy generative channel under a learned acoustic likelihood and a learned language‑model prior. Each paradigm refined that skeleton on one of its sides — the acoustic model first, the language‑model integration next, the representation last. Each paradigm, however, is predominantly designed and evaluated on high‑resource languages. Chapter II examines how the four paradigms extend to multilingual and low‑resource settings, identifies the four open methodological problems on which the present thesis is centred, and describes the data substrate constructed for the empirical instantiation on Malagasy.