Chapter II: Low‑Resource and Multilingual ASR

Why the four paradigms do not close the gap

The four paradigms reviewed in Chapter I are, in spite of their successive refinements, predominantly designed for and evaluated on high‑resource languages. When transferred to a low‑resource language such as Malagasy, each of them exhibits a characteristic failure mode. Classical and hybrid systems require pronunciation dictionaries and decision trees whose construction is costly. Attention‑based end‑to‑end systems are too data‑hungry to converge on the few hundred hours of supervised Malagasy currently available. Self‑supervised representations close part of the gap by reducing the supervision requirement at fine‑tuning time, but no single self‑supervised model, fine‑tuned on Malagasy alone, has been reported to attain the WER thresholds required by deployment.

The methods reviewed in the present chapter address that gap from three angles: by sharing representation across languages (cross‑lingual self‑supervision and weak supervision), by adapting large models efficiently (LoRA, adapters, prompts), and by exploiting unlabeled data through pseudo‑labeling. The chapter then describes the data substrate of the present thesis, lists the foundation models retained, and concludes by enumerating the open methodological problems that motivate the framework of Chapter III.

II.1 Cross‑lingual and multilingual methods

The first part of the chapter addresses the three families of methods that produce a single model usable across many languages.

II.1.1 Cross-lingual self-supervised representations

The earliest cross‑lingual self‑supervised model, XLSR‑53 (Conneau et al. 2020), extended Wav2Vec 2.0 to a multilingual setting by joint pre‑training on 53 languages drawn from CommonVoice, BABEL, and MLS. XLS‑R (Babu et al. 2022) scaled the same recipe to 128 languages and 436 000 hours of unlabeled audio, and is the standard reference for cross‑lingual self‑supervised speech models at the time of writing. MMS (Pratap et al. 2023, Meta AI) further scaled to more than 1 100 languages, primarily through the alignment of religious text recordings (the New Testament corpus) supplemented by other licensed sources. Wav2Vec2‑BERT (Seamless Communication 2024) combines a Wav2Vec 2.0 encoder pre‑trained on roughly 4.5 million hours with a BERT‑style masked language modelling head trained jointly on text, producing a model that doubles as a speech encoder and as a phonetic representation extractor.

All four of these models nominally cover Malagasy. Their zero‑shot WER on FLEURS‑MG is, however, high — typically far above the threshold of usability — and varies substantially across the dialects represented in the test set.

II.1.2 Large-scale weak supervision: Whisper and its variants

Whisper (Radford et al. 2023) takes a different approach. Instead of self‑supervised pre‑training followed by supervised fine‑tuning, it trains a single encoder‑decoder on 680 000 hours of audio paired with weakly supervised transcripts collected from the open web. The training objective is straightforward cross‑entropy on the paired data, with special tokens that distinguish languages, tasks (transcription, translation), and conditions (no‑speech, timestamp).

USM (Universal Speech Model, Zhang et al. 2023, Google), Canary (NVIDIA), OWSM (Open Whisper‑style Speech Model, ESPnet community), and OLMoASR are subsequent variants or open reimplementations. Whisper‑large‑v3 currently produces the strongest open zero‑shot transcripts for Malagasy and is therefore used as the first‑pass teacher in the pipeline of the present work.

II.1.3 Speech-LLMs and unified models

Recent systems feed speech to a large language model either as discrete tokens or through a learned projection (SALMONN, Qwen‑Audio, Phi‑4‑Multimodal). For high‑resource languages, quality rivals that of dedicated ASR. For low‑resource languages, performance is bounded above by the text‑side coverage of the underlying LLM; for Malagasy in particular, the text coverage of contemporary LLMs is uneven and the resulting transcripts exhibit characteristic hallucinations on low‑frequency lexical items. The present work does not place speech‑LLMs on the load‑bearing path of the methodology, but considers them as candidates for a future comparison.

II.2 Adaptation, pseudo‑labeling, and license‑first data

The second part of the chapter addresses the three families of methods that adapt or supplement a pre‑trained model with target‑language resources, and presents the data substrate constructed for the present thesis.

II.2.1 Parameter-efficient adaptation: LoRA, adapters, prompts

Full fine‑tuning of a foundation model with several hundred million parameters on a few hundred hours of Malagasy audio is computationally expensive and risks over‑fitting. Parameter‑efficient adaptation methods address both concerns. Low‑Rank Adaptation (LoRA, Hu et al. 2022) inserts a trainable low‑rank update \(\Delta W = B A\) alongside the frozen weight matrix \(W\):

where \(A\) and \(B\) have rank \(r\) much smaller than the model dimension. Adapter layers (Houlsby et al. 2019) insert small bottleneck residual modules after each Transformer sub‑layer. BitFit trains only the biases. Prompt tuning learns soft prefixes prepended to the input. For low‑resource ASR these methods reduce both GPU cost and over‑fitting risk, and they admit composition: language‑specific LoRA adapters can be combined with dialect‑specific adapters, which is the route taken in Component 3.4 of Chapter III.

II.2.2 Semi-supervised learning, pseudo-labeling, and self-training

Iterative pseudo‑labeling (ITP) and its variants — slimIPL (Likhomanenko et al. 2021), Kaizen (Manohar et al. 2021), Noisy Student (Park et al. 2020), momentum pseudo‑labeling (Khurana et al. 2022), the massively multilingual pseudo‑labeling regime of Hsu et al. (2022) — share a common procedure. A seed model is trained on the supervised set, transcribes the unlabeled pool, filters the resulting pseudo‑labels by confidence, and is retrained on the union of supervised and filtered pseudo‑labeled data. Variations concern the filtering criterion (raw confidence, posterior entropy, agreement between augmented versions), the teacher‑update rule (replacement at each iteration, exponential moving average, fixed teacher), and the prevention of collapse (confidence thresholds, KL drift bounds, noise injection on the student). Recent extensions to foundation models include uDistil‑Whisper (Park, Berrebbi et al. 2024), STAR for unsupervised adaptation of speech foundation models (Hu et al. 2024), and large‑scale pseudo‑labeling for low‑resource Indic ASR (Bhogale et al. 2024). The empirical collapse phenomenon — rising confidence accompanied by accuracy degradation — is documented systematically by Likhomanenko, Lugosch & Collobert (2023) for continuous soft pseudo‑labels, and is the empirical motivation for the constraints introduced in Chapter III.

Beyond ASR, the theoretical reading of pseudo‑labeling as constrained posterior inference has been established for two decades. Posterior Regularization (Ganchev, Graça, Gillenwater & Taskar 2010) formalizes the E‑step as q* = argmin KL(q || p_theta(. | x)) subject to expectation constraints, and Generalized Expectation Criteria (Mann & McCallum 2008, 2010) cast weakly supervised learning as imposing expectation constraints on the model posterior. Alternating projections (Bellare, Druck & McCallum 2009) and Unified EM (Samdani, Chang & Roth 2012) unify the parametric family of constrained‑EM variants. Cross‑lingual transfer with posterior regularization was demonstrated by Das & Petrov (2013). On the theoretical side, KL‑constrained updates between iterates are a standard trust‑region device (Schulman et al. 2015), and recent work has begun to characterize the non‑collapse of iterative retraining of generative and discriminative models (Bertrand et al. 2024; Aubin et al. 2025; Wei, Shen, Chen & Ma 2021). Bayesian and uncertainty‑aware self‑training (Mukherjee & Awadallah 2020; Rizve et al. 2021; Lienen & Hüllermeier 2021; Rodemann et al. 2023) and the FixMatch / FlexMatch / FreeMatch confidence‑threshold family (Sohn et al. 2020; Zhang et al. 2021; Wang et al. 2023) constitute the immediate methodological neighbourhood. The present thesis re‑formalizes iterative pseudo‑labeling as a constrained variational EM procedure in Chapter III (Component 3.3); its constraint set instantiates posterior regularization for the ASR setting and adds a between‑iteration KL drift bound together with an explicit non‑collapse argument that, to our knowledge, are not stated in this combined form in the prior literature.

II.2.4 Multi‑teacher and heterogeneous‑teacher pseudo‑labeling

System combination has a long history in ASR. ROVER (Fiscus 1997) combines the hypotheses of several recognizers by voting at the level of a confusion network. Confidence‑based ensembles of end‑to‑end ASR models, in which the most‑confident model's output is selected per utterance, were revisited by Gitman et al. (2023) and shown to beat dedicated language‑identification routing. Knowledge distillation from multiple foundation models was explored by Yang et al. (2023) and, in the multilingual setting, by Yan et al. (2024) through simple N‑best re‑ranking. The recent literature on multi‑ASR pseudo‑labeling is the closest immediate antecedent of the heterogeneous co‑inference of Chapter III. The Pratinidhi framework (Kulkarni et al. 2024) combines multiple ASR transcribers with agreement‑based filtering, Rényi entropy, and a SONAR similarity evaluator; Carofilis et al. (2025) implement multi‑domain consensus filtering across Whisper, Zipformer, and Parakeet by CER majority vote; Rangappa et al. (2025) propose multi‑stage filtering of pseudo‑labels generated by two teachers; Prakash et al. (2025) arbitrate three‑way ASR disagreement with a speech‑conditioned large language model. Together these works establish that aggregating several foundation‑style teachers improves pseudo‑label quality on low‑resource and multi‑domain ASR. None of them, however, treats the teachers as distinct posteriors within a probabilistic framework, none uses a learned router over their disagreement signal, and none combines Whisper‑large‑v3, MMS, and Wav2Vec2‑BERT specifically; these gaps motivate Component 3.1.

II.2.5 Cross‑family transfer and typological priors

External language models are conventionally integrated into ASR decoding through log‑linear shallow fusion (Klakow 1998), revisited in the end‑to‑end era by Kannan et al. (2018) as the canonical analysis of external‑LM fusion in sequence‑to‑sequence ASR. Cross‑lingual and related‑language LM rescoring for low‑resource ASR has been demonstrated by Khare et al. (2024) for Indic languages and by Currey et al. (2024) for African code‑switching. Massively multilingual adversarial ASR (Adams et al. 2019) and typologically aware rescoring (Chan et al. 2025) explicitly use typological features as a design principle, and Adilazuarda et al. (2024, LinguAlchemy) fuse typological and geographical features for unseen‑language generalization. The URIEL/lang2vec typological knowledge base (Littell et al. 2017) and its extension URIEL+ (Khan et al. 2024) provide the standard machine‑readable representation of language typology used in this line of work; Ponti et al. (2019) survey the broader use of typology in NLP. For the specifically Austronesian context relevant to the present thesis, large‑scale Bayesian phylogenetic analyses of the family (Gray, Drummond & Greenhill 2009; Bouckaert et al. 2018) and Bayesian phonotactic modelling of unit discovery (Ondel et al. 2017) provide the linguistic and statistical underpinnings of the prior P_typ introduced in Chapter III. FormosanBench (Cui et al. 2025) benchmarks low‑resource Austronesian languages in the era of large language models and confirms the present need for family‑level transfer mechanisms. None of these works, however, constructs a phonotactic and morphotactic Austronesian prior over a shared sub‑word vocabulary and incorporates it as an explicit constraint on the pseudo‑label posterior of an ASR system — the contribution claimed in Component 3.2.

II.2.6 Joint dialect, register, and code‑switching ASR

Each pairwise combination among dialect, register, and code‑switching has been addressed in the literature. Joint dialect‑conditioned ASR with a single sequence‑to‑sequence model was introduced by Li, Sainath et al. (2018); subsequent work includes VoxArabica (Waheed et al. 2023), DialectMoE (Yu et al. 2024), GLoRIA gated low‑rank adaptation (Mundnich et al. 2026), and dedicated work on Swiss German (Khosravani et al. 2021). Joint code‑switching and ASR with explicit factorization was formalized by Yan et al. (2021) as p(y_bilingual) = f(p(y_lang1), p(y_lang2)); Kulkarni et al. (2023) model the code‑switching identity as a per‑frame latent binary sequence that gates the flow of information through language‑specific adapters — the closest published precedent for the q_c factor of Component 3.4. The intersection of dialect and code‑switching is treated in the multilingual Arabic system of Chowdhury et al. (2021). Language‑routing mixtures of experts have been proposed for multilingual and code‑switching ASR (Wang/Kong et al. 2023; Zhao et al. 2024). Disentangled latent‑variable speech models (the factorised hierarchical VAE of Hsu, Zhang & Glass 2017; Hsu & Glass 2018) and factorised speaker‑environment adaptive training (2023) are conceptual antecedents for the variational treatment of the latents. Register as an explicit latent variable inferred jointly with dialect and code‑switching, and the three‑way mean‑field coordinate‑ascent inference with expected adapters inside the decoding loop, are not, to our knowledge, attested in the published literature.

II.2.7 Confidence calibration for ASR and self‑training

The canonical reference for feature‑based confidence calibration in ASR is Yu, Li & Deng (2011), which trains a maximum‑entropy calibrator on word‑level and lattice features and uses the calibrated confidence to gate unsupervised data for semi‑supervised acoustic‑model training — the pipeline of which Component 3.5 of the present thesis is the modern, foundation‑model‑era extension. Modern post‑hoc calibration is dominated by temperature scaling and isotonic regression (Platt 1999; Naeini, Cooper & Hauskrecht 2015; Guo et al. 2017), evaluated by the Expected Calibration Error. Multi‑task confidence estimation with explicit deletion modelling was proposed by Kumar et al. (2021). Fast entropy‑based word‑level confidence for end‑to‑end ASR was introduced by Laptev et al. (2023). Recent work has adapted Whisper itself as a confidence estimator (Aggarwal et al. 2025) and addressed the specific problem of identifying and calibrating overconfidence in noisy speech recognition (Huo, Zhang & Tang 2025). In the broader semi‑supervised learning literature, FixMatch (Sohn et al. 2020), FlexMatch (Zhang et al. 2021) and FreeMatch (Wang et al. 2023) supply the canonical confidence‑threshold and curriculum‑thresholding mechanisms that the present work generalizes to a learned, linguistically‑informed calibrator feeding both a router and a constrained‑EM confidence floor.

II.2.8 The state of Malagasy and Austronesian low‑resource ASR

The published literature on Malagasy ASR is, at the time of writing, exceedingly thin. VoxMg (Ramanantsoa 2023), a 10.8‑hour transcribed corpus accompanied by a baseline model at approximately 33 % WER, is the single peer‑reviewed contribution dedicated to Malagasy ASR. The Malagasy NLP survey of Ratianantitra (2023) treats ASR briefly, and the recent Malagasy sentiment‑analysis corpus (Ratianantitra et al. 2025) is concerned with text rather than speech. A small community dataset of approximately three hours has been released by Angelina & Rabetokotany (2020). On the foundation‑model side, MMS (Pratap et al. 2024) reports an aggregate average across more than one thousand languages, of which Malagasy is one, but does not publish a separately interpretable Malagasy WER on a public benchmark. AfriHuBERT (Alabi et al. 2025) continues pre‑training on 1 226 African languages but does not include Malagasy in its FLEURS evaluation. The two systematic reviews of African low‑resource ASR published in 2025 (Olatunji et al.; the arXiv survey 2510.01145; the benchmarking study 2512.10968) cite Malagasy at most once. To the best of the present search, no peer‑reviewed WER on FLEURS‑Malagasy has been published at the time of writing — itself a benchmarking gap that the empirical chapter will address.

At the level of the Austronesian family, the closest methodological cousin is the Te Reo M⁁ori work of Coto‑Solano et al. (2024), which uses synthesized data selection by score‑distribution matching to fine‑tune Whisper‑large‑v3 on a single foundation model; the multilingual evaluation of Indonesian ASR by Aisyah et al. (2024) compares foundation models on dialectal variability; FormosanBench (Cui et al. 2025) is the dedicated low‑resource Austronesian benchmark. None of these works deploys the heterogeneous‑teacher routing and constrained variational EM architecture of the present thesis. At the regional level, Comorian work (Diarra et al. 2024 on Bambara; transfer‑learning from Swahili 2024) and Mauritian Creole (Noormamode et al. 2018) are useful neighbours but not direct competitors. The ethical reflection of Mahelona et al. (2023) on the use of large Whisper‑style models for indigenous languages is taken as a normative reference for the data‑licensing tier of Chapter II.

Active institutional actors in Madagascar — LIAM (Laboratoire d'Intelligence Artificielle de Madagascar), IndabaX Madagascar, university groups at Antananarivo and Fianarantsoa, and emerging commercial offerings — signal that the field will not remain empty for long; the present thesis takes that prospective competition as an additional argument for publishing an explicit methodological framework rather than a single benchmark number.

II.2.3 Data construction, licensing, and reproducibility

License‑first data construction has become a research contribution in its own right. The present work constructs its data substrate through a nine‑stage pipeline whose outputs are partitioned into four tiers.

• Tier A — gold supervised: professionally transcribed audio with double annotation, dialect and register labels, and a permissive license. Target volume: 200–400 hours.
• Tier B — silver supervised: machine‑assisted transcription verified by a single annotator. Target volume: 600–1 000 hours.
• Tier C — licensed unlabeled: audio collected under licenses that permit derivative model training, without transcripts. Target volume: 10 000–30 000 hours.
• Tier D — restricted: data whose license requires explicit, per‑use authorization. Used only with explicit consent.

Each stage of the pipeline produces machine‑readable manifests, persistent identifiers for each segment, and provenance metadata sufficient to reconstruct the corpus from public sources. Datasheets, model cards, and a Zenodo‑archived DOI accompany every release. The data substrate is treated, in the methodology of Chapter III, as a fixed input rather than as a degree of freedom of the modeling procedure.

Foundation models and evaluation datasets

The following two tables summarize, respectively, the foundation models retained as teachers in the methodology of Chapter III and the evaluation datasets used in the experimental matrix of Chapter IV.

Table II.1 — Foundation models considered

Table II.2 — Evaluation datasets

II.3 The four open methodological problems

Several problems are well established at thesis scale and would not, in themselves, constitute a methodological contribution: continued self‑supervised pre‑training for a single low‑resource language; pseudo‑labeling with simple confidence filtering; LoRA cross‑lingual fine‑tuning; dialect‑conditioned adapters; code‑switched ASR with a language‑identification head. Applying any of these to Malagasy would be linguistically informative but methodologically incremental.

The following four problems, by contrast, remain open at thesis scale and structure the methodology of Chapter III.

• P1. The principled probabilistic integration of multiple heterogeneous foundation models, treated as distinct posteriors and exploiting their disagreement structure as a routing signal rather than as noise to be averaged away.
• P2. The injection of typological priors from a known language family as Bayesian priors that constrain the decoding hypothesis space where a target‑language‑only language model cannot.
• P3. The formal treatment of iterative pseudo‑labeling as a constrained variational EM procedure with explicit non‑collapse conditions and a convergence analysis under stated simplifying assumptions.
• P4. The unification of dialect, register, and code‑switching as latent variables in a single graphical model with explicit conditional independence structure and a joint approximate inference procedure.

To these four problems is added a transversal requirement, P5, that the confidences consumed by every downstream constraint be calibrated and that the calibration generalize from a small supervised set to a larger, distributionally shifted unlabeled set.

II.4 Closest prior art and the position of PROVA‑MG

The present section makes explicit, for each of the five concerns identified above, the closest published work and the increment claimed by the methodology of Chapter III. The intent is to be honest about lineage: every component of PROVA‑MG has identifiable antecedents, and its novelty lies in their integration and in specific technical additions, not in the invention of a wholly new framework.

• P1 / Component 3.1 — heterogeneous co‑inference. Closest precedents: Gitman et al. (2023, confidence‑based ensembles), Kulkarni et al. (2024, Pratinidhi), Carofilis et al. (2025), Prakash et al. (2025), Rangappa et al. (2025). PROVA‑MG extends this lineage by (i) treating Whisper‑large‑v3, MMS, and Wav2Vec2‑BERT as three posteriors within a single probabilistic framework, (ii) replacing the heuristic argmax with a learned gradient‑boosted router, (iii) producing four routing outcomes (high‑confidence training, human review, reserve, discard) rather than a single keep/discard decision, and (iv) using Jensen‑Shannon divergence over a shared lattice as a primary disagreement feature.
• P2 / Component 3.2 — typological prior. Closest precedents: Klakow (1998) and Kannan et al. (2018) for log‑linear LM fusion; Chan et al. (2025) for typologically aware rescoring; Khare et al. (2024) for related‑language LM rescoring; Adams et al. (2019) and Adilazuarda et al. (2024) for typology‑aware multilingual transfer. PROVA‑MG extends this lineage by constructing an Austronesian phonotactic and morphotactic prior P_typ over a shared sub‑word vocabulary (Indonesian + Malay + Cebuano + Tagalog + Malagasy) and embedding it as an explicit expectation constraint (C4) on the pseudo‑label posterior of an iterative EM. To the best of the present search, no published work has built such a prior for Malagasy or used a family‑level phonotactic prior as a pseudo‑label gate in ASR.
• P3 / Component 3.3 — constrained variational EM. Closest precedents: Posterior Regularization (Ganchev et al. 2010), Generalized Expectation Criteria (Mann & McCallum 2008, 2010), Alternating Projections (Bellare et al. 2009), and Unified EM (Samdani et al. 2012); empirically, the soft‑label collapse analysis of Likhomanenko, Lugosch & Collobert (2023). PROVA‑MG is an explicit instantiation of posterior regularization for iterative ASR pseudo‑labeling; the claimed increment is (i) the integrated four‑constraint set C1 (calibrated confidence floor) + C2 (between‑iteration KL drift) + C3 (multi‑teacher agreement) + C4 (typological consistency), (ii) the between‑iteration KL drift bound C2 which is not part of the classical PR/GE formulation, and (iii) a non‑collapse argument tailored to the iterative pseudo‑labeling regime in the spirit of the recent iterative‑MLE stability results of Bertrand et al. (2024) and Aubin et al. (2025). The thesis explicitly disclaims any reinvention of the constrained‑E‑step framework itself and positions Component 3.3 as its principled extension to the foundation‑model ASR setting.
• P4 / Component 3.4 — latent decomposition. Closest precedents: Yan et al. (2021) for joint code‑switching factorization; Kulkarni et al. (2023) for code‑switching as a latent binary sequence routing language adapters; Chowdhury et al. (2021) for joint dialect and code‑switching Arabic ASR; Li, Sainath et al. (2018) for dialect‑conditioned end‑to‑end ASR; the disentangled latent‑variable speech models of Hsu, Zhang & Glass (2017). PROVA‑MG extends this lineage by including register as an explicit latent jointly inferred with dialect and code‑switching, by formalizing the joint as a graphical model with a mean‑field variational posterior, and by performing coordinate‑ascent updates with expected dialect and register adapters inside the decoding loop. The three‑way (d, r, c) joint latent factorization for ASR has not, to our knowledge, been published.
• P5 / Component 3.5 — calibration. Closest precedent: Yu, Li & Deng (2011), which is the canonical feature‑based ASR calibration paper and itself uses calibrated confidences to gate semi‑supervised data. PROVA‑MG extends this lineage by (i) replacing the maximum‑entropy classifier with a modern post‑hoc calibrator (temperature, isotonic, or gradient‑boosted), (ii) introducing typological and dialectal features drawn from Components 3.2 and 3.4, (iii) piping the calibrated confidence simultaneously into the router of 3.1 and the confidence‑floor constraint C1 of 3.3.
• Integrated framework. The closest integrated systems on multi‑teacher pseudo‑labeling for low‑resource ASR are Pratinidhi (Kulkarni et al. 2024) and Carofilis et al. (2025); on Malagasy specifically, VoxMg (Ramanantsoa 2023) is the only published baseline. None of these integrates the five concerns of Components 3.1–3.5 within a single probabilistic system, and none applies such an integration to Malagasy or to its Austronesian sister languages.

Conclusion of Chapter II

The present chapter has surveyed the cross‑lingual, weakly supervised, and adaptation methods that extend the four paradigms of Chapter I to low‑resource and multilingual settings; it has described the license‑first data substrate retained for the empirical instantiation on Malagasy; it has identified the four open methodological problems — heterogeneous co‑inference, typological priors, constrained variational EM, latent‑variable decomposition — together with the transversal calibration requirement; and it has positioned each of those five concerns against its closest published precedent, naming explicitly the lineages from posterior regularization, multi‑ASR pseudo‑labeling, typological rescoring, dialect‑conditioned ASR, and feature‑based confidence calibration that the framework of Chapter III extends. The methodology of Chapter III is structured around exactly those five concerns. The empirical validation of Chapter IV will test, for each of them, whether the proposed solution delivers a measurable and statistically significant improvement over the closest single‑component baseline, and will report the result whether positive or negative.