PROVA‑MG — Probabilistic Routing Of Variational Alignments for Malagasy — is formulated as a latent‑variable probabilistic system that aggregates heterogeneous evidence under explicit calibration and explicit constraints. The present chapter derives the framework, specifies its five components and their algorithmic implementation, analyses their complexity, and describes their integration with the license‑first data pipeline.
Low‑resource ASR is treated, in the present work, not as a single model trained on a single objective with a single source of supervision, but as a probabilistic system that aggregates heterogeneous evidence. The substrate of that system is a graphical model in which the transcript y is jointly generated with several latent variables — dialect d, register r, code‑switching identity c — conditioning the acoustic emission and the language‑model prior. The estimation procedure is a constrained variational EM whose constraints are derived from the agreement of heterogeneous teachers, from a calibrated confidence head, and from a typological prior trained on the Austronesian family.
The chapter is organized in three parts. Part III.1 derives the theoretical formulation of the system: graphical structure, likelihood, prior, posterior approximation, and learning objective. Part III.2 specifies the five interlocking components in algorithmic form, with derivations, pseudocode, and complexity analysis. Part III.3 describes the training protocol, the optimization schedule, and the integration with the data pipeline of Chapter II.
Let x denote the acoustic signal and y the transcript. The latent variables of the system are d (dialect, categorical with K_d levels), r (register, categorical with K_r levels, distinguishing read, broadcast, conversational, and ceremonial speech), and c (per‑segment code‑switching identity, taking values in {MG, FR, mixed}). The generative model factorizes as
which encodes three modelling decisions. The dialect and register are marginally independent, reflecting the empirical observation that a Merina speaker can be recorded in either a read or a conversational register. The code‑switching identity is conditioned on register and dialect because code‑switching is itself sociolinguistically structured. The acoustic emission depends on the transcript, the dialect, and the register, but not on the code‑switching identity beyond what is already encoded in the transcript and the dialect.
Exact inference under the model above is intractable. A mean‑field variational posterior is therefore introduced:
Each factor of q is parameterized by a neural network. The factor \(q_y\) is the transcript posterior of an ASR head; the factors \(q_d\) and \(q_r\) are classification heads predicting dialect and register from the acoustic input; the factor \(q_c\) is a sequence labeller predicting code‑switching identity per segment of the transcript. The mean‑field factorization is admissible because the model decouples d, r, and c at the prior level; residual dependence is captured by the conditioning of every factor on x.
The learning objective is the standard evidence lower bound (ELBO) on the marginal log‑likelihood of the observed data:
where \(\theta\) parameterizes the generative model and \(\phi\) the variational posterior. On the supervised subset of the data the transcript y is observed and the corresponding factor of q is replaced by a delta on the gold transcript; on the unsupervised subset y is latent and the variational posterior is unrolled jointly with the other latent factors.
On the union of the supervised and unsupervised subsets the parameters \(\theta\) and the variational posterior q are estimated jointly by an EM procedure. The E‑step minimizes the Kullback‑Leibler divergence between q and \(p_\theta(\,\cdot \mid x)\) subject to a constraint set C; the M‑step maximizes the expected complete‑data log‑likelihood plus a parameter regularizer:
The constraint set C is the locus of the methodological contribution. It comprises four constraints, each derived from one of the open problems identified in Chapter II.
Under the four constraints, the E‑step is a constrained optimization that admits, in the discrete‑support case, a projected‑gradient or barrier‑method implementation; the M‑step retains its standard gradient form on \(\theta\). A short convergence argument (Section III.3) shows that under C1‑C4 the iteration cannot exhibit the runaway collapse characteristic of unconstrained pseudo‑labeling.
The theoretical formulation of Part III.1 is instantiated by five interlocking components. Each is specified below with its derivation, its pseudocode, and a complexity analysis.
Whisper‑large‑v3, MMS, and Wav2Vec2‑BERT are treated as three approximate posteriors over the transcript rather than as competing baselines. Each model encodes a different inductive bias (weak supervision, religious‑text alignment, self‑supervision); their disagreement on a given Malagasy segment is therefore informative. For each unlabeled segment x, the approximate posteriors \(q_W(y \mid x)\), \(q_M(y \mid x)\), \(q_B(y \mid x)\) are computed in n‑best form and a learned routing function is defined:
The features consumed by r include: per‑teacher top‑1 confidence (calibrated), n‑best overlap between teachers, pairwise edit distance between top‑1 transcripts, posterior entropy of each teacher, length agreement, and the Jensen‑Shannon divergence between teacher posterior distributions over a shared lattice. The routing function is trained as a gradient‑boosted classifier on the supervised set, with the four routing outcomes as targets supplied by the agreement of each teacher's top‑1 with the gold transcript.
Complexity. The dominant cost is the three forward passes through the foundation models. Let n denote the number of segments and T the mean segment length in frames. Inference is O(n T (d_W + d_M + d_B)), where d_* is the model dimension. Feature extraction and routing add O(n B^2) where B is the n‑best width. The routing classifier itself is sub‑linear in n. The dominant memory cost is the storage of the three posteriors, mitigated by retaining only the top‑B paths.
Malagasy belongs to the Malayo‑Polynesian subgroup. Its phonotactics and morphology overlap measurably with Indonesian, Malay, Cebuano, and Tagalog, all of which have substantially larger text corpora. A phonotactic and morphotactic prior \(P_{\mathrm{typ}}\) is trained on Austronesian text and incorporated into decoding through a calibrated log‑linear mixture:
The hyperparameters \(\alpha\) and \(\beta\) are tuned on the development set by grid search subject to the constraint that the mixture remains a valid log‑posterior (positive weights summing to a fixed total). \(P_{\mathrm{typ}}\) is implemented as a sub‑word language model over a shared phonotactic vocabulary; transfer to Malagasy proceeds through a morphological mapping that aligns Austronesian affixes to their Malagasy cognates.
Complexity. The prior is evaluated during beam search, adding O(B T |V|) to the decoding cost, where |V| is the vocabulary size and B the beam width. The cost is amortized by caching the language‑model state across beam expansions.
On the extended dataset \(D = D_{\mathrm{sup}} \cup D_{\mathrm{unsup}}\), parameters \(\theta\) and latent transcripts are inferred jointly through the constrained EM derived in Part III.1. The four constraints C1–C4 act on the E‑step support.
Non‑collapse argument. Without constraints, iterative pseudo‑labeling can exhibit confidence inflation accompanied by accuracy degradation — the canonical collapse mode. Under C1 (confidence floor on calibrated confidence) the support of \(q^*\) cannot expand into low‑confidence regions; under C2 (KL drift bound) the variational posterior cannot move arbitrarily fast across iterations; under C3 (teacher agreement) the support is intersected with the union of teacher n‑best lists, which by construction reflects independent inductive biases; under C4 (typological consistency) the support is pruned of transcripts that violate Austronesian phonotactics. The intersection of these four constraints bounds the iterative drift in both confidence space and hypothesis space.
Complexity. Each EM iteration is \(O(|D_{\mathrm{unsup}}| \,(\text{decoding} + \text{feature extraction}))\) in the E‑step and O(|D| epochs) in the M‑step. The total cost is N times that of a single fine‑tuning run, with N typically between three and six.
The graphical model of Part III.1 makes dialect, register, and code‑switching latent variables jointly inferred with the transcript. The acoustic model factorizes as p(x | y, d, r) and the language model as p(y | d, c, r). Dialect‑specific adapters condition on d, register‑specific parameters on r, and code‑switched segments are routed through a French acoustic model.
Approximate inference proceeds by mean‑field updates: each factor \(q_d\), \(q_r\), \(q_c\) is updated holding the others fixed, and the transcript factor \(q_y\) is updated by decoding with the current expected dialect and register adapters.
Complexity. The dominant cost is the repeated decoding inside the loop. In practice, two to three coordinate‑ascent passes suffice for convergence; total cost is O(I T (encoder + decoder)) where I is the number of passes.
ASR models are typically over‑confident, and miscalibrated confidences would compromise both the routing of Component 3.1 and the constraints of Component 3.3. A post‑hoc calibration model is therefore trained on the supervised set.
where \(g_\phi\) is implemented either as temperature scaling (single parameter) or as isotonic regression (non‑parametric) or as a small gradient‑boosted classifier on the feature set. The features are: per‑teacher posterior entropy, three‑way teacher agreement, typological log‑likelihood, segment duration, voice‑activity confidence, dialect‑posterior entropy, and the rank of the top‑1 hypothesis in the consensus lattice. The calibration objective is the negative log‑likelihood on the supervised set plus a smoothness penalty:
The calibration quality is measured by the Expected Calibration Error with n bins:
Complexity. Calibration training is linear in \(|D_{\mathrm{sup}}|\). Use at inference is O(1) per segment after feature extraction.
The acoustic head is initialized by fine‑tuning Wav2Vec2‑BERT on the Tier‑A supervised set with a CTC objective and LoRA adapters of rank 16. The dialect, register, and code‑switching heads are trained jointly on the same supervised set, supervised by the available metadata labels.
The three teachers are run on Tier‑C unlabeled data. The output is stored as compressed n‑best lattices with per‑path posterior scores. This stage is embarrassingly parallel and dominates the compute budget; on the target hardware it requires of the order of one GPU‑week per 10 000 hours of audio.
The routing function and the calibration head are trained on the supervised set with features extracted from the Stage‑2 teacher outputs. The calibrator is then evaluated for distribution shift between the supervised and the unlabeled pool by a held‑out reliability diagram.
The constrained EM loop of Component 3.3 is executed for between three and six iterations, with each iteration consuming the Stage‑2 teacher outputs and producing an updated \(\theta\). The loop terminates either when held‑out loss stops improving or when the KL drift bound saturates.
The final model performs joint mean‑field inference over dialect, register, code‑switching, and transcript on the frozen test sets. Stratified evaluation is then conducted across all three test sets and all four dialect strata.
Optimization uses AdamW with a cosine schedule, peak learning rate 3e‑5 for the LoRA adapters and 1e‑4 for the heads, batch size 16 with gradient accumulation to an effective 64, dropout 0.1 in the adapters and 0.0 in the frozen backbone, label smoothing 0.1 for attention‑based decoders, mixed‑precision (bf16) on the target hardware, gradient clipping at 1.0, and a small auxiliary CTC loss in joint CTC‑attention runs with weight 0.3.
The present chapter has formulated PROVA‑MG as a latent‑variable probabilistic system; specified its five components — co‑inference, typological prior, constrained variational EM, latent decomposition, calibration — in algorithmic form with derivations and complexity analyses; and described a five‑stage training protocol that integrates the framework with the license‑first data pipeline of Chapter II. The next chapter presents the prototype implementation, the experimental matrix, the expected results, the discussion of limitations, and the perspectives of further work.