Chapter IV: Application, Results and Discussion

From methodology to evidence

The framework specified in Chapter III is, at the moment of writing, instantiated in a prototype implementation whose components have been brought separately to functional readiness and whose integration is under way. The present chapter is organized in four substantive parts plus the validation plan inherited from earlier drafts. Part IV.1 describes the prototype: software stack, model registry, infrastructure. Part IV.2 lays out the experimental matrix in placeholder form, with full headers and labels so that, on each completed experiment, the corresponding cell is to be filled. Part IV.3 discusses the expected interpretation patterns, limitations beyond the existing list, and threats to validity. Part IV.4 develops the perspectives of further work, deployment, ethics, and dialect equity. The publication plan, the positioning with respect to prior art, and the eight‑phase validation procedure are then preserved in their earlier form.

IV.1 Application: prototype implementation

IV.1.1 Software stack

The prototype is built in Python 3.11 on top of the PyTorch 2.x runtime. Data loading and tokenization go through the Hugging Face datasets and transformers libraries; foundation model inference uses transformers and, for Whisper, optionally the faster‑whisper port of CTranslate2. Training uses accelerate for distributed orchestration and peft for LoRA adapter management. Evaluation uses jiwer for WER and CER, scipy for paired bootstrap and Wilcoxon tests, and scikit‑learn for calibration (temperature scaling and isotonic regression). Experiment tracking goes through Weights & Biases with offline fallback to a local SQLite store; every run logs the pinned environment hash and the seed.

IV.1.2 Model registry

A local model registry — a versioned directory tree with a JSON index — holds, for each foundation model and each fine‑tuned variant: the weights (or a pointer to the public checkpoint), the configuration, the tokenizer, the model card, and the training manifest. Three foundation checkpoints are registered as teachers; subsequent fine‑tuned variants are tagged with the upstream commit hash of the configuration and the data manifest version on which they were trained.

IV.1.3 Infrastructure

Training and inference run on a small heterogeneous cluster comprising consumer‑grade and server‑grade GPUs (A100‑80GB, A40, RTX‑4090‑class), with mixed‑precision (bf16) where supported and fp16 elsewhere. Long inference passes on the Tier‑C unlabeled pool are sharded across nodes and the resulting n‑best lattices are compressed and stored on shared object storage. A reproducible Docker image, archived with its hash, packages the runtime environment.

IV.2 Results (placeholder structure, pending experimental run)

The following tables fix the structure of the experimental matrix. Cells are marked tbd pending the corresponding experimental run; column headers and row labels are the load‑bearing content at the present stage. Significance tests reported in the manuscript will be paired bootstrap and Wilcoxon with Bonferroni correction across the family of comparisons.

Table 4.1 — Headline WER (%)

Table 4.2 — Component leave-one-out ablation (delta WER vs full)

Table 4.3 — Routing vs averaging vs voting (WER %)

Table 4.4 — Typological prior leave-one-language-out (WER %)

Table 4.5 — Constrained EM, leave-one-constraint-out (WER %)

Table 4.6 — Expected Calibration Error before and after calibration

Table 4.7 — Dialect-stratified WER (%)

Table 4.8 — Code-switching density buckets

Table 4.9 — Compute cost comparison

Figure 4.1 — Reliability diagram (placeholder description)

The reliability diagram, to be produced after the run of Phase 3, plots, on the horizontal axis, the binned predicted confidence (ten equal‑width bins from 0 to 1) and, on the vertical axis, the observed fraction of correct top‑1 transcripts in each bin. The diagonal corresponds to perfect calibration. Curves are reported for raw confidence, temperature‑scaled, isotonic‑regressed, and boosted‑classifier calibration, on the FLEURS‑MG development set and on a Tier‑C subset to test distribution shift.

IV.3 Discussion: expected patterns, limitations, threats to validity

IV.3.1 Expected interpretation patterns

Under the working hypotheses of the General Introduction, four interpretation patterns are anticipated. First, the gain of routing over averaging is expected to concentrate on segments of intermediate teacher agreement: where all three teachers agree, averaging and routing coincide; where teachers disagree strongly, routing should defer to human review rather than commit to a likely‑wrong consensus. Second, the typological prior is expected to act selectively on low‑frequency morphology and on rare segmental sequences, with little effect on high‑frequency lexical items already well covered by the Malagasy language model. Third, the constrained EM is expected to improve held‑out WER monotonically across iterations under the full constraint set, and to exhibit clear divergence under the constraint‑ablated runs. Fourth, the joint latent inference is expected to improve performance on under‑represented dialects (Betsimisaraka, Sakalava, Antandroy) without degradation on Merina, and to handle code‑switched segments better than a single Malagasy‑only system.

IV.3.2 Limitations beyond the existing list

Several limitations condition the interpretation of the expected results beyond those listed in Part IV.D below.

• Dialect category granularity. The four‑way dialect partition (Merina, Betsimisaraka, Sakalava, Antandroy) is a simplification: at least eighteen dialect groups are recognized linguistically, and several of the four groups are themselves internally heterogeneous. The latent variable d is therefore coarser than the linguistic reality.
• Calibration set size. The supervised set on which the calibrator is fit is, in absolute terms, small. The distribution‑shift study of Phase 3 is intended to detect calibration breakdown, but it cannot in itself prevent it.
• Anchoring bias in correction. Tier‑B annotators see the seed transcription before correcting it, which is known to introduce an anchoring bias toward the seed. A double‑annotation sample is therefore retained on which the bias is measured and reported.
• Compute‑limited iteration count. The constrained EM is in principle iterated until convergence; in practice the iteration count is bounded by available compute. A budget of six iterations is planned and reported as a hyperparameter rather than as a converged result.
• Speech‑LLM comparisons. Speech‑LLMs are not on the load‑bearing path; a comparison may therefore not be available at the date of defense and is recorded as a future work item.

IV.3.3 Threats to validity

Four threats to validity are identified at the methodological level. Construct validity: WER and CER are imperfect proxies for downstream task quality; a small qualitative evaluation on transcripts intended for accessibility and educational applications is therefore retained. Internal validity: the integration of five interacting components creates a risk of confounding between component effects; the leave‑one‑out ablation of Table 4.2 is designed precisely to bound that risk, complemented by selected leave‑two‑out runs on the components whose interaction is theoretically expected to be largest. External validity: the methodology is instantiated on Malagasy; its transfer to other low‑resource Austronesian languages (Tagalog with its own resource gradient, Cebuano, the languages of eastern Indonesia) is a future work item, not a claim of the present thesis. Conclusion validity: significance tests across the full matrix invite multiple‑comparison inflation; Bonferroni correction is applied to the family of headline comparisons.

IV.3.4 Comparison to prior numbers in the literature

Published numbers for Malagasy ASR are sparse and use heterogeneous test sets. Whisper‑large‑v3 zero‑shot WER on FLEURS‑MG has been reported around the high‑forties; MMS fine‑tuned on Common Voice‑MG has been reported in the low‑thirties; XLS‑R fine‑tuned on Common Voice‑MG has appeared in the high‑twenties on the same test set. The thesis targets — at or below 15% WER on read speech and at or below 25% WER on spontaneous speech — are therefore ambitious relative to the published state of the art, and their attainment, if confirmed, would constitute a substantive empirical contribution in addition to the methodological one.

IV.4 Perspectives: future work, deployment, ethics, dialect equity

IV.4.1 Future technical work

Several technical extensions are envisaged beyond the scope of the present thesis. Speech‑LLM integration: a comparison and possibly an integration of the PROVA‑MG output with a speech‑LLM as a downstream rescorer. On‑device deployment: distillation of the integrated framework into a compact (sub‑100M‑parameter) student suitable for inference on mobile hardware. Streaming variant: replacement of the offline encoder by a streaming Conformer with chunked attention, enabling real‑time transcription. Cross‑lingual transfer: instantiation of the framework on another low‑resource Austronesian language to test the generality of the typological prior.

IV.4.2 Deployment and applications

A small set of applications is identified as priority deployment targets, conditional on the attainment of the WER thresholds. Educational support tools transcribe classroom audio for review by pupils and teachers, with manual correction as a learning activity. Accessibility services produce captions for live broadcast and recorded media. Administrative transcription accelerates the conversion of public‑hearing audio into searchable text. Journalistic indexing produces searchable transcripts of interview archives. Each application requires a domain‑specific qualitative evaluation and a written consent regime appropriate to the recording setting.

IV.4.3 Ethics

Three ethical considerations are integrated into the design. Consent and licensing: every recording entering the corpus is accompanied by a verifiable consent and license document, and Tier‑D recordings are excluded from any open release. Speaker privacy: voice models pose a potential re‑identification risk; the published student model is therefore distilled from a teacher trained on a privacy‑screened subset, and a model card discloses the residual risk. Use restriction: model cards explicitly disallow surveillance applications and require the propagation of the use restriction in any derivative work.

IV.4.4 Dialect equity

Dialect equity is treated as an evaluation dimension rather than as a secondary concern. The stratified evaluation of Table 4.7 reports per‑dialect WER explicitly; a deployment is considered usable only when the worst‑dialect WER is within a stated factor of the best‑dialect WER. The factor is conservatively set to two for the present thesis and is intended to tighten as the corpus expands.

Positioning with respect to prior art

Each component of PROVA‑MG draws on existing literature; the contribution lies in the specific formulation of each component and in their integration. The following table summarizes the nearest prior art and the elements that are new in the present work.

The integration of the five components is itself a contribution: the calibrated confidence (3.5), the agreement signal (3.1), and the constraint set (3.3) are shared across components, so that they reinforce one another rather than operating in isolation.

Discussion: limitations and mitigations

The methodology presents several limitations that condition the interpretation of its expected results. They are listed below together with their mitigations.

• Scope. Five components constitute an ambitious programme. A defensible minimal thesis comprises Components 3.1, 3.3, and 3.5; Components 3.2 and 3.4 may be reported as future work if time becomes insufficient.
• Theoretical depth. Components 3.3 and 3.5 require non‑trivial variational inference, EM convergence, and calibration theory. Adequate probabilistic supervision is therefore necessary; if it is unavailable, Component 3.3 is restricted to empirical claims supported by proof sketches.
• Empirical uncertainty. The typological prior of Component 3.2 may fail to reduce WER if the Malagasy language model is sufficiently strong. The corresponding ablation is therefore essential, and a negative result is itself a publishable outcome.
• Compute. Operating three teachers at scale is costly. Distillation of the co‑inference into a single student model after corpus construction is therefore planned; teachers are required at training time but not at deployment.
• Priority of novelty. Early publication is preferred. The SIGUL submission anchors priority on the data substrate, and an early Interspeech submission on Component 3.1 anchors priority on the methodological contribution.