Blog Post

Part 2 of our series on HARMLET, our healthcare speech-AI collaboration with Aarhus University. [Part 1]

“Mewɔ headache na me body temperature wɔ soro.”

Twi supplies the grammar; English supplies the medical vocabulary. Linguists call this code-switching, and in Ghanaian clinical conversation it is not an occasional lapse but the ordinary register — medical terms are learned in English, so that is the language they surface in, mid-sentence, for speakers whose lives are otherwise conducted in Twi.

Two points are worth stating. Code-switching is fluent, rule-governed speech, not broken language: a system built for Ghanaian healthcare must model it as the norm rather than treat it as noise. And it is the hardest thing we are asking our models to do.

Why recognition fails at the switch

Speech recognition models are trained overwhelmingly on monolingual data, and three problems follow. Most speech recognisers assume a single language per utterance, so a sentence that changes language mid-stream forces the model to commit to the wrong language or oscillate between hypotheses. The acoustics shift: an English word inside Twi speech is pronounced with Twi phonology, so it no longer sounds like the English the model learned. And therefore the vocabulary gap — the English medical term is absent from the recogniser’s Twi lexicon, so the audio is forced onto the nearest Twi word available.

Where these failures land is the problem. The words most likely to appear in English inside a Twi sentence are drug names, dosages, symptoms and test results — the content a clinical system cannot afford to lose.

Why the standard metric hides it

The field’s standard measure, word error rate (WER), averages over all words. Our example sentence has eight. A model that transcribes seven correctly scores 12.5% WER, a respectable result for a low-resource language — but if the missed word was headache, the transcript is clinically useless while the metric looks fine. Averages are forgiving; medicine is not.

HARMLET therefore evaluates with a second measure alongside WER: the error rate computed only in the neighbourhood of each language switch, which we call CS-WER. Failures at the switches appear in their own column instead of being diluted across the sentence. We expect switch-point accuracy to lag overall accuracy, and would rather track that gap than obscure it.

Making switches visible to the model

The work begins in the annotation. Every word in a training transcript carries a language, and every boundary where the language changes is marked:

Mewɔ   headache      na       my       body   temperature   wɔ    soro
TWI    ENG·MED       TWI      ENG      ENG    ENG·MED       TWI   TWI
       ↑switch       ↑switch  ↑switch

Medical terms carry an additional flag, checked against a controlled vocabulary developed with health workers. The scheme follows the corpora that defined this area — SEAME for Mandarin–English, Bangor Miami for Spanish–English — but to our knowledge no comparable clinical resource exists for any African language pair. Building one is part of HARMLET’s contribution.

These annotations then shape training: switch points are weighted more heavily, so the model spends its capacity where it fails most. The corpus is designed for the same purpose, with clinical dialogue scripts that include code-switching as it naturally occurs, supplemented by a strictly bounded amount of synthetic material — never a substitute for real speech.

Releev AI builds AI-powered telemedicine and clinical decision support that extends hospital care into rural, peri-urban communities and persons living with vision impairment(more on this coming soon), with speech technology for under-resourced languages as its access layer. HARMLET is a research collaboration with Aarhus University supported by iClimate seed funding.