Teaching Qwen3-TTS to speak Swahili

Qwen3-TTS is the most natural sounding speech model I run locally, and it clones a voice from a few seconds of audio. It speaks ten languages. Swahili, spoken by something like a hundred million people across East Africa, is not among them.

This post walks through how I taught it Swahili on a 16 GB MacBook. Most of what I learned is about the model rather than about Swahili, and each lesson changed how I think about adding a language that was never in the training data.

How the model reads text

It helps to picture what Qwen3-TTS is doing before trying to change it. The model does not produce audio directly. It produces a stream of audio tokens, 12.5 frames per second, each frame a stack of 16 codes, and a separate decoder turns those codes back into a waveform. (I wrote about the architecture in detail earlier.)

Two details matter for everything below.

First, the text is streamed. The model does not read a whole sentence and then speak it; roughly one text token enters per audio frame, so the model is already producing sound for the start of a word while later tokens are still arriving. Keep that in mind, because it explains one of the stranger bugs later on.

Second, the prompt has a slot for a language tag, a learned vector that says “this is English” or “this is Italian”. A new language needs a vector of its own there.

Tricks that do not need training

My first hope was to avoid training entirely. If Swahili is written phonetically, maybe the model could be persuaded to read it as, say, Italian, which has similar vowels.

It half works, which surprised me. Most words come out recognisably, and every language tag performs about equally badly. The slips cluster where you would expect: none of the ten languages begins a word with a nasal followed by a consonant, as Swahili does in mbuzi (goat) or ndugu (brother), so those are the words the model stumbles on. In the sample below it drops the n from ndugu and says something closer to “enchi” for nchi (country), adding a vowel to make the word pronounceable to itself. Respelling the text (“M-buzi”) helped a little and introduced new errors elsewhere.

The lesson was that a prompt can steer an accent, but it cannot teach sounds the model has never produced.

A small adapter instead of a new model

Retraining a model of two billion parameters is out of reach on a laptop, and it is also unnecessary. The approach that worked is a LoRA adapter: small low-rank matrices added beside the existing weights, trained while the original weights stay frozen. I also meant to train one unused row of the language table into a Swahili tag. A later audit showed my training loop never updated it: the row sits in the part of the prompt that runs without gradients, to save time, so every adapter here uses the average of the ten existing language rows, and the LoRA does all of the work.

The training examples copy how the model is used at inference. Qwen3-TTS clones a voice by continuing a reference clip, so each example is a pair of clips from the same speaker: one plays the reference, the other the target, and the loss is only measured on the target’s audio. The model learns to speak Swahili in whatever voice it is given, which is what lets the finished adapter speak in my own voice from a 16 second recording.

A few practical choices kept this inside 16 GB and a reasonable time budget:

• The reference part of each example runs without gradients, and only the target frames are trained.
• The part of the model that predicts the finer acoustic codes is trained on a quarter of the frames, which halved the time per example.
• The result is saved as an 86 MB file the engine overlays on the original weights at load time, rather than a second 3.9 GB copy of the model.

Training peaks at about 8 GB and takes around 4.5 seconds per step, so a 2,000 step run fits in an afternoon.

One more knob matters. The adapter’s strength can be scaled when it is exported. At full strength the model sometimes failed to stop at the end of long sentences, while at 0.75 it kept most of the gain and stopped reliably. On 25 held-out sentences, read three times each, the error rate of a speech recogniser transcribing the output fell from 5.45% without the adapter to 4.01% with it.

Telling the tokenizer about Swahili sounds

This was the part I least expected to matter, and it turned out to decide which sounds the model could learn at all.

Before the model sees text, a tokenizer breaks it into pieces. Qwen’s tokenizer learned its pieces from other languages, so it splits Swahili in ways that suggest the wrong sounds. “Bunifu” became b + unifu, which the model read as something like “bimoni”. Cutting every word into open syllables (bu ni fu) before tokenizing fixed that class of error.

Syllables were not enough for the harder sounds. Three examples show how much a tokenizer’s choices can mislead:

• mb in mbili. The syllable “mbi” tokenized as m + bi, exactly the same shape as the m in mtoto, where it really is a separate syllable. So the model said “mibili”. Qwen happens to have a single token for mb, and splitting the syllable as mb + i removed the ambiguity. The same applies to nd, ng, nj, nz and mv.
• j. The syllables “ja” and “ji” are common tokens in German, where j sounds like English y, and that is how the model said them. On its own, j mostly begins English words like just, so splitting j from its vowel pulls it toward the right sound.
• ng’. Swahili writes the velar nasal (the ng in English singer) as ng’, as in ng’ombe (cow), and plain ng for the sound in finger. The tokenizer read ng’ as ng followed by an apostrophe. Remember that the text is streamed: by the time the apostrophe arrives, the model has already committed to the finger sound. Mapping ng’ to the phonetic letter ŋ, a token that means nothing else, lets the very first token carry the distinction.

The ng’ fix also taught me to distrust my own explanations. My first version produced three different tokens for the same sound depending on where it appeared: capital Ŋ at the start of a sentence, ŋ after a space, and ŋ inside a word. A forced aligner (more on that below) showed the model getting ng’ right 18 times out of 18 in the middle of a sentence and dropping it 8 times out of 12 at the start. I blamed the rare capital form, made every occurrence the same token and retrained, and the drops got worse: 12 out of 12. I then cut 71 real clips so that each began on an ng’ word, to give the model practice starting on that sound, and that barely helped either.

The cause turned out to be where the sentence sits in the prompt. The target text starts right where the reference audio ends, and my best explanation is that the model often begins speaking a token or so in. A word made of several tokens survives losing its first piece, but ng’, now a single token, simply vanished. Putting a short lead-in ("… “) before every sentence moves the real first word away from that seam. Over 20 renders, a sentence-initial ng’ was dropped 15 times without the lead-in and 3 times with it, and nothing in the middle of sentences changed.

All of these rules live in a small JSON file per language rather than in code: which letters are vowels, which consonant clusters to split, which spellings to map to a single token. The adapter carries that file with it, and the engine tokenizes exactly as the training did. The same format should describe Kikuyu’s ĩ and ũ without new code.

The data mattered more than anything

After all of that, the speech was intelligible and the accent was improving, and it still sounded wrong. Listening to a long paragraph, the problem was the flow: the model read word by word, pausing where no Swahili speaker would.

Measuring this properly was the turning point. I used a forced aligner, a model that takes audio plus its known transcript and finds where each word starts and ends, and counted pauses longer than 0.15 seconds that were not at a comma or full stop. Those are the halting pauses.

The model had learned exactly what it was shown. My main training corpus was recorded by people reading prompts one at a time, carefully and slowly. That is clean audio, and a poor model of how Swahili flows. I had even doubled the weight of two of its speakers for their accent, and they turned out to be the most halting of all.

News anchors are the opposite: fluent, practised, well recorded, and in my case Kenyan, which is the accent I wanted. So I built a small harvesting pipeline around broadcast news. None of its steps is specific to Swahili:

1. Cut each broadcast at natural pauses into clips of 2 to 15 seconds with a voice activity detector.
2. Ask a large multimodal model to transcribe clips four at a time and label each one: studio or field recording, how many speakers, which language, which accent. In batches of fifteen it sometimes attached a transcript to the wrong clip, and in batches of four it rarely did.
3. Check every transcript against its audio with the forced aligner. Correct transcripts fit the audio closely, while a neighbouring clip’s transcript fits badly, and the two cases separate cleanly.
4. Group clips by speaker with a speaker-verification model, because training pairs must share a voice.
5. Score fluency, and keep only the prompted reads that flow as well as the anchors.

About a third of the transcribed clips survive the filters, which left 11.5 hours of news speech from more than 500 speakers, plus 3.8 hours of the more fluent prompted reads. Retraining on that mix changed the result more than every earlier idea combined:

Twelve sentences is a small test, so the error rates are indicative rather than precise. The worst sentence improved from 39% to 14%, and the pause count moved the most. More importantly, the result sounds like reading rather than recitation. Here is the same unseen news sentence, a presenter’s segue into a break, from each adapter:

Long passages

One last problem only shows up in long text. The engine speaks a chapter in segments, and each segment is an independent draw from the model. In Swahili, where the adapter’s voice varied more than the original model’s, that was audible: across a three paragraph passage the loudness spanned 17 dB, and the voice shifted at paragraph breaks.

Two engine changes fixed most of it. Each segment is now generated as a continuation of the one before it, so the model hears its own previous sentence and keeps the same delivery. Then every segment is scaled toward the median loudness of the whole passage, within 6 dB. Over three paragraphs, the loudness range fell from 17 dB to 5.7 dB. These two clips straddle the same paragraph break:

What is left

Even with the lead-in, a sentence-initial ng’ is right 13 times out of 20, the misses mostly a plain n. It sometimes swaps a short word, and it has only been judged by one native listener, me.

Swahili was also never the whole goal. It is the first of the languages I care about; Kikuyu, Luo, Kalenjin, Kamba, Maasai and others have far less recorded speech and no presence at all in models like this one. The parts that transferred are the reason this was worth writing up. The model can learn a new language from a few hours of good speech through a small adapter. The tokenizer can be told about a language’s sounds with a short data file. And fluency can be measured before training rather than discovered afterwards, by counting pauses in the data the model is about to copy.