A 1.75M-Parameter Decoder for Kokoro: No Neural Network at Audio Rate

The previous post shrank Kokoro-82M’s decoder by pruning and distillation, and ran into a wall: the speed ceiling is set by the generator’s last stage, which runs a neural network 4,800 times a second, and that stage is exactly where the quality lives. This post takes the opposite approach. The decoder does all of its learned work at 160 frames per second, and the audio-rate part is fixed signal processing with nothing to learn.

The decoder has 1.75 million parameters, 30 times fewer than Kokoro’s 53.3 million, and runs 33 times faster on one CPU thread. In the last blind test on randomly drawn sentences I heard it as the same as Kokoro on 9 of 12 pairs and preferred Kokoro on the other 3. Two pages earlier, without the critic described below, Kokoro won 12 of 12. Most of this post is about what stood between those two results: a “dirty microphone” sound that spectrogram losses could not remove, and the training signal that did. The last two sections cover what came after, a flaw in how the training targets were made and a faint machine-like tone that two purpose-built measures failed to follow. The weights, sample pairs and a paper with the full method are on Hugging Face .

If spectrograms, harmonics and phase are unfamiliar, the primer in the compression post covers them in one picture.

The design

Kokoro’s text side stays exactly as it is: it decides the sounds, their timing, the pitch contour and the loudness. Only the decoder, which turns that description into a waveform, is replaced.

It has two learned parts and one fixed one:

• The mapper (0.96M parameters) is six convolutional blocks that turn Kokoro’s internal features into a compact description of the sound, 80 times a second. It is conditioned on the voice.
• The renderer network (0.79M parameters) is four more blocks, running at 160 frames a second. For each frame and each of 301 frequency bins (40 Hz apart, 0 to 12 kHz) it outputs how loud the voiced buzz should be there and how much to shift its phase. It also outputs how much noise to add, on a coarser frequency grid but five times per frame.
• The synthesiser has no parameters. A pulse train, one click per vocal-fold cycle at the pitch Kokoro’s text side predicted, supplies every harmonic at once, and white noise supplies breath and consonants. Each is taken into the frequency domain, multiplied by the network’s gains, and turned back into a waveform with an inverse STFT. The pulses use 25 ms windows, long enough to separate harmonics. The noise uses 5 ms windows, for a reason that took a listening session to find and is described below.

Why every frequency bin needs its own gain. The first experiment, before any training, asked how fine that control has to be. Taking Kokoro’s own audio and the pulse-and-noise excitation that drove it, I computed the best possible gain per frequency band and measured how close the result could get to Kokoro:

Smooth, banded gains, the classic source-filter vocoder, are nowhere near Kokoro, while per-bin complex gains are exact by construction. The network therefore has to predict a full per-bin mask 160 times a second, and the project is the question of whether a very small network can learn that.

Speed and size. Nothing runs at audio rate except the STFTs and some multiplications. On one CPU thread the released model, mapper included, produces audio at 0.0042 of real time against 0.1378 for Kokoro’s decoder on the same 240 validation sentences, which is 33 times faster. Whisper (small.en) makes 238 word errors on its audio and 235 on Kokoro’s, a word error rate of 4.69% against 4.63%. From phonemes to waveform the whole pipeline runs at 0.0127 of real time against 0.1411, 11 times faster. Its peak memory is 0.75 GB against 1.4 to 1.7 GB, because Kokoro’s audio-rate activations add 550 to 830 MB during synthesis and the tiny decoder adds under 70 MB. Kokoro’s text side, at 28.5M parameters, is unchanged and now takes 69% of the remaining time. The first models had 8.6M parameters; the 1.75M version sounded the same to me, and a later test (below) confirmed that size is not what limits the sound. These are PyTorch measurements on a laptop, and nothing has been measured on a phone.

Training needed one piece of engineering. A transposed-convolution inverse STFT with a 1,200-sample kernel took 2.9 s per forward pass on the Apple GPU, so both directions were rewritten as unfold plus matrix multiply, which brought a training step from 3.3 s to 0.24 s.

What I heard, round by round

Every candidate went through blind A/B pages: loudness-matched pairs, random sides, hidden repeats, every clip at least five seconds long. The descriptions in quotes are from my notes at the time.

The middle of the project was the hardest part. Over five blind tests, every model without buzz sounded the same as every other (43 of 46 such pairs “same”), and Kokoro beat or tied all of them. Meanwhile I wrote detectors for half a dozen measurable differences (dips, holes, pulse depth, envelope roughness), moved each of them with training, and heard no change. That led to a judge fitted to my own verdicts, which predicts about 80% of them on pages it has not seen.

The dirty microphone

Four fine-tunes with new losses and discriminators all landed where they started, so the question became whether the network was failing or the renderer itself could not sound like Kokoro.

Perfect gains were not enough. For 12 sentences I discarded the network’s gains and fitted them directly to Kokoro’s audio, every frame and every bin free. The spectral loss against Kokoro fell from about 4 to under 1, yet Kokoro still won all 9 blind pairs, and I ticked “dirty” on 16 of 20 fitted clips and none of Kokoro’s.

The renderer’s machinery was fine. I then rebuilt Kokoro’s own audio through the renderer: the pulse train times a complex gain chosen to match Kokoro, plus noise for whatever the pulses cannot carry. Blind, that copy was “same” as Kokoro in 10 of 10 pairs.

The 31 ms averaging made the copy dirty. With the averaging of the harmonic part switched on, the fix for “fragmented”, the same copy lost 10 of 10 pairs to Kokoro and was ticked dirty every time.

The reason is in how voices behave. From one vocal-fold cycle to the next, timing and strength wobble a little (jitter and shimmer), and at high harmonics that wobble pulls the voice away from a perfectly smooth pulse train within a few tens of milliseconds. A 31 ms average cannot follow it, so whatever it cannot follow has to be supplied by the noise, which is fresh and random:

Removing the averaging changed nothing. That was the obvious fix, and I trained it twice, once with human speech and Kokoro as the discriminators’ real examples and once with Kokoro alone. Both models landed on the same spectrogram error (0.416), and on a 16-pair blind page against the smoothed model my verdict was “there is no difference”. Freed of the averaging, the network did produce fast detail, more of it than Kokoro has: 7.2 dB of frame-to-frame gain movement against Kokoro’s 4.3 dB. It was not Kokoro’s detail, since the correlation between the network’s fast phase movement and Kokoro’s was 0.01.

Two oracle tests then closed the cheap exits. Giving the pulse train Kokoro’s exact per-cycle loudness and timing recovered little, because the detail that matters differs from harmonic to harmonic. Adding random detail of the right size, per-harmonic phase and amplitude walks, made my fitted judge rate the audio worse than no detail at all. Only detail aligned with Kokoro’s helped, and a 0.79M-parameter network looking at frame-rate features does not predict it.

The WavLM critic

The training losses up to this point were paired: they compare the model’s audio with Kokoro’s audio for the same sentence, energy at each frequency and frame. Random noise at the right level scores almost as well on those as the real wobble, so noise was the best available answer and every fine-tune converged to it. That pointed at the loss rather than the model.

WavLM is a speech model pretrained on a large amount of real speech. A test on its internal layers showed that the early ones hear the dirt. A linear classifier on a single layer, trained to tell Kokoro’s frames from the tiny model’s, was right on 92% of held-out frames at layers 2 and 3 and on 60% at layers 9 and 10. It also agreed with my ears on the two rebuilt copies: the unsmoothed copy scored as Kokoro (0.90 at layer 3), the smoothed, dirty one as the tiny model (0.08). An earlier attempt of mine had used WavLM as one more paired distance, frame matched against frame, and had done nothing.

The critic uses those early layers the other way round, following StyleTTS 2. WavLM’s first six layers stay frozen, and a small head on top of them (2.4M parameters) is trained to tell 0.8 s clips of Kokoro from 0.8 s clips of the tiny decoder, on unrelated sentences. The decoder is trained to make that guess fail. No frame is compared with any other frame, so noise at the right level no longer scores well: the critic only has to notice that the clip sounds like the tiny decoder. It exists only during training and adds nothing to the shipped model.

With the critic added and the paired losses turned down to a fifth of their weight, I preferred the new model to the old one on 8 of 11 blind pairs and heard the other 3 as the same, the first time any change had beaten that plateau by ear. Dirty ticks fell from 23 of 23 clips to 16 of 22. Kokoro still won 10 of 11 pairs against it.

Which measures saw it coming. My fitted judge predicted “same” for that pair of models and was right on 3 pairs of 11. UTMOS, a public quality predictor that I had set aside because it lost to the fitted judge on earlier pages, rose from 3.83 to 4.05 (Kokoro 4.38). On the later page, the gap in UTMOS between a clip and Kokoro’s version of the same sentence was the best predictor of my verdict: clips I heard as “same” averaged 0.17 below Kokoro, clips where I chose Kokoro averaged 0.40 below, and a cut at 0.3 was right on 19 of 24. The fitted judge got 13 of those 24. A judge fitted on models that all shared one defect was weakest on the first model without it.

Attacks and the 5 ms noise window

Listening to the critic-trained model, I wrote that the voice was clean but “the key points where the strength is placed (final sound of ‘per’ in temper, the starting ’te’) seem smoothened out slightly”. Measured at the twelve strongest onsets of each clip, with 1 ms envelopes at 2 to 8 kHz, the level contrast was intact. The timing was not: the model put 2 to 4 dB more energy than Kokoro into the 3 to 15 ms before each burst and about 2 dB less into the burst itself.

The cause was the window. Noise shaped through a 25 ms window cannot start faster than the window allows: Kokoro’s own magnitudes on random phase smear by 25 dB at that window and by 3.6 dB at 5 ms. So the noise branch moved to its own STFT with 5 ms windows every 1.25 ms, giving it five gains per network frame, while the pulses kept the 25 ms window that separates harmonics. No parameters were added, because the same output layer is read as a coarser frequency grid at five time steps.

The smear before bursts fell from 7.3 dB above Kokoro to 1.7 dB, and on the sentence where I had first noticed the problem I heard the new model as “clearly sounding punchier and better”. On the blind page that followed, I heard this model as the same as Kokoro on 5 of 10 pairs and ticked 6 of its 20 clips dirty, down from 13 of 24 for the model before it. The bursts themselves are still 1 to 2 dB weaker than Kokoro’s, which tripling the weight of the short-window loss did not change.

What did not limit the sound

Three tests ruled out explanations that would have been cheap to act on.

Feeding the renderer perfect features improved the paired fit and nothing a listener or UTMOS would notice. A fourth test, on rebuilt copies of Kokoro, asked whether the pulses should also move to a shorter window. They should not: at matched smoothing time the copy’s error against Kokoro rose from 2.6 at 25 ms to 3.5 at 10 ms and 13.2 at 5 ms, because between pulses there is nothing for a gain to act on.

Creak, overshoots and the weakest voices

A few voices carry the gap. With the 5 ms noise window, 86% of validation clips are within 0.3 UTMOS of Kokoro. The exceptions concentrate in three or four voices, one of which is 0.5 to 0.6 below Kokoro on every model so far. A probe on WavLM’s layer 3 says the same of low-pitched voices, which have not moved since the critic was introduced. That probe reads the layers the model now trains against, so it can be fooled.

Creaky voice. Some of Kokoro’s voices end words in vocal fry: slow, irregular pulses, about five in 100 ms. Kokoro’s pitch track marks those stretches as unvoiced, so the pulse train is silent there and only the noise branch can sound, which I heard as “a heavy hiss”. It is rare, 1.4% of loud 50 ms stretches overall and 7.5% for the worst voice. The fix adds a slow, irregular pulse train wherever the pitch track is unvoiced and lets the existing harmonic gains decide when it sounds. After training, the level at those moments went from 2.7 dB below Kokoro to 1.2 dB below, and UTMOS reached 4.20.

Local overshoots. The “machine whirl” was one harmonic near 1,140 Hz held 8 dB above Kokoro for about 75 ms. Counting every 96 ms patch of spectrum that is at least 6 dB too loud, Kokoro rendered twice with different noise never produces one, and the models produce 6 to 12 per 10 s of audio, most at 1 to 2 kHz. The count grew with each step that moved weight from the paired losses to the critic, from 6.2 before the critic to 12.1 at its highest weight. The critic buys texture and gives up some local accuracy. A loss that penalises only block-energy errors beyond 3 dB brought the count down from 11.3 to 8.5 per 10 s, and the other measures improved with it rather than against it: UTMOS rose to 4.23 and the spectrogram error fell from 0.472 to 0.460. It was the first change since the critic where the paired fit and the outside measures moved in the same direction.

The blind page for both fixes. Against Kokoro I heard the model with the creak pulses and the overshoot loss as the same on 9 of 12 pairs, and the model before them on 6 of 11. Compared directly, the new model won 4 of 11 pairs and the rest were the same. I ticked 9 of its 23 clips dirty, 11 of 22 for the older model and 5 of 23 for Kokoro, against 1 of 47 for Kokoro on the earlier pages. All 5 of Kokoro’s ticks came in pairs where I ticked the model as well: with the two this close, the dirt I was hearing belonged to the voice. In pairs against Kokoro, I ticked the new model alone on 2 of 12 and the older one alone on 5 of 11. One of four hidden repeats changed verdict. The fitted judge again expected about 2 “same” verdicts where I gave 9. The pairs Kokoro still won were on three voices, including the one with the largest UTMOS gap.

A flaw in the training targets

After those fixes the remaining gap had a shape. Over all 240 validation sentences the UTMOS gap to Kokoro grew by about 0.10 for every 10 s of sentence length, and it concentrated in a few voices. Training the whole model on the worst voice alone left that voice’s gap where it was (0.37 below Kokoro before, 0.35 after) while every other voice got worse, which ruled out capacity shared between voices. Neither the pitch track nor the voice vector singled that voice out.

The cause was in how I had prepared the targets. To save time, Kokoro’s audio for each 1.6 s training window had been made by running Kokoro’s decoder on that window alone. Kokoro’s decoder normalises every channel over its whole input, so a window decoded alone is a different signal from the same stretch of the full sentence. Over 128 windows, UTMOS for the window decode was 3.80 against 4.16 for the same 1.6 s cut from a whole-sentence decode. The tiny model, which always runs on whole sentences, already scored 3.82 on those stretches, above its own training target. Every paired loss, and every “real” example shown to the discriminators and the critic, had been a lower-quality Kokoro since the first run.

Cutting each target window from a whole-sentence decode, and running the mapper on whole sentences during training, removed about 70% of the UTMOS gap in one run of 6,000 steps and halved its growth with sentence length. A control that trained equally long on the old targets, with half the weight on the overshoot loss, moved the gap from 0.096 to 0.089. The new targets also made the voice slightly more periodic than Kokoro and put more energy ahead of onsets, 20 to 30 dB below the bursts themselves.

My first attempt at this experiment produced the project’s most instructive bug. A memory watchdog stopped the run before its first checkpoint, the retry branch of my launch script lacked the new flag, and the result I first wrote down was a second control. Every run since then records its arguments, and they are checked against the intended ones before a result is written down.

The remaining machine-like tone

On a sample from the whole-sentence model I heard “a machine whirl”, strongest at the word “expeditiously”, where the voice had “pure vibrations that don’t make sense”. A splice test located it by ear: one frequency band of the model’s audio was inserted into Kokoro’s audio and the splice compared blind with Kokoro. With the band from 1 to 4 kHz I preferred Kokoro on 6 of 6 pairs, while with the band below 1 kHz or above 4 kHz I heard the pair as the same on 4 of 6 and 5 of 6.

At that vowel Kokoro is voiced below 1 kHz and breathy above it. Periodicity in the 1 to 4 kHz band, measured as the autocorrelation at the pitch period (1 for a perfect buzz), reads 0.33 to 0.45 for Kokoro over three frames and 0.74 to 0.82 for the model. Kokoro’s breathiness there is the same for every noise seed, and feeding the renderer Kokoro’s own features does not remove the buzz, so the renderer is producing harmonics where its target has noise.

Two measures built for the defect then failed as training targets:

Extending the harmonic-to-noise balance loss below 1 kHz recovered about a third of a deficit measured there at no cost elsewhere, so it is in the released model, but tripling its weight moved the measure further and I heard no difference. A loss on the periodicity itself halved the count of too-periodic frames, from 9.6 to 4.7 events per 10 s. On the blind page that followed I preferred Kokoro to that model on 7 of 7 pairs, heard it as the same as the released model on 5 of 6, and ticked the whirl on 3 of its clips against 1. My fitted judge had predicted no audible change for it and was right, which makes it more useful as a filter for bad ideas than either hand-built measure.

My reading is that the pulse branch and the noise branch are only ever trained through their sum, and on the minority of frames where Kokoro is irregular the deterministic pulse branch is the easier one to fit. Supervising the two branches separately, frame by frame, is the experiment that follows from that, and I have not run it.

Where it stands. The released model is the one from before those two attempts. On five clips chosen because they show the defect, over two pages, I heard it as the same as Kokoro on 5 of 11 pairs, preferred it once and preferred Kokoro 5 times; on sentences drawn at random the last result is the 9 of 12 above, from three training stages earlier. UTMOS is 4.33 against Kokoro’s 4.38. All of the listening is mine alone, the model covers only Kokoro’s 28 English voices, and Kokoro’s sound is its ceiling, since Kokoro was the target. Its text side is still Kokoro’s, which is now 94% of the parameters and 69% of the CPU time, so the next gain in speed has to come from there. The model page has the weights, an inference file that needs only PyTorch, sample pairs against Kokoro, and the paper with the method and the negative results in full.