Compressing Kokoro-82M: Most of the Decoder Can Go, Its Last Layer Cannot

Kokoro-82M is a small, good-sounding English text-to-speech model with 28 voices. Two thirds of its 81.8 million parameters sit in one place: the decoder that turns an internal description of speech into a waveform. I wanted a smaller, faster decoder that a listener could not tell apart from the original, while keeping everything else about Kokoro (pronunciation, timing, intonation, the voices) exactly as it is.

The result is four drop-in decoders, published at huggingface.co/drmhse/compressed-kokoro , with a side-by-side listening page and a paper with the full method and negative results:

Speed is the decoder’s real-time factor relative to Kokoro’s own decoder on the same input. The whistle rate is explained below; lower is better.

The interesting part is not the table but what it took to get there. Most of the decoder compressed easily. One layer did not, and finding out which layer, and why, took most of the work.

How Kokoro turns phonemes into sound

Kokoro has two halves.

The text side (28.5M parameters) reads phonemes and predicts everything about how the sentence is said: how long each sound lasts, the pitch contour (F0), and the loudness contour. Its output is a sequence of 512-number feature vectors at 40 frames per second, plus pitch and energy at 80 frames per second.

The decoder (53.3M parameters, 65% of the model) turns those features into 24,000 audio samples per second. It has two parts:

1. A stack of style-conditioned residual blocks (33.6M) that refine the features. They use adaptive instance normalisation: each block normalises its activations over time and then rescales them according to the voice.
2. A waveform generator (19.7M), an iSTFTNet. It upsamples the features with two transposed convolutions (10x, then 6x) and finishes with a tiny inverse short-time Fourier transform: 20-sample windows, a new frame every 5 samples. At that point it runs at 4,800 frames per second, which makes its last stage by far the most expensive part of the model to run.

The generator does not invent the waveform from nothing. It is driven by an excitation: a few sine waves at the predicted pitch and its harmonics, plus noise, the way a source-filter model of the voice works. The network shapes that raw buzz into speech.

Every compressed variant keeps the text side untouched and replaces only the decoder. The names give the widths: H is the residual-block width (Kokoro uses 1024), G the generator input width (Kokoro uses 512), and S the generator’s per-stage widths when they differ from G/2, G/4.

The method

Start from Kokoro’s own strongest channels

A pruned student starts with Kokoro’s weights, not random ones. Each channel is scored by the summed, normalised size of every weight that reads from it or writes to it, and the same channels are kept across every tensor that touches them. As a check, pruning to full width reproduces every block of the original to within about 1e-5. A student therefore begins as a damaged copy of Kokoro rather than as noise, and recovers quickly.

Share the excitation, and the phase becomes learnable

This is the trick the rest of the project rests on.

Kokoro’s excitation is random: its sine waves start at random phases and the noise is fresh on every call. Normally that makes the exact waveform unpredictable, which is why vocoders are usually trained with spectrogram losses and an adversarial discriminator rather than by matching samples.

But in distillation I control both models. If the teacher and the student receive the same excitation draw, the teacher’s waveform becomes a deterministic function of the student’s inputs. The student can then be trained to match the waveform’s phase directly, with an ordinary complex-spectrogram loss.

I ran the comparison with equal budgets: a 5,000-step reconstruction-only baseline, then 5,000 more steps either without the phase term (ft_recon) or with it (ft_phase), on 100 validation sentences:

The phase locked on within 1,000 steps. By ear, the reconstruction-only model had a coarse, granular crackle, “bigger grains, cracking on the microphone” in my notes; with the phase term that crackle was gone and what remained was finer, “like a dirty microphone”. I preferred ft_phase “by far”. No GAN was needed for this.

Note the first row: ft_recon has the better mel distance and the worse sound. Mel distance, the most common way to compare vocoder outputs, pointed the wrong way here.

Run the teacher on exactly what the student sees

Adaptive instance normalisation normalises over the whole sequence it is given. Running Kokoro on a full sentence and cropping a window out of the result gives different targets from running it on that window alone. Training used 1.6-second windows, so the teacher was run on each window by itself, matching what the student sees.

Data and throughput

The training text was 12,000 sentences from 18 public-domain books on Project Gutenberg, phonemised with Kokoro’s own G2P (misaki) and spread across all 28 voices, 10% of them at a random speed between 0.8x and 1.2x. That is 21.5 hours of Kokoro’s decoder inputs, cut into 36,416 windows. Caching the teacher’s outputs, compiling the student with torch.compile and prefetching batches in the background brought training to about 39 seconds of audio per second on an Apple M4 GPU, 4.2 times faster than running the teacher online.

What the measurements taught me

Spectral losses can look healthy on scrambled input

The first training run looked fine: mel distance fell from 3.99 to 0.317 in 3,000 steps. It was wrong from the start. The pitch and energy arrays were stored interleaved, and the loader sliced the wrong axis, so both teacher and student were fed scrambled prosody. Both produced speech-like audio from it, the student matched the teacher, and every paired metric was happy.

Whisper caught it: the teacher’s word error rate on that input was 56%. Paired metrics only tell you that two models agree; an absolute check such as speech recognition on the teacher’s own output tells you whether either of them is right.

The whistle, and three explanations that were wrong

From the first listening session there was a “shrill whistle” on some voiced sounds. To track it I wrote a paired measure, the voiced whistle rate: the share of voiced frames in which a narrow band at 2-3 kHz is more than 15 dB louder than in Kokoro’s output for the same frame. At 5,000 steps it was 17.7% of voiced frames, against 5.3% for unvoiced speech and 0% for silence.

I tested three plausible causes, and the data rejected all three:

1. One misbehaving output bin. The generator’s inverse STFT has bins every 1.2 kHz, so 2.4 kHz is a single bin. But that bin had the fewest frames with a 15 dB excess (0.21%); the bin at 3.6 kHz had the most (6.0%).
2. Energy hidden by overlap-add. The generator’s internal spectrograms could carry energy that cancels when frames overlap. Whistle frames carried less hidden excess than ordinary frames (+3.8 to +5.2 dB against +7.4 to +7.5 dB).
3. Imaging around 2.4 kHz, the Nyquist frequency of the last stage, which would place bursts at 4,800 Hz minus a multiple of the pitch. Across 11,674 burst components, their mean distance from those frequencies was 0.250 of the harmonic spacing, against 0.251 for random positions (0.25 is chance).

What the data did show was that in whistle frames the student’s 2-3 kHz band was flatter than Kokoro’s (spectral flatness 1.43 times Kokoro’s, against 1.18 in other voiced frames): energy leaking into the gaps between harmonics.

The width of one layer decides it

So I changed the architecture instead. The first diagnostic kept the generator at Kokoro’s full width and pruned only the residual blocks. After 1,000 steps its whistle rate was 6.1%; after 5,000 it was 3.8%, and I wrote down “amazing even at 3k”. A blind test later found it indistinguishable from Kokoro.

Then I widened only the generator’s final stage, the one at 4,800 frames per second, leaving the rest narrow:

Most of the improvement arrives by 96 channels. And extra training does not move it: training the 128-channel variant from 5,000 to 15,000 steps improved its closeness to Kokoro (test log-mel 0.245 to 0.222) while its whistle rate stayed at 5.4%. The whistle is set by the capacity of the last, highest-time-resolution layer, not by training length. Everything before that layer compresses well: the residual blocks shrink about 3.5 times in parameters without audible loss.

It cannot be cut out with weight surgery

If the whistle lived in a few channels of the final stage, zeroing them would be a training-free fix. On 6,020 voiced frames, each of the 64 channels’ 2.0-2.4 kHz energy predicted whistle frames only weakly (median correlation 0.24, best 0.38). A classifier using all 64 channels reached an AUC of 0.874, and the best four channels only 0.806. The artifact is spread across the layer, so there is nothing local to remove.

A learned judge can be confidently wrong

To rank checkpoints without listening every time, I trained a judge: a logistic regression on WavLM features, fitted to separate clean speech from deliberately damaged copies. It tracked my impressions across training checkpoints. On the test set it then preferred H512_G512 to Kokoro itself (+0.029, 95% interval +0.015 to +0.046), which I heard as parity. Worse, it preferred the 128-channel variant to Kokoro by more (+0.064, winning 88% of clips), while in the blind test I preferred Kokoro in 7 of 12 pairs and the student in none.

A judge that resolves differences below hearing, or ranks in the wrong direction, is easy to build and hard to notice. Claims about how something sounds need listeners. This lesson led to the next post, where I built a judge that is checked against my own blind verdicts.

Kokoro on an Apple GPU is not Kokoro on a CPU

This one matters to anyone distilling Kokoro on a Mac. With identical inputs and excitation, Kokoro’s output on PyTorch’s MPS backend differs from its CPU output by a mel distance of 0.234, with 1.3-1.5 dB more energy between 4 and 12 kHz (+5.1 dB at the 95th percentile). Two CPU runs with different random seeds differ by only 0.057.

The cause is torch.angle inside the generator’s input STFT. For the DC and Nyquist bins the imaginary part is exactly +0 on CPU but -0 or a tiny negative value on MPS, so their phase flips between +π and -π in 24% and 8% of frames. The network turns those flipped phase channels into extra high-frequency noise. Forcing those two imaginary parts to +0 before taking the angle makes MPS match CPU to a mel distance of 0.0001. My training cache had been decoded on MPS, so these students learned a slightly hissier Kokoro; decode the teacher on CPU, or patch it, if you repeat this.

Results

The test set is 112 sentences from three books not used in training, four per voice for all 28 voices. Each decoder is compared with Kokoro’s own decoder on identical inputs. The published weights are the 15,000-step checkpoints where trained that long:

No variant loses intelligibility; the word-error differences (57 to 62 errors) are within the noise of a test this size.

The blind listening tests used loudness-matched pairs in random order, one sentence per voice, with hidden catch pairs of Kokoro against itself:

The catch pairs were answered “same” every time. Every preference I did express came down to the same cue: a slight, almost inaudible whistle in one clip of the pair.

Where the trade-off lives

The residual blocks decide size but not speed: halving their width again (512 to 256) at full generator width gave the same 1.03x speed and measurably worse quality (validation log-mel 0.233 against 0.177). The generator’s last stage decides both speed and the whistle. That is the tension in one sentence: the cheapest place to save compute is the one place the model cannot spare capacity. H512_G256_S128-96 sits at the knee of that curve.

Using it

pip install kokoro==0.9.4 soundfile huggingface_hub
hf download drmhse/compressed-kokoro --local-dir compressed-kokoro --exclude "samples/*"
cd compressed-kokoro

import numpy as np, soundfile as sf
from kokoro import KPipeline
from slim import load

model = load('H512_G512')   # or 'H512_G256_S128-96', 'H512_G256_S128-128', 'H512_G256'
pipe = KPipeline(lang_code='a', model=model, repo_id='hexgrad/Kokoro-82M')
audio = np.concatenate([a.numpy() for _, _, a in pipe('Hello from a smaller Kokoro.', voice='af_heart')])
sf.write('hello.wav', audio, 24000)

slim.load builds Kokoro’s model from the official checkpoint and swaps in the smaller decoder, so the rest of the kokoro package (voices, KPipeline, speed control) works unchanged.

What this points to

Pruning and distillation got Kokoro’s decoder to 55-73% of the model size with intelligibility intact and, at the largest variant, no audible difference. But the speed ceiling is set by the generator’s audio-rate final stage, and that stage is exactly where quality lives. Going much faster means not running a neural network at 4,800 frames per second at all. That is the problem I am working on now: a decoder that does all of its learned work at a few hundred frames per second and leaves the audio-rate part to fixed signal processing. Judging that kind of model fairly is what the next post is about.