A Tiny Speech Judge That Agrees With My Ears

I am building a very small text-to-speech model, whose decoder (the part that makes the sound) has about 1.75 million parameters, and I want it to sound as good as Kokoro-82M and eventually better. Improving it means trying many variants and keeping the ones that sound better. The bottleneck is the word sound. Every candidate has to be listened to, the next training run waits on each listening round, and the automatic metrics I tried kept disagreeing with my ears.

So I built a judge whose only job is to predict what I hear. It is a fixed model of the ear with no learned weights, plus a head of a few hundred numbers fitted on my own blind listening verdicts and on public data. On listening pages it never saw, it predicts my verdict (A, B or “same”) 79% of the time. When I pick the better of two of my own models, it agrees 66% of the time. The published judges I compared it with, CDPAM and UTMOS included, do worse on both.

They are not worse at everything. On general speech-quality benchmarks UTMOS still wins clearly, and I show those numbers too. This post is about what it took to get a judge that tracks one careful listener on the subtle differences that matter while hill-climbing a model, and what I learned about hearing along the way.

Why the existing measures were not enough

The usual tools each failed in a specific way:

• Spectrogram distance (mel L1) ranked two of my earlier models in the opposite order to my ears. The one with the better number had a coarse crackle.
• A distance on WavLM features, a large self-supervised speech model, could not hear buzz. It rated a buzzy model as the closest to the teacher when I heard it as clearly worse.
• A learned “quality” classifier preferred a compressed model to its own teacher, which I could not hear, and in another case ranked them in the wrong direction. I described that in the previous post.

The most instructive failures were my own. Each time I heard something wrong I wrote a detector for it: short dropouts in the low harmonics, too little noise between harmonics, a 1 dB dip in the presence band, too little pulsing of high frequencies within each glottal cycle, too little cycle-to-cycle irregularity. Every one of them measured a real difference. Where I trained a model to fix one, the next blind test said “same”. A detector can be correct and still not be what the ear uses. That is the gap a judge has to close, and the only way to close it is to fit and check the judge against actual verdicts.

The data: blind verdicts, plus a few things the ear cannot hear

Every listening page follows the same protocol. Two clips of the same sentence, randomly assigned to A and B, loudness-matched so level cannot decide; I answer A, B or “same”; a few pairs are hidden repeats to measure my consistency. After some lessons I added two rules: every clip is at least five seconds long, because two seconds is too short to judge, and no whispering voice, because whispering hides the very artifacts being compared.

Eight pages gave 272 verdicts: 176 decided, 96 “same”, with repeat consistency between 75% and 100% per page.

Some pages were not model comparisons at all but controlled probes: Kokoro’s own audio with one thing changed, to find out what hearing actually responds to. The answers shaped the judge:

Above 4 kHz the ear follows only the envelope. Envelope errors of a decibel go unnoticed. But between 1 and 4 kHz, where several harmonics share one auditory filter, the fine timing matters, and any regular phase rule sounds artificial while a natural, irregular one does not. Splicing one part of my own model’s output into Kokoro’s localised its defect the same way: its 0-1 kHz part was transparent, everything above 1 kHz was not.

Part 1: a judge that predicts my verdicts

A fixed model of the ear

The front end has no trained parameters, so it cannot overfit 272 verdicts. Audio passes through 48 gammatone filters spaced like the cochlea’s critical bands, from 80 Hz to 11 kHz. From each channel I keep only the envelope, sampled at 2 kHz, because that is what the probes say matters at high frequencies. In 32 ms windows every 8 ms, each channel yields three statistics that do not depend on the exact noise realisation:

• level, the window’s envelope power;
• pulse, the share of envelope fluctuation at 50-500 Hz, which is how strongly the band pulses with each glottal cycle;
• crest, the envelope’s peak-to-RMS ratio, which is how peaky the band is within the window.

On one sentence with settled verdicts, the 1-4 kHz pulse statistic already separated them: every pair I called “same” differed by at most 0.03, the borderline 3.5 dB probe by 0.06-0.08, and every pair I could hear by 0.11-0.16.

“Same” means equally good, not identical

My first head predicted “different” from the physical difference between the two clips. Validated by holding out one page at a time and predicting it from the other seven, it scored 71% against 65% for always answering “different”. It failed badly on pages where I compared two of my own models, which I usually called “same” although they differ physically. When I say “same” to two flawed models I mean they are equally good.

The second head models that directly. Each clip gets a quality deficit Q: a weighted sum of the ear model’s differences between that clip and Kokoro’s version of the same sentence, with non-negative weights, so Kokoro’s own deficit is zero. A pair is then a comparison with ties: I prefer A with probability sigmoid(Q_B - Q_A - τ), prefer B with probability sigmoid(Q_A - Q_B - τ), and say “same” otherwise. The threshold τ is learned with the weights; 31 parameters in total.

Held out page by page, it predicts 79% of my verdicts. A simple rule that captures most of the structure (“Kokoro always wins; two models always tie”) gets 68%. On pairs that include Kokoro it is right 87% of the time. It puts almost all its weight on one cue, the peakiness of the 1-4 kHz bands relative to Kokoro, which is the region my splice tests pointed to.

Against the published judges

The same framework, with each judge’s own score in place of Q, held out page by page:

Chance on the second column is 50%; it covers the 44 trials where I chose between two of my own models. CDPAM and WavLM are scored on the 219 trials for which their pair distances were available. To be fair to UTMOS: when one clip is Kokoro and the other is one of my models, UTMOS picks Kokoro in all 132 of the trials where I did, and DNSMOS in 94%. Spotting that my current models are worse than Kokoro is easy for a good judge. Predicting when I will hear no difference, and ranking my own candidates against each other, is where this judge earns its place.

Part 2: do not make the teacher the ceiling

The paired judge has a built-in limit: Kokoro’s deficit is zero by construction, so it can never say that something sounds better than Kokoro. My goal is to beat Kokoro, so the quality target has to be anchored to real human speech, with Kokoro scored like any other system.

Attempt 1: human versus synthetic, and a judge that learned the corpus

The obvious design is a classifier that separates human recordings from synthetic speech. I trained one on 600 recordings from LibriTTS-R against 2,700 clips from many TTS systems. It separated held-out LibriTTS-R from each held-out TTS source with an AUC of 0.91-0.95, which looked excellent.

It was not. Shown natural recordings from a different corpus (the LJ Speech recordings in SOMOS ), it could barely tell them from synthetic speech: AUC 0.59. On my verdicts between models it was right only 21% of the time. It rated the pulse-locked-noise model I had rejected in 14 trials out of 14 as very human, and the top-band hybrid I could not tell from Kokoro as the least human clip of all. It had learned what LibriTTS-R recordings sound like (their microphones, rooms and restoration processing), not what makes speech human.

Attempt 2: pairs that differ only in how they were synthesised

The fix comes from synthetic-voice detection research: take each real recording and resynthesise it, so the human clip and the synthetic one share speaker, words, prosody, microphone and room. The only remaining cue is synthesis itself. From 655 LibriTTS-R recordings (36 speakers) I made seven synthetic versions each: WORLD analysis-synthesis, Griffin-Lim, the Vocos neural vocoder, EnCodec at 1.5 and 6 kbps, and two versions of my own renderer’s approach (the recording’s exact magnitudes rebuilt on a pulse train and noise, with and without envelope errors).

I also added three copies changed only in ways the probes showed I cannot hear (random phase above 4 kHz, ±1 dB envelope noise, mild level and tilt changes) and labelled them human. That makes the judge blind exactly where my ear is blind, which no general-purpose judge is.

It worked. On new speakers it separated originals from each synthesis method with AUCs of 0.80-0.99. On LJ Speech, a recording set it never saw, it told the real recordings from BigVGAN copies of the same recordings with an AUC of 0.85, and SOMOS natural speech from SOMOS’s TTS systems with 0.91, up from 0.59. With no reference at all, it agreed with 86% of my decided verdicts against Kokoro.

Two weaknesses remained. It could not rank my models against each other (48%, chance), and it rated my buzzy models as especially human.

Attempt 3: an artifact zoo, and why a straight line cannot work

The buzz problem had a clear cause. Every synthesis method in training adds noise or roughness, so the judge learned “clean harmonics mean human”. An over-periodic, buzzy voice looked pristine to it. So I generated an artifact zoo from the same recordings: buzz (WORLD with the aperiodicity set to zero), breathiness, whistle bursts, hiss, frame-level phase crackle, pulse-locked noise, MP3 at 32 kbps, and flattened dynamics.

The results traded off rather than improved. Unseen Avocodo copies became easier to detect (AUC 0.70 to 0.79), but agreement with my verdicts against Kokoro dropped from 86% to 75%. The figure shows why, using two of the ear statistics on real recordings and on buzzy and breathy versions of the same recordings:

Human speech is not one end of a scale; it is a region. Artifacts leave it in opposite directions: hiss and breathiness add noise, buzz removes it. A linear score can say “more noise is worse” or “less noise is worse”, never both. Every fix for one direction had broken the other.

The fix: distance from real speech

So the score became a distance. I fitted a Gaussian to real speech alone (a mean and a shrunk covariance over the ear statistics) and scored each clip by how far it falls from that region, measured in units of how much real speech naturally varies in each direction. This model never sees a synthetic example.

With all 38 statistics it already ranked my models correctly 73% of the time. Dropping the statistics that mostly encode the recording chain (the band above 8 kHz and the overall spectral balance) helped further. I chose the final feature set by its score on the external SOMOS test, not on my own verdicts, to avoid tuning the judge to the answers it is graded on. That set is 16 numbers per clip: the mean and spread of pulse and crest in four bands below 8 kHz. It agrees with 95% of my decided verdicts against Kokoro and picks the better of two of my models 71% of the time.

The distance model has one blind spot: LJ Speech as a whole sits far from LibriTTS-R, so it cannot tell LJ Speech from its own vocoded copies. The features that catch subtle vocoders within one recording chain are the same ones that encode the chain. The frozen judge therefore blends the two: 80% distance from real speech, 20% of the matched-pair classifier. The blend agrees with 98% (129 of 132) of my decided verdicts against Kokoro and 66% of my choices between models, and separates SOMOS natural speech from TTS with an AUC of 0.91. It needs no reference, it is anchored to human recordings, and Kokoro is an ordinary system to it, so it can in principle rank something above Kokoro.

How it compares outside my ears

On general benchmarks, which test things this judge was deliberately not built for, UTMOS is stronger:

Two caveats explain most of the gap. SOMOS and SpeechJudge differ mostly in prosody and pronunciation, which my judge ignores on purpose, because every candidate I compare shares Kokoro’s prosody. And UTMOS is a fine-tuned wav2vec 2.0 model whose ONNX export is 411 MB, while this judge is a fixed filter bank and a few hundred numbers. It is a specialist: built for the question “which of these sounds more human to this listener”, with every design choice checked against blind verdicts.

What it says about my models today

On a scale where Kokoro is 0 and higher is more human, Kokoro decoded with a different random seed scores 0.00, as it should. My models sit between -1.7 and -3.9. The older 8.7M-parameter model, which I described at the time as having the “richest” sound, scores best at -1.7. The buzzy one I rejected scores -3.6. And the judge sees steady progress across my recent 1.75M-parameter runs, from -3.9 to -2.2: changes too small for me to hear in any single blind test, but consistent. That is exactly what hill-climbing needs: a direction to move in, between the listening rounds that confirm it.

Lessons for building a judge

• Start from verdicts, not from what seems important. Six detectors I designed from listening notes all measured real differences, and none of them was what my ear used.
• Test what hearing ignores before you build anything. A handful of controlled probes (random phase above 4 kHz, 1 dB envelope errors) told me more about the right features than any amount of modelling.
• Model “same” honestly. Between two flawed systems, “same” usually means “equally good”. A judge that predicts only physical difference cannot represent that.
• Hold out whole pages, not trials. Each page tests a contrast seen nowhere else, which is exactly the situation a judge meets during real hill-climbing.
• Suspect a high AUC on a single corpus. A classifier will learn microphones and rooms before it learns synthesis. Domain-matched pairs remove that shortcut.
• Quality is a distance, not a direction. Artifacts leave natural speech on every side.
• Never let the teacher define the ceiling if you hope to beat it.

What it does not replace is the listener. Every gain it predicts still goes to a blind page before I believe it, and every page I judge becomes more training data. The point is to spend my listening on the few candidates that have actually moved.