Taking the 2B Qwen3-TTS from 0.876 RTF to 0.253 RTF on Apple Silicon M4

I have replaced the model behind my book narrator four times, from Kokoro through Chatterbox and CosyVoice3 to Audio8 and now Qwen3-TTS, and every move bought quality at the cost of speed: Kokoro’s 82 M parameters are quick and cannot clone a voice at all, Chatterbox clones and carries emotion at six times the size, and CosyVoice3 sounded better again behind a diffusion decoder that is two thirds of its total cost.

Then the trade broke, because Qwen3-TTS is both the largest model I have run and the fastest, on the same laptop and harness:

Two to three and a half times the parameters, two to nearly three times faster, and to my ear the best of the five. Kokoro and Chatterbox are absent because I ran them before this harness existed, so their placement in the story is by sound rather than by stopwatch.

Predicting audio codes directly and running no diffusion decoder raises the ceiling, but a ceiling is not a speed: my first working port measured 0.876, slower than CosyVoice3 despite the simpler pipeline. Every bit of the advantage came from one day of profiling on an M4 MacBook Pro with 16 GB of memory, and what is worth writing down is not the speedups but that I predicted their sources wrong nearly every time, always for reasonable-sounding reasons.

The setup

RTF is the unit above: if the program spends 30 seconds producing 60 seconds of speech it runs at half speed, which we write as 0.5, and lower is better. The eleven-minute chapter I test with went from six minutes of computation to under three.

Three networks run in sequence. The talker reads text and emits codes, compact descriptions of the form “this eighty-millisecond slice sounds like this,” at 12.5 slices per second. The depth predictor adds fifteen codes per slice for finer texture, the difference between a flat synthetic voice and one with breath behind it. The codec decoder expands each slice into 1,920 waveform samples. The first two are autoregressive, meaning each slice depends on the one before it, so neither can be parallelised across time; the third can process a whole chapter at once.

Counted from the file headers rather than the model card:

It ships as “1.7B” because that counts only the language backbone. Published names round down; file headers do not.

The stopwatch was lying

A GPU queues an instruction and returns control immediately, which is excellent for throughput and disastrous for naive timing: when I timed a stage at 26 milliseconds, that number covered the stage plus every queued job ahead of it that drained at the same moment. Emptying the queue on both sides of the clock put the stage at 2 ms, since it had merely been holding someone else’s receipt. The trap caught me four times in one day, twice after writing myself a note about it, so when a measurement surprises you, suspect the measurement first.

Batching broke two assumptions

A decode step costs what it costs because it reads the model’s weights, and that read does not get more expensive when more sentences share it, so I run 24 at once. Both surprises below fall out of that one decision.

The number format that got faster by doubling. Weights are re-read every step, so shrinking them is the obvious lever: quantisation stores each parameter in one byte instead of four, and I had it on. But a batch only shares that read if the multiplication reads each weight once, which a dense multiplication does and the compressed path does not. It re-reads the weights for every lane, so 24 lanes cost 24 reads and batching buys nothing.

Doubling the data made it four times cheaper. An optimisation is a property of a technique in a situation, and “smaller is faster” held only until the situation became 24 lanes at once.

Half the work was discarded. Sentences differ in length so short ones finish early, but a GPU runs in lockstep and charges for all 24 lanes every step whether or not they still hold anything, which makes a group run as long as its longest member. A counter put useful work at 47% to 56%.

Sorting the sentences by length before grouping them, so short ones share a group, is nine lines of code. Useful work rose to between 53% and 90%, steps fell from 768 to 605, and the talker went from 0.239 to 0.182, which is the argument for examining how work is organised before examining how fast each piece runs.

I published a post some time ago about a different speech model whose headline conclusion was “scheduling beat kernels.” I had already learned this, written it down, and put it in front of an audience, and I still spent half of this day writing GPU code before checking the schedule.

Three things I built and mostly deleted

A replacement kernel for the one-byte problem above. A kernel is a small program that runs on the GPU, written here in Metal. Mine did what I designed it for, sharing one memory read across all 24 sentences where the library shared none: 5.2x amortisation against the library’s 1.0. It was still slower, because fixing the memory problem exposed an arithmetic one, running at a quarter of the chip’s capability while Apple’s library used matrix hardware I was not touching. Rewriting it to use that hardware did not help either, and across six configurations the best of them lost, because a library that is wrong about your case still embodies enormous work on every other case.

A second kernel rearranged waveform data so convolutions became contiguous matrix multiplications, removing a large intermediate buffer. That one won and survives.

Half-precision for the codec decoder, which is 95% of its stage. It delivered 17% and failed a correctness check by six times: my test permits 0.5% deviation from the original Python implementation and I measured 3%. I had reasoned that 0.1% per rounding across 24 steps, partly cancelling, would land near 0.5%. The arithmetic was right; the pipeline model was not. Each step computes sin(A × x) where A is learned and can be large, so multiplying x by a large constant before the sine turns a small perturbation into a proportionally larger one. Error is not carried forward so much as amplified, twenty-four times over, which is why estimating error growth requires establishing what the error passes through.

Queue-based refilling of finished sentence slots was worth perhaps another 30%. I designed it, started writing, and reverted every line, because partway through I had added two inputs to a kernel without the code that supplies them. Reading an unsupplied input on a GPU raises no error; it reads whatever occupies that address and continues, producing subtly wrong results. Earlier that day I had lost hours to that exact failure mode inside the library, where a sorting routine silently returns zeros above a certain input size. Something that breaks loudly costs a minute, whereas something that breaks quietly costs a day.

Where it ended up

An eleven-minute chapter takes 2 minutes 56 seconds, so a sixteen-hour audiobook costs about five hours of computation instead of twelve.

Sorting a list before a loop was the largest contribution, changing a number format was the one that looked backwards, and rearranging memory was the single hand-written kernel that worked, while everything else, roughly half the day, was deleted. I suspect that distribution is typical and simply goes unpublished.

Every mistake above came from confidence about something I had not measured in the configuration where it ran. Theory produced the right list of things to try and the wrong order every time.