How I Made CosyVoice3 2.2× Faster on Apple Silicon

I have been using a local CosyVoice3 service to narrate audiobooks—a few thousand clips of production output, one segment at a time, over a couple of weeks. It worked. It was also slower than real time, which I had accepted as the price of running a 0.5B TTS model on a laptop.

Then I looked at Activity Monitor and found python3.10 sitting at 38.87 GB on a machine with 32 GB of unified memory.

This is the story of what that turned out to be, and how the fix cleared the way to make the same service 2.2× faster, ending at RTF 0.757: comfortably faster than real time. Along the way I had to make a decision I had been avoiding: whether to move off PyTorch onto Apple’s MLX, and if so, how much.

All measurements in this article come from an M2 Max with 32 GB of unified memory, running CosyVoice3-0.5B for long-form narration. Your ratios will differ; the reasoning should not.

Code and reproduction: The complete service is public at drmhse/tts-funaudio , including the MLX LLM port, disposable-worker supervisor, memory soak harness, and tests behind the measurements. Model weights are downloaded separately and are not committed to the repository.

The measurement was lying

The first thing I did was check my own telemetry, because the service already recorded memory on every request. It reported 4.2 GB.

Activity Monitor said 38.87. My telemetry said 4.2. One of them was wrong, and until I knew which, I had nothing.

The telemetry was wrong for a reason worth internalizing. It used resource.getrusage().ru_maxrss: peak resident set size. On macOS, resident size did not answer the question I needed to ask. Under memory pressure, the kernel can compress pages or move them out of physical memory, reducing what RSS sees while the process continues to impose a large memory cost.

The more useful process-level measure for this investigation was phys_footprint , exposed through task_info(TASK_VM_INFO). There is no direct Python API for it, so I used a small ctypes shim:

# mac_memory.py — reads the task's physical footprint
class _TaskVMInfo(ctypes.Structure):
    _fields_ = [..., ("phys_footprint", ctypes.c_uint64), ...]

def self_footprint_bytes() -> int:
    ...  # task_info(mach_task_self(), TASK_VM_INFO, ...)

I validated it byte-for-byte against /usr/bin/footprint before trusting it. With that in place, the picture resolved instantly: 38.9 GB of footprint, 40.9 GB of system swap, and 2.3 GB resident. The growth was real and had been invisible to my telemetry for weeks.

Lesson one: on macOS, verify that your memory metric measures the pressure you actually care about.

You cannot free it in place

With a real measurement I could attribute the growth. vmmap -summary pointed at the Metal driver, and the shape of the growth—big jumps on new inputs, nothing on repeats—narrowed the culprit.

In the PyTorch build I was using, the MPS backend compiled graphs for distinct combinations of operations, input shapes, and dtypes, then retained them. New sentence lengths introduced new shapes and footprint kept growing. I found no supported eviction path for that compiled-graph state.

The reflex is torch.mps.empty_cache() . I was already calling it at every request boundary. PyTorch documents it as releasing unoccupied memory held by the caching allocator. It did not reclaim the compiled-graph growth I was measuring.

I looked for an in-process control and did not find one in this stack. Ending the process was the only reliable reclamation mechanism I could demonstrate.

That reframes the problem usefully. If the only reliable way to free the memory is to exit, the thing holding the memory must be something you are willing to exit.

Make the model disposable

So the service split in two.

The API process—HTTP, the job store, SQLite state, and artifact writing—no longer imports torch at all. It sits flat at 49 MB. The model lives in a child process that the parent supervises and can kill and respawn at any time. Requests cross the boundary as small messages; audio comes back as bytes.

The worker is recycled on a memory budget. Getting that budget right took two tries, and the first failure is the interesting part.

My first attempt used a simple absolute ceiling: 9216 MB, kill above that. It was a disaster—three recycles in twelve segments, and RTF collapsed from 1.58 to 2.40 because every recycle paid both the model-load cost and the full shape-compilation cost again.

The mistake was treating all growth as equal. It was not:

• Segment 1 legitimately grew the process by 2642 MB. That was the shape compilation for the first real workload.
• Segment 2 grew it by 57 MB.

An absolute ceiling charged the unavoidable one-off cost against the budget, leaving almost no headroom for real work. The fix was to measure growth from a warm baseline latched after the first completed request:

def note_request_complete(self, *, sample_baseline: bool = True) -> None:
    self.requests_served += 1
    if sample_baseline and self.baseline_footprint is None:
        self.baseline_footprint = self.footprint_bytes()

That sample_baseline flag is not decoration. A failed request must not latch the baseline: it may have died halfway through allocation, and a stale low baseline makes the worker recycle constantly from then on.

The budget now allows 2048 MB of growth above warm, which works out to roughly 25–60 segments per worker in this workload. Recycling is quantified rather than hoped about: forcing a recycle every two requests costs RTF 1.58 → 2.22, so the budget is an explicit trade of throughput for a memory ceiling, tunable through one environment variable.

Result: system swap went from 40.9 GB to about 2.7 GB. The API process is flat. The worker settles around 7.5 GB and is replaced before it becomes a problem. A forced recycle mid-job produced byte-identical audio for every segment, so the process boundary is invisible in the output.

Lesson two: when a resource can only be reclaimed by process exit, make the process that holds it disposable, and put nothing else in it.

Now, speed

With memory bounded I could look at the clock. Baseline was RTF 1.65: 1.65 seconds of compute per second of audio. RTF is wall time divided by audio duration; lower is faster, and 1.0 is real time.

Profiling by stage produced this breakdown as a fraction of audio duration:

The LLM dominated, so that is where I started.

The fix that was free

I read the sampler. nucleus_sampling had this shape:

while cum_prob < top_p:        # cum_prob is a tensor on the GPU
    cum_prob += sorted_value[i]
    i += 1

Comparing a device tensor to a Python scalar forces a GPU-to-CPU synchronization. Inside a loop. Once per candidate. ras_sampling also did a device-side comparison to count repeats. At about 25 tokens per second of audio, every token was blocking the pipeline several times over for no reason.

The fix was to sort once, move the small top-k head to the host, and do the accumulation in plain Python using the tensor’s dtype. The arithmetic is unchanged, so the output should be identical—but that claim needed a test. I wrote an equivalence harness that runs the verbatim original implementation beside the new one over 83 cases, including the boundary conditions where a naive rewrite silently diverges: the first probability already exceeding top_p, exact ties, and top_k larger than the nucleus.

RTF 1.65 → 1.58, output byte-identical. No accuracy trade and no new dependency; the loop simply stopped stalling the GPU.

Lesson three: before you parallelize anything, check whether you are simply waiting.

Pipelining, and the argument that makes it safe

Multi-segment jobs process segments in sequence, and the three stages use different hardware: the LLM and flow run on the GPU, while the vocoder runs on the CPU because its float64 pitch predictor has no MPS implementation. The LLM stage of segment n+1 can therefore overlap the flow-plus-vocoder work of segment n.

The risk is reproducibility. Every clip must be regenerable from its recorded seed, which means the order in which random numbers are consumed cannot change. Overlapping stages would normally scramble exactly that.

It is safe here for a specific, checkable reason: the flow decoder and vocoder draw no randomness at inference time. Their noise is precomputed at construction in CausalConditionalCFM.rand_noise, SineGen2.sine_waves, and SourceModuleHnNSF.uv. The LLM’s multinomial is the only consumer of a torch generator in the path. As long as stage one seeds and decodes strictly in order, its RNG sequence is identical no matter what stage two is doing.

fp16, and two things that did not work

Autoregressive decode is bandwidth-sensitive: each token reads every weight. Halving the weight width was the largest single lever available, and the dtype boundary was unusually clean because CosyVoice3’s LLM consumes its own embeddings and emits integer speech tokens. Nothing downstream sees a float16 tensor. RTF 1.52 → 1.31, and the WhisperX forced-alignment quality gate scored identically.

Two attempts failed and are worth recording:

• fp16 for the flow decoder crashed inside MPS: 'mps.add' op requires the same element type. Its cached float32 conditioning tensors met half-precision weights. The flow stayed float32.
• bfloat16 was slower than fp16, used more memory, and failed the alignment gate on one segment with a score of 0.8889. Wider exponent, fewer mantissa bits, no benefit here.

I also measured and rejected removing torch.mps.empty_cache(): it bought 3.6% RTF and cost 2.4 GB of footprint. On a 32 GB machine, that was not a worthwhile exchange.

I left flow steps, or NFE, at 6 even though dropping to 3 would have reached roughly RTF 1.02 on its own. The alignment gate was unaffected, but that gate measures whether the words are intelligible. Log-spectral L1 was 0.97, showing that the audio had materially changed.

The MLX question

At RTF 1.31 the LLM was still the largest stage, and I was out of easy PyTorch moves. Would Apple’s MLX help here, or was it a different set of tools for a different problem?

The useful question was what kind of bound I was hitting.

I had assumed the decode loop was bandwidth-bound—the usual model for autoregressive inference, and consistent with fp16 helping. But when I compared achieved bandwidth with the M2 Max’s theoretical figure, the LLM stage reached only about 25% of it. The trace instead pointed to dispatch overhead: 24 transformer layers, several kernels per layer, and one dispatch for each, per token, with gaps between them.

That is the regime in which I expected MLX to help. MLX uses lazy evaluation and dynamically constructs its compute graphs; its own documentation notes the fixed cost of graph evaluation. The benchmark, rather than the framework description, had to decide whether that execution model helped this workload.

The measurement used the same weights and the same GPU:

End to end: RTF 1.31 → 0.757. The LLM stage fell from 0.71 to 0.26 of audio duration. The service was now faster than real time on a laptop, with a 0.5B model doing zero-shot voice cloning.

Why the port was tractable—and why it stayed a port

I did not move the service to MLX. I moved one stage. That distinction is the engineering content of the decision.

The port was feasible because of the seam mentioned earlier: the LLM emits integer speech-token IDs. The two frameworks never exchange a tensor. There is no dtype negotiation, memory-layout conversion, autograd interop, or shared allocator. MLX produces a list of integers; torch’s flow decoder consumes it. Either side could be replaced independently.

That left a small, bounded amount to reimplement. mlx-lm already supports Qwen2, the LLM’s backbone, so only the non-Qwen2 pieces needed writing: the speech-token embedding, output projection, decode loop, and RAS sampler. About 320 lines.

Two traps are worth flagging:

1. The weights are not where the config is. The model directory contains a CosyVoice-BlankEN subdirectory with a config.json; that is the pretrained initialization, not the fine-tuned model. The real weights are in llm.pt under an llm.model. prefix. Load the architecture from one and the weights from the other. Getting this wrong produces a model that runs, sounds plausible, and is wrong.
2. Two frameworks means two copies of the model. After MLX loads, the torch LLM is dead weight still holding GPU memory. It must be explicitly released; in this implementation I did that by pointing each parameter’s storage at an empty tensor.

The flow decoder and vocoder stayed in PyTorch, and I would keep them there. They are a U-Net/DiT with weight_norm, convolution stacks, and a float64 pitch predictor that already depends on CPU fallback. Porting them would be weeks of work against operation coverage to optimize stages that were not the bottleneck.

The useful answer to “MLX or PyTorch?” was “both, at the seam where they do not have to talk.”

The part I got wrong

I also implemented batched decoding: several segments through the LLM at once. Batching is the textbook fix for a dispatch-bound loop—same number of dispatches, more work per dispatch. It measured 151 tokens per second versus 94. The code is left-padded and masked, with RoPE’s relative-position behavior making the padding safe. A test asserts that the mask blocks every pad column in both prefill and decode.

It is off by default, and it will stay off.

When I checked whether batching changed the output, it did. My first instinct was a masking bug, so I ran the cleanest possible experiment: batch a sequence with an exact copy of itself, eliminating padding. The prefill logits still moved—3.5e-2 in fp16 and 4.6e-5 in fp32.

That is floating-point reduction order. Adding a batch dimension changes matmul tiling, which changes accumulation order and the final bits. Normally nobody cares. Here it can reorder the top 25 sampling candidates, select a different token, and make the sequences diverge within about ten steps.

The consequence is unacceptable regardless of speed: a segment’s audio would depend on which other segments happened to share its batch. Regenerating one clip from its recorded seed—the property on which the artifact and job design rests—would silently produce different audio. The upside was roughly RTF 0.76 → 0.70.

So batching remains a documented opt-in, with the measurement in the module docstring so nobody re-enables it thinking they found free performance.

Lesson four: “faster” is not a scalar. Reproducibility is a feature, and a change that quietly breaks it is a regression even when the benchmark improves.

Reproducibility came out stronger than before. The torch sampler drew from one global generator, so seed determinism held only because segments decoded strictly in order. MLX gives each request its own PRNG key, so a segment now depends on its seed alone, not global draw order or its position in a job.

Where it ended up

Memory went down while speed went up: the worker settled around 7.5 GB versus 9–10 GB with torch fp16, the API process stayed flat at 49 MB, and system swap fell from 40.9 GB to 2.7 GB.

There are caveats:

• fp16 and MLX both change the audio. Several thousand clips generated before this change cannot be regenerated identically from their recorded seeds. Reverting to torch and fp32 restores them exactly, which is why both remain configurable and every job result records the backend and precision that produced it. A recorded dtype that lies is worse than a slow one.
• Streaming regressed. MLX has no incremental decode path in this implementation, so /tts/stream synthesizes the whole clip and then emits chunks. The bytes and contract are unchanged, but time to first audio now equals full synthesis. One environment variable restores true streaming at the cost of speed.
• mlx-lm must be installed with --no-deps in this environment, or it upgrades transformers past the pinned 4.51.3 and breaks the torch stack. A test catches this because I will otherwise forget.

What is next, and why I am not doing it yet

The flow decoder is now the bottleneck, accounting for about 57% of the remaining wall time. There is an obvious-looking 38% of it lying on the floor: the flow integrates the ODE over prompt_mel + target_mel, then slices the prompt off and discards it. For an 11.76-second reference voice, that is 588 of 1542 mel frames computed and thrown away.

The tempting move is to cache it because the reference never changes. But the non-streaming DiT builds a fully bidirectional attention mask, so prompt frames attend forward into target frames. Their activations are a function of the sentence being spoken. Precomputing them is not merely difficult; it is ill-defined. Dropping them changes the output because that cross-attention is part of the voice-cloning mechanism.

That leaves two real options. Fine-tuning the speaker into the weights would delete the prompt entirely and project to about RTF 0.60, but 11.76 seconds is nowhere near enough data to train a speaker into 0.5B parameters. That is why in-context cloning exists, and the change would reset seed reproducibility again.

The simpler option is a shorter reference clip. The measured cost is approximately linear in reference length, so a four-second reference projects to about RTF 0.63 with no training and no new code.

There is a subtlety even there. The flow’s noise is a fixed precomputed buffer sliced to the total length, so target frames currently receive noise columns 588 onward. Change the reference length and they receive different noise: different audio with identical weights.

Five things I would carry to the next project

1. Verify your instrumentation before trusting a conclusion. RSS hid the memory pressure that mattered to this service. Two weeks of clean-looking telemetry described a machine deep in swap.
2. When a resource can only be reclaimed by exit, make something disposable. The fix for an in-process cache I could not reclaim was a process boundary, not a cleverer free().
3. Check whether you are waiting before optimizing computation. The best speed-per-effort change was removing synchronization from a loop, with zero output change.
4. Ask what kind of bound you face before changing frameworks. MLX was a 2.2× stage-level win because this decode loop was dispatch-bound. A bandwidth-saturated stage would need a different lever.
5. Keep a bit-identity harness and let it veto changes. Some changes shipped proven byte-identical; others were rejected after measured output changes despite running faster.

The MLX-versus-PyTorch question had an answer that generalizes: do not choose at the application boundary. Find the stage that is actually slow, then find a seam where the data crossing it is simple enough that framework identity stops mattering—in this case, integers—and port only that. One 320-line module, two frameworks that never exchange a tensor, and a service that runs faster than real time on a laptop.