Narrating Book-Length Text on One Machine

You cannot proofread sixteen hours of audio.

Everything below follows from that. Correctness has to be established by machinery, because the one method you would trust — sitting down and listening to all of it — costs two working days per pass and nobody does it twice. A twelve-hour render must also survive being interrupted, and if the audio drives an interface that highlights words as they are spoken, every word needs a timestamp good to a fraction of a second.

Narrating three paragraphs proves nothing about any of this. Here is what actually holds up over a book, and the parts I got wrong on the way.

The synthesis engine is Rust on candle . Everything around it is a few scripts.

The chain

markdown source
  -> narration text + page-word map      (text preparation)
  -> WAV master                          (Rust TTS over HTTP)
  -> WebM/Opus delivery                  (ffmpeg)
  -> alignment manifest                  (recognition + matching)
  -> published audio + timings

Two rules about that chain, both learned the hard way.

Keep the WAV master. Delivery encoding and alignment are both cheap and both derive from it. Synthesis is the only expensive step in the pipeline, so it is the only one that must never repeat.

Align the file you ship, not the master. Encoder delay is then measured instead of assumed, and you find out at alignment time if the encode did something unexpected.

The engine

Six crates, split along the lines where change actually happens:

Both engines beat real time on an M4:

RTF is synthesis time over audio duration. At 0.7, sixteen hours of speech costs about eleven hours of compute, which sets the tempo of every decision that follows. You do not iterate casually at those prices.

CosyVoice runs in three stages:

text -> BPE tokens
     -> LLM (autoregressive)    -> speech tokens, 25 per second
     -> flow matching (DiT)     -> mel frames, 2 per speech token
     -> vocoder (HiFTGenerator) -> waveform at 24 kHz

Those ratios are fixed, and they buy something specific. A segment of n speech tokens is exactly n * 2 mel frames and exactly n * 960 samples. So you can decode several segments in one flow call and cut the waveform back apart afterwards at sample-exact boundaries. No estimation, no drift. Most of the scheduling below depends on that being exact.

Two scheduling decisions

These bought more than any kernel I wrote.

Batch the autoregressive stage across segments, with a cap. A decode step costs what it costs because it reads twenty-four layers of weights and pays a host round-trip for sampling. Neither scales with the sequence being extended, so batching pays both once for many segments.

Then bound it, because each lane carries its own KV cache — about 101 MB here. A sixteen-minute section has 118 segments. One lane per segment asks for 11.9 GB of cache alone, which on a 16 GB machine means swap, and swap shows up in the timings even when your memory metrics cannot see it: the stage went from RTF 0.19 to 0.70, the engine from 0.70 to 1.49. Eight lanes costs 0.81 GB. The throughput curve is flat by four.

Group the flow decoder, and do not group everything. Each flow call re-prepends the reference voice’s prompt — 588 mel frames. Decode segment by segment and you pay that every time; on one passage, 61% of the flow’s work was the same prompt redone.

The obvious conclusion is to fuse the whole utterance. That is wrong, and it was wrong in my engine for a while. DiT attention is quadratic, so per-block cost behaves like a*n + b*n². Fitting measured coefficients and projecting over a seventeen-minute section gives a modelled 3.44× penalty for fusing everything, against 0.67× for grouping at 1600–2400 frames. Fusing looked good only because I measured it on a passage short enough to sit near the optimum.

There is a middle, it is broad, and you have to look for it.

Driving it

As a library:

use tts_core::{EngineConfig, SynthesisRequest, Voice};

let config = EngineConfig::new(tts_engines::default_root("cosyvoice"));
let engine = tts_engines::load("cosyvoice", &config)?;
let voice = Voice::load("voices/cosy-default-cosyvoice")?;

let request = SynthesisRequest::new("Hello from Rust.").with_voice(voice);
engine.validate(&request)?;              // refuses a mismatched voice asset up front

let out = engine.synthesize(&request)?;
tts_core::wav::write("hello.wav", &out.audio)?;

Over HTTP for a book run, so the model loads once instead of sixty-one times:

TTS_API_KEY=… tts-serve --port 3098 \
    --engine cosyvoice --voice voices/cosy-default-cosyvoice

curl -X POST localhost:3098/tts -H "X-API-Key: $TTS_API_KEY" \
     -H 'content-type: application/json' \
     -d '{"text":"…","seed":1234}' -o section.wav

Responses carry x-audio-seconds and x-rtf, which is how you get cost per section without building any instrumentation.

Bound your segments or lose text

Text splits into paragraphs, then into segments of whole sentences under a character budget. Default 220. That number decides prompt length, batch shape, where silence goes, and where the waveform can be cut.

My budget applied only when merging sentences. A single sentence longer than the budget sailed through whole — one reached 566 characters, roughly 38 seconds of speech.

Meanwhile the autoregressive loop masks the end-of-sequence token for only the first 2 * text_tokens steps. After that the model can stop wherever it likes. On long segments it did, mid-clause.

Nothing downstream noticed, and nothing downstream could. The vocoder renders the tokens it receives, so output duration matches the tokens produced rather than the text that should have produced them. The file is the right length, internally consistent, and missing a clause. Twenty-four segments across one book. Roughly two minutes of text present in the source and absent from the audio, in a file that passed every check I had.

So bound every segment. Split an over-long sentence at clause punctuation first, word count only as a fallback — the inserted gap then lands where the voice would have paused anyway.

A guard is worth adding on top, with a known ceiling. Compare each segment’s speech tokens per character against the median for that request, regenerate outliers, fail with the offending text if they persist. Measuring the reference from this voice and this text beats any words-per-minute constant you could pick.

Its ceiling: sentences merge up to the budget, so a segment that loses a 49-character clause out of 220 sits around 0.78 of the median, inside any threshold loose enough to avoid false positives. Token counts cannot see a partial loss inside a merged segment. Recognition can. Do not spend a week tuning that threshold — I spent an afternoon on it before accepting that recognition is the acceptance test and the ratio guard just catches the severe cases cheaply.

The voice reads whatever you leave in

Every rule here exists because its absence produced audible damage in shipped audio.

Strip everything that is not speech. Front matter, code blocks, HTML comments, shortcodes, images. An HTML comment holding art direction got narrated in full.

Handle emphasis exhaustively, then sweep. **bold** and *italic* are easy. ***bold italic*** defeats both: \*\* matches, [^*]+ then fails on the third asterisk, and the italic rule’s negative lookbehind refuses to start there. So the markers survived, and the voice read them — “asterisk, asterisk, asterisk local design” — before collapsing into "asterisksisks". A repeated nonsense token is exactly what sends an autoregressive model into a loop. Delete any surviving asterisk unconditionally; no stray one has a reading that belongs in speech.

Anchor underscore italics to word boundaries. Unanchored, _([^_]+)_ matches across identifiers. Given `agency_account` → `client` → `source_export`, it paired the underscore in the first with the one in the third, removed both, and produced agencyaccount and sourceexport. While you are there: read snake_case as separate words, and speak arrows as “, then”. A dropped arrow leaves a list of nouns with no relationship between them, which is both wrong and the kind of input that triggers the loop above.

Render tables row by row as sentences. Flattened into a paragraph buffer, a table becomes one run-on with no sentence boundaries, separator row included. The model degenerated on one and produced twenty-one seconds of babble that two independent recognisers heard as “tampoligation, tampolition, sambolition”.

| Partner | Provides           | Requires        |
| Product | field synthesis…   | roadmap context…|

-> "Product. Provides: field synthesis… Requires: roadmap context…"

Unwrap brackets and placeholders. Task-list markers reached the voice thirty-four times in one section. [specific customer] and <name> should keep their words and lose the punctuation, which is what a page tokeniser does anyway.

Capitalise the first word of every paragraph. Speech has no case, so case never occurred to me as an input. Six of eight sampled lowercase openers were mispronounced: “complain” as “Dock and plane”, “recurring” as “Career go will lead E”, “reliability” as “Our liability”. And lowercase openers were not exotic — snake_case identifiers had become ordinary spaced words, each at the head of its own list item. Fixing one thing created the conditions for the next thing to break, which is most of what maintenance is.

Space the compounds the voice cannot say. timezone came out as “Heideheb”, signup as “SignGen”. Keep that list short and evidence-based; the audit further down finds them without guessing.

Warn about anything that still looks like markup. Surviving asterisks, pipes, backticks, brackets, headings, blockquotes, tags, shortcodes. A silent converter is how most of the defects above reached a published file.

Alignment comes from the audio

Knowing the exact script suggests a shortcut, and it is a trap.

Skip recognition; cut the script into sentence windows; give each window a slice of the timeline proportional to its character count; let a forced aligner place the known words inside. Roughly five times cheaper than recognising first.

Median error, measured against the audio: 4.8 seconds per word. Words landing within half a second of where they were actually spoken: 11.9%.

The failure is structural, not a tuning problem. A forced aligner places words inside the span you hand it. It cannot reject a bad span and cannot tell you it got one. Character-proportional spans assume a constant speaking rate, which narration violates at every heading, paragraph break and inserted pause, and each boundary error is inherited by every word inside the window.

The statistics I was reporting could not see any of it. alignedShare: 0.994 says words received timestamps. A coverage figure says those timestamps span the file. Both were true. Neither has anything to do with whether a timestamp is where its word is.

What works instead — recognise the audio, use the script only for spelling:

1. Recognise with batched faster-whisper and word_timestamps=True. 20.4× realtime on CPU, 51 s for a seventeen-minute section. Batch it; the unbatched path is 7.8× on the same machine.
2. Match the recognised sequence to the document’s canonical words with difflib.SequenceMatcher. Each canonical word inherits a measured time while the document stays authoritative for spelling, so recognition errors never reach the page.
3. Interpolate the remainder, and mark every interpolated word. A manifest should never imply a measurement nobody made.
4. Cut cues at measured pauses.

Three things will bite you here.

Turn autojunk off. SequenceMatcher treats tokens appearing in more than 1% of a long sequence as junk, which in a 2500-word section quietly discards the, to, of and a — several hundred of the most reliable anchors you have.

Require support for an anchor. A size-1 matching block on and pairs an occurrence in the script with an unrelated one in the transcript. Two of those pinned 62 canonical words into 1.1 seconds of audio. Demand a run of at least three words, then reject any anchor pair implying more than five words per second — narration runs 2.5 to 3 — and drop the weaker member so the region becomes an honest gap instead of a compressed lie.

Do not add a second acoustic pass. I tried wav2vec2 forced alignment inside the recognised windows, reasoning that acoustic onsets must beat recognition timestamps. It placed 1764 of 2524 words against recognition’s 2488, drift p90 of 4.03 s, and subdividing the failures made it worse. It was throwing away timings that were already right.

Cue length is what the listener sees

If the player interpolates the highlight across the active cue rather than sweeping word timings — most do — then cue length bounds the visible error no matter how precise your timings are. A cue must also not span a pause, or the words after the pause highlight during silence.

Measured word times make the pauses visible, so cut on them: target three seconds, break at sentence punctuation or a gap over 220 ms, cap the maximum. Median lands at 2.4–3.0 s, worst case under 6 s.

Then clamp everything forward. A player selecting with findIndex(t >= s.start && t < s.end) returns the earlier cue on an overlap, and the highlight jumps backwards.

Gates

Every section carries a quality block that names what failed, instead of a number that always looks fine.

Add the third one first, whatever else you skip:

ffmpeg -v info -i section.webm \
  -af silencedetect=noise=-40dB:d=0.20 -f null -

Silence detection knows nothing about your script, your windows, or your aligner. It reads the waveform. If your cues were cut where the voice pauses, their boundaries should land inside intervals ffmpeg independently calls silence.

That property is the entire point. A manifest can be perfectly self-consistent and still be shifted against the audio, and every check derived from the aligner will agree with the aligner. This one can disagree.

Then check integrity: transcribe the closing seconds of each delivered file, which proves both that synthesis reached the end and that the deliverable decodes, and transcribe every span the aligner could not measure. Compare what was heard against what should have been said. Low overlap separates an audio defect from a recognition miss — one needs a re-render, the other needs nothing, and guessing which costs hours either way.

The audit you have already paid for

A word the voice mangles is never recognised. So it is already flagged as interpolated, in every occurrence, in manifests you have on disk.

Aggregate them. Ask which word types are unrecognised in at least 80% of their appearances with at least three occurrences. On a 146,000-word corpus that produced about thirty-seven suspects at no compute cost.

Then listen to them, because most are innocent. countermetric, tradeoff, codebase and quickstart are spoken correctly and merely transcribed as two words. thirty is spoken correctly and transcribed as 30. vale and “Vail” are homophones. Six of the loudest signals in that list were orthography differences between recogniser and document.

Acting on the list without listening would have triggered hours of re-rendering to fix nothing. Cheap detectors produce suspects. They stay useful exactly as long as you keep calling them suspects.

One related trap: if your renders are deterministic — mine are, under a fixed seed — re-rendering unchanged text reproduces it byte for byte. I “fixed” two sections twice before noticing the verifier returned identical scores at identical timestamps. The reproducibility you wanted for auditing is the same property that makes retrying useless. Put the seed on the command line.

Running a book

scripts/narrate-book.sh \
    --book path/to/document --out narration --engine cosyvoice
scripts/verify-narration.py narration/*.webm
scripts/publish-narration.py --narration narration --site … --slug …

Resume per stage, not per section. Synthesis is skipped when a WAV master exists, delivery encoding when the delivery file exists, everything when a manifest exists. The server starts only if something actually needs synthesising. A run killed three sections in cost nothing to restart, which is the whole reason for that granularity.

One guard, because the alternative fails quietly: when a master exists and the converter now produces different text, refuse to overwrite the text and say so. Otherwise deleting a manifest to force a re-align will align yesterday’s audio against today’s text and produce a manifest describing words the voice never said.

What a book costs, M4 with 16 GB:

WebM rather than a bare .opus: Safari’s support for Opus in an Ogg container is unreliable, Opus in WebM plays, and the failure mode is silence. Container choice is a compatibility decision.

Sample first

Render two or three sections. Run the full verification against them. Fix what it surfaces. Then start the other fifty-eight.

Nearly every defect in this article was visible in the first two sections I looked at closely. I found them one at a time across five re-render rounds instead, because after each fix I checked the thing I had just changed rather than asking what class of problem I had never tested for. That ordering mistake cost about eight hours of GPU time. Forty minutes of sampling would have replaced all of it.

Limits

Memory is not characterised. There is no MPSGraph cache to reclaim, which is the argument for not needing a supervisor, but an argument is not a measurement — and two attempts to measure it saturated at fixed values. What I can state: one process renders sixteen minutes of audio at RTF 0.54 on a 16 GB machine without the system struggling, and two engines resident do not fit.

Streaming is not implemented; the endpoint that would offer it buffers. Upstream’s text normalisation — FST-based normaliser, number spell-out, punctuation rewriting — is not ported, which is why the text rules carry so much weight here. They are doing that job by hand for the cases that matter.

Pronunciation quirks are model-specific. timezone and signup fail on this voice. Another voice will fail on different words, and the audit that finds them outlives any list I could give you.

The pipeline’s real output is not the audio. It is the evidence that the audio matches the text, produced cheaply enough to run on every section, every time.