Narrating Book-Length Text on One Machine

You cannot proofread sixteen hours of audio. The one method I would trust costs two working days per pass, so correctness had to come from machinery instead. What follows is the pipeline I ended up with, and the defects that shaped each part of it. The engine is Rust on candle ; everything around it is a few scripts.

The pipeline

markdown source
  -> narration text + page-word map      (text preparation)
  -> WAV master                          (Rust TTS over HTTP)
  -> WebM/Opus delivery                  (ffmpeg)
  -> alignment manifest                  (recognition + matching)
  -> published audio + timings

The WAV master is never discarded, because synthesis is the only step expensive enough to be worth storing around, and alignment runs against the delivered file rather than the master, which makes encoder delay a measured quantity rather than an assumed one.

Three engines sit behind one trait, all faster than real time on an M4:

RTF is synthesis time over audio duration, and it sets the tempo of everything downstream: sixteen hours of speech costs eleven hours of compute at 0.7 and four at 0.25, which is the difference between a re-render being an afternoon and being overnight. qwen3tts is the current default, having got there through a day of profiling, and the structure below did not change when I swapped it in.

Each engine is a chain of fixed ratios, which is the property the scheduling relies on. CosyVoice goes text to BPE tokens, then an autoregressive LLM to speech tokens at 25 per second, then flow matching to mel frames at 2 per token, then a vocoder to 24 kHz audio. Because those ratios are fixed, a segment of n speech tokens is exactly n * 2 mel frames and exactly n * 960 samples, so several segments can be decoded in one call and the waveform cut apart afterwards at sample-exact boundaries, with no estimation and no drift.

Two scheduling decisions

Both bought more than any kernel I wrote, and both are worked out in detail elsewhere.

The autoregressive stage batches across segments. A decode step’s cost is reading the weights plus a host round-trip for sampling, neither of which scales with the sequence being extended, so one batch pays both once for many segments. It is capped at eight lanes because each lane carries about 101 MB of KV cache, and one lane per segment over a sixteen-minute section would ask for 11.9 GB, which on a 16 GB machine means swap. Swap was visible in the timings while every memory metric I had stayed flat. Eight lanes costs 0.81 GB, and the throughput curve is flat by four.

The flow decoder groups segments, but not all of them. Each call re-prepends the reference voice’s prompt of 588 mel frames, and on one passage 61% of the flow’s work was that prompt redone. Fusing the whole utterance is the obvious conclusion and it is wrong: DiT attention is quadratic, so fusing everything models out at a 3.44x penalty against 0.67x for grouping at 1600 to 2400 frames. It looked good only because I first measured it on a passage short enough to sit near the optimum. The minimum is broad, which is why a single measurement found the wrong end of it.

Driving it

One server serves the whole book, so the model loads once rather than sixty-one times:

TTS_API_KEY=… tts-serve --port 3098 \
    --engine qwen3tts --voice voices/cosy-default-qwen3tts --quant f16

curl -X POST localhost:3098/tts -H "X-API-Key: $TTS_API_KEY" \
     -H 'content-type: application/json' \
     -d '{"text":"…","seed":1234}' -o section.wav

Responses carry x-audio-seconds and x-rtf, which gives cost per section without any separate instrumentation. A library API exists behind the same trait, but for a book the server is the right shape: one load, one GPU, many requests.

What the text does to the audio

Segment length, and the text it swallowed

Text splits into paragraphs, then into segments of whole sentences under a character budget, 220 by default. That number decides prompt length, batch shape, where silence goes, and where the waveform can be cut.

My budget applied only when merging sentences, so a single sentence longer than it sailed through whole, one of them 566 characters and roughly 38 seconds of speech. Meanwhile the autoregressive loop masks the end-of-sequence token for only the first 2 * text_tokens steps, after which the model can stop wherever it likes, and on long segments it did, mid-clause.

Nothing downstream could notice. The vocoder renders the tokens it receives, so duration matches the tokens produced rather than the text that should have produced them: the file is the right length, internally consistent, and missing a clause. Twenty-four segments across one book, roughly two minutes of text present in the source and absent from the audio, all passing every check I had.

The converter now bounds every segment, splitting an over-long sentence at clause punctuation and falling back to word count, which lands the inserted gap where the voice would have paused anyway. On top of that sits a ratio guard: each segment’s speech tokens per character are compared against the median for that request, outliers are regenerated, and a persistent one fails the run with the offending text attached. A reference measured from this voice and this text beats any words-per-minute constant I could have picked.

Its ceiling is worth stating, because I spent an afternoon on the wrong side of it. Sentences merge up to the budget, so a segment that loses a 49-character clause out of 220 sits around 0.78 of the median, inside any threshold loose enough to avoid false positives. Token counts cannot see a partial loss inside a merged segment; recognition can. The ratio guard catches severe cases cheaply, and recognition is the actual acceptance test.

What the converter removes, and why

Each rule below exists because its absence produced audible damage in a published file.

Anything that is not speech goes first: front matter, code blocks, HTML comments, shortcodes, images. An HTML comment holding art direction got narrated in full.

Emphasis markers, exhaustively, then a sweep. ***bold italic*** defeats naive rules, since \*\* matches, [^*]+ fails on the third asterisk, and the italic rule’s negative lookbehind refuses to start there. The markers survived and the voice read them as “asterisk, asterisk, asterisk local design” before collapsing into "asterisksisks", and a repeated nonsense token is what sends an autoregressive model into a loop. Any surviving asterisk is now deleted unconditionally.

Underscore italics, anchored to word boundaries. Unanchored, _([^_]+)_ matches across identifiers: given `agency_account` → `client` → `source_export`, it paired the first underscore with the third, removed both, and produced agencyaccount and sourceexport. snake_case is read as separate words, and arrows are spoken as “, then”, because a dropped arrow leaves a list of nouns with no relationship between them.

Tables, row by row as sentences. Flattened into a paragraph buffer a table becomes one run-on with no sentence boundaries, separator row included. The model degenerated on one and produced twenty-one seconds of babble that two independent recognisers heard as “tampoligation, tampolition, sambolition”.

| Partner | Provides           | Requires        |
| Product | field synthesis…   | roadmap context…|

-> "Product. Provides: field synthesis… Requires: roadmap context…"

Brackets and placeholders, unwrapped. Task-list markers reached the voice thirty-four times in one section; [specific customer] and <name> keep their words and lose the punctuation.

The first word of every paragraph, capitalised. Speech has no case, so case never occurred to me as an input, and six of eight sampled lowercase openers were mispronounced: “complain” as “Dock and plane”, “recurring” as “Career go will lead E”, “reliability” as “Our liability”. They were not exotic either, because the snake_case rule above had turned identifiers into ordinary spaced words, each heading its own list item. Fixing one thing created the conditions for the next to break.

Compounds the voice cannot say, spaced. timezone came out as “Heideheb” and signup as “SignGen”. That list stays short and evidence-based, since the audit further down populates it without guesswork. Anything still resembling markup raises a warning, because a silent converter is how most of the defects above reached a published file.

Alignment comes from the audio

Knowing the exact script suggests a shortcut, and I took it first. Skip recognition, cut the script into sentence windows, give each a slice of the timeline proportional to its character count, and let a forced aligner place the known words inside. It is roughly five times cheaper, and measured against the audio it produced a median error of 4.8 seconds per word, with 11.9% of words landing within half a second of where they were spoken.

The failure is structural rather than a tuning problem. A forced aligner places words inside the span it is handed, cannot reject a bad span, and cannot report having received one. Character-proportional spans assume a constant speaking rate, which narration violates at every heading, paragraph break and inserted pause, and every word inside a window inherits its boundary error.

My statistics could not see any of it. alignedShare: 0.994 says words received timestamps and coverage says those timestamps span the file, both true and neither related to whether a timestamp is where its word is.

What replaced it recognises the audio and uses the script only for spelling:

1. Recognise with batched faster-whisper and word_timestamps=True, at 20.4x realtime on CPU, 51 s for a seventeen-minute section. The unbatched path is 7.8x on the same machine.
2. Match the recognised sequence to the document’s canonical words with difflib.SequenceMatcher, so each canonical word inherits a measured time while the document stays authoritative for spelling and recognition errors never reach the page.
3. Interpolate the remainder, marking every interpolated word, so the manifest never implies a measurement nobody made.
4. Cut cues at measured pauses.

Three details in that matching cost me time. autojunk has to be off, because SequenceMatcher treats tokens appearing in more than 1% of a long sequence as junk, which in a 2500-word section discards the, to, of and a, several hundred of the most reliable anchors available.

Anchors need support. A size-1 matching block on and pairs an occurrence in the script with an unrelated one in the transcript, and two of those pinned 62 canonical words into 1.1 seconds of audio. The matcher now requires a run of at least three words, rejects any anchor pair implying more than five words per second against narration’s 2.5 to 3, and drops the weaker member so the region becomes an honest gap rather than a compressed lie.

A second acoustic pass made it worse. I tried wav2vec2 forced alignment inside the recognised windows, reasoning that acoustic onsets must beat recognition timestamps, and it placed 1764 of 2524 words against recognition’s 2488 with a drift p90 of 4.03 s. It was discarding timings that were already right.

Cue length is what a listener actually perceives. If the player interpolates the highlight across the active cue rather than sweeping word timings, which most do, cue length bounds the visible error however precise the underlying timings are, and a cue spanning a pause highlights words during silence. Measured word times make the pauses visible, so cues are cut at three seconds, broken at sentence punctuation or a gap over 220 ms, and capped; the median lands at 2.4 to 3.0 s with a worst case under 6 s. Cue boundaries are then clamped forward, because a player selecting with findIndex(t >= s.start && t < s.end) returns the earlier cue on an overlap and the highlight jumps backwards.

Gates

Every section carries a quality block naming what failed, rather than a number that always looks fine.

The third one earns its place differently from the others:

ffmpeg -v info -i section.webm \
  -af silencedetect=noise=-40dB:d=0.20 -f null -

Silence detection knows nothing about the script, the windows or the aligner. It reads the waveform, so if cues were cut where the voice pauses, their boundaries land inside intervals ffmpeg independently calls silence. That independence is the point: a manifest can be perfectly self-consistent and still be shifted against the audio, and every check derived from the aligner agrees with the aligner. This one can disagree with it.

Integrity comes next. The closing seconds of each delivered file are transcribed, which proves both that synthesis reached the end and that the deliverable decodes, and so is every span the aligner could not measure, compared against what should have been said. Low overlap separates an audio defect from a recognition miss, and guessing between them costs hours either way.

One audit was already paid for. A word the voice mangles is never recognised, so it is already flagged as interpolated, in every occurrence, in manifests sitting on disk. Aggregating them for word types unrecognised in at least 80% of their appearances, with at least three occurrences, produced about thirty-seven suspects across a 146,000-word corpus at no compute cost.

Most of them were innocent. countermetric, tradeoff, codebase and quickstart are spoken correctly and merely transcribed as two words, thirty is transcribed as 30, and vale and “Vail” are homophones. Six of the loudest signals were orthography differences between recogniser and document, so acting on the list without listening would have triggered hours of re-rendering to fix nothing. A cheap detector produces suspects, and it stays useful for exactly as long as they are treated as suspects.

There is a related trap in determinism. Renders here are reproducible under a fixed seed, so re-rendering unchanged text reproduces it byte for byte, and I “fixed” two sections twice before noticing the verifier returned identical scores at identical timestamps. The reproducibility that makes auditing possible is what makes retrying useless, so the seed is a command-line argument.

Running a book

scripts/narrate-book.sh \
    --book path/to/document --out narration --engine qwen3tts
scripts/verify-narration.py narration/*.webm
scripts/publish-narration.py --narration narration --site … --slug …

Resume is per stage rather than per section: synthesis is skipped when a WAV master exists, delivery encoding when the delivery file exists, everything when a manifest exists, and the server starts only if something needs synthesising. A run killed thirty sections in costs nothing to restart, which is the whole reason for that granularity.

One guard exists because the alternative fails quietly. When a master exists and the converter now produces different text, the pipeline refuses to overwrite the text and says so. Otherwise deleting a manifest to force a re-align would align yesterday’s audio against today’s text and produce a manifest describing words the voice never said.

What a book cost, on an M4 with 16 GB:

Delivery is WebM rather than a bare .opus because Safari’s support for Opus in an Ogg container is unreliable while Opus in WebM plays, and the failure mode is silence rather than an error.

Nearly every defect described here was visible in the first two sections I looked at closely. I found them one at a time across five re-render rounds instead, because after each fix I checked the thing I had just changed rather than asking what class of problem I had never tested for. That ordering mistake cost about eight hours of GPU time against the forty minutes of sampling that would have replaced it.

Limits

Memory is not characterised. There is no MPSGraph cache to reclaim, which is the argument for not needing a supervisor, but an argument is not a measurement and two attempts to measure it saturated at fixed values. What I can state is that one process renders sixteen minutes of audio at RTF 0.54 on a 16 GB machine without the system struggling, and that two engines resident do not fit. Streaming is not implemented. Upstream’s text normalisation is not ported, which is why the text rules carry so much weight here. Pronunciation quirks are model-specific, so another voice fails on different words than timezone and signup, and the audit that finds them outlives any list I could write down.

The output that matters is not the audio. It is the evidence that the audio matches the text, produced cheaply enough to run on every section, every time.