Machine LearningPerformanceApple Silicon

Running Qwen3.8 9B on a 16 GB M4: 36.4 tok/s Decode and a 57x Prefill Cut

Which quantisation fits 16 GB, why prefill is already at a hardware limit, and the two defects that stop prompt caching from working. Prefill on a 4,312 token prompt falls from 19.42 s to 0.35 s, and a 9B native drafter carries decode from 20.8 to 36.4 tok/s without changing a token of the output.

Two horizontal bars on a shared twenty second axis. The first turn is almost entirely prefill in rust with a short grey decode segment; the second turn collapses to a sliver of the same width scale.
Lead image Two horizontal bars on a shared twenty second axis. The first turn is almost entirely prefill in rust with a short grey decode segment; the second turn collapses to a sliver of the same width scale.
On this page

Audio edition

0:00 / 17:07

Ready

A 9B model on a 16 GB machine sits close enough to the edge that most of the tuning advice written for larger machines stops applying. Everything below ran on an Apple M4 with 10 cores, 4 performance and 6 efficiency, and 16 GB of unified memory, on macOS 15.7.2, with mlx 0.32.0 and mlx-vlm 0.6.8, and the speculative decoding at the end under dflash-mlx 0.1.8 on mlx-lm in a separate environment. The serving scripts, the two cache repairs and the harness that produced the numbers below are in drmhse/qwen38-9b-mlx . The weights are the MLX conversions of Qwen3.8 9B, whose config.json declares model_type: qwen3_5 and whose manifests record it as a full-parameter distillation of Qwen3.5 9B. That distinction decides which runtime code paths apply, since every runtime keys its model implementation on the architecture rather than the release name.

The architecture explains most of what follows. From the tensor headers: 32 layers, hidden size 4096, intermediate size 12288, vocabulary 248,320, and a 27 block vision tower left at BF16. Full attention appears every fourth layer, so 8 layers use it and the other 24 are gated DeltaNet linear attention carrying recurrent state.

Quantisation and the 11.5 GB Metal budget

The publisher ships 4-bit at 5.57 GiB, 8-bit at 9.74 GiB and BF16 at 17.55 GiB. The reported peak for 8-bit is 11.37 GB, measured on an M5 Max with 128 GB. On this machine the Metal working set is 11.5 GB of the 16 GB total, and the 8-bit variant exceeds it in practice. MLX raised kIOGPUCommandBufferCallbackErrorOutOfMemory partway through generation, and that allocation failure took the whole machine down with it. The 4-bit variant peaked at 6.74 GB on a short prompt and 7.99 GB at 7,022 tokens, which leaves room for the rest of the system.

Two signals pointed the same way before the crash, and one of them was misleading. A capacity planner recommended 4-bit for an 11.5 GB budget, which was sound; a device mapper also refused the 8-bit load, but on an estimate that priced the already quantised weights as BF16 and so claimed 19,169 MB. MLX failing at the Metal layer is the measurement that settles it. A peak memory figure measured on a 128 GB machine describes the model, not the machine it has to run on.

Measured limits on decode and prefill

Decode reached 20.8 tok/s and did not move when the runtime or its settings changed. Upgrading mlx from 0.32.0 to 0.32.1 gave 20.69 tok/s against 20.84. Quantising the KV cache at a 7,022 token prompt gave 20.80 tok/s at BF16, 20.01 at 8-bit and 21.13 at 4-bit, which is noise rather than an effect.

The reason is bandwidth. A streaming read of a 512 MB buffer measured 102.5 GB/s. A 8192 by 8192 BF16 matrix vector product, which is the access pattern of a decode step, measured 81 GB/s. Decode reads every weight once per token, so 20.8 tok/s implies roughly 4.3 GB to 4.9 GB moved per token. That is the language model’s share of the 5.98 GB of weights, putting decode at 85% to 100% of the bandwidth this GPU can actually reach. The ceiling is bytes per token rather than tokens per second, which is the gap a drafter later walks through.

Prefill measured 224.1 tok/s at 2,523 tokens, 217.0 at 7,022 and 198.7 at 15,922. Its limit is arithmetic rather than bandwidth, so the comparison is against GEMM throughput. A dense BF16 product at prefill shapes measured 3.73 TFLOP/s, the same product with 4-bit affine quantised weights 3.29 TFLOP/s, and a 4096 cubed square product 3.62 TFLOP/s. Counting the two dimensional tensors that participate in a prefill step gives 6.919 B parameters: 4.832 B in the MLPs, 1.617 B in the linear attention layers and 0.470 B in full attention. A further 1.017 B output projection runs only on the last position. At 13.8 to 15.9 GFLOP per token, 224.1 tok/s works out to 3.10 to 3.56 TFLOP/s, or 94% to 108% of the measured quantised GEMM ceiling.

A Rust runtime was slower, against my expectation. Assuming a compiled serving stack would beat Python, I ported the same weights to mistral.rs, which needed three transforms first. MLX stores convolution weights channels last, and it bakes the +1.0 offset into the five RMSNorm families that GemmaRmsNorm adds again at load. It also nests lm_head under language_model, where the loader reads it from the root. The norm mismatch was the instructive one: the model loaded with no error and produced fluent looking multilingual noise. Decode then measured 17.4 tok/s against 20.8. Both stacks dispatch the same Metal kernels, and at 9B the interpreter is not on the critical path.

Optimisations with no headroom

Two saturated limits remove most of the usual tuning surface, and the removals are worth naming because each one has a plausible story behind it.

changemeasured effectreason
raising prefill step size above 1024nonequantised GEMM is flat at 3.28 to 3.29 TFLOP/s for M of 1024, 2048 and 4096
dequantising to a dense GEMMnot attempted4-bit already reaches 88% of dense BF16, bounding the gain at 1.13x
flash or sparse attention, attention sinksnot attempted2,523 to 15,922 tokens cost only 224.1 to 198.7 tok/s, since 8 of 32 layers use attention
3-bit or 2-bit weightsnot attemptedfewer bytes per token helps bandwidth bound decode, not compute bound prefill
concurrent prefillsnot attemptedprefill saturates compute, so concurrency divides the same FLOPs

Prefill costs about 4.5 s per 1,000 tokens on this hardware, and a 40 token reply costs about 2 s. Prompt length therefore sets latency, and the only remaining moves are to skip prefill or to send fewer tokens.

Speculative decoding belonged in that table when I first wrote it, dismissed on the grounds that it proposes future tokens while prefill already processes the prompt in parallel. The first half of that is right and the conclusion is wrong, because it answers a question about prefill with an argument that never applied to decode, where the bound is bytes moved rather than work done. It is the one lever here that turned out to have headroom, and the last section measures it.

Prompt caching and its two defects

mlx-vlm ships an automatic prefix cache that is off unless APC_ENABLED=1 is set. Turning it on and repeating a byte identical 4,312 token prompt cut prefill from 19.5 s to 0.27 s. Growing a conversation turn by turn, which is the shape agent traffic actually takes, still reported zero cached tokens on every turn. Two separate defects were responsible, and repairing either alone changed nothing.

The first is in the chat template. Under enable_thinking=false the generation prompt ends with <think>\n\n</think>\n\n, while assistant turns already in the history render as plain content, because the template keeps that scaffold only for the most recent turn. Tokenising both prompts showed turn one ending at token 21 of 23 with <think>, where turn two continued with the reply text. Turn one was therefore not a prefix of turn two, and no prefix cache could match. Rendering history turns with the same scaffold makes each prompt an extension of the last.

The second is in mlx-vlm itself, in the salt that keys the cache. _apc_extra_hash folds inputs_embeds and attention_mask into that key, and the server always supplies inputs_embeds, noting in a comment that its batch generator requires them. Embeddings are a function of the entire prompt, so every distinct prompt produced a distinct key, and prefix reuse was impossible by construction rather than by accident. Instrumenting the lookup confirmed the mechanism: candidate snapshots were rejected with reason=extra_hash and never on a token mismatch, with the two turns hashing to -1907502081116388294 and 2680841521439465829. The salt exists to separate media payloads that share token ids, so omitting embeddings for text only requests and keeping the conservative key whenever pixels or audio are present restores reuse without weakening it.

With both repairs in place, on a 4,312 token system prompt:

turnprompt tokenscached tokensprefill (s)wall (s)
14,312019.4220.10
24,3354,3120.350.69
34,3594,3350.340.68
44,3804,3590.312.77

Prefill falls by 57x after the first turn, and end to end latency by about 30x. Three defaults also matter more than their names suggest. APC_DISK_MIN_FREE_RAM_GB defaults to 2.0 and is compared against psutil’s available memory, which reads about 1.7 GB on macOS even at 74% free. Every warm restore was therefore skipped, with a log line rather than an error. APC_EXACT_CACHE_ENTRIES defaults to 2, and the in-memory snapshots are the only place prefix matching happens, since the disk tier is keyed by exact prompt. APC_EXACT_PREFIX_GUARD_TOKENS defaults to 16, which is why a hit reports the previous prompt’s length rather than the current one.

Concurrent requests

Decode reads all weights per step whatever the batch size, so concurrent requests share one pass over them. Four concurrent requests produced 38.18 tok/s in aggregate against 18.74 for one, a factor of 2.04. Both figures are client side wall clock and include request overhead, which is why the single request number sits below the 20.8 tok/s the server reports for decode alone. Aggregate throughput is what rises: each of the four requests is served at roughly 9.5 tok/s, so concurrency raises total work done and slightly worsens the latency of any one request.

Exact-mode caching and the recurrent state

The cache runs in exact mode rather than block mode, and the reason is architectural. Reusing an arbitrary prefix means rebuilding state at a token boundary. That works for attention keys and values, but not for the recurrent state in the 24 linear attention layers. mlx-vlm reports this directly, through a per layer capability list marking the attention layers pageable and the rest checkpoint only. Independent queries that share a long system prompt but no conversation history therefore still miss, because no snapshot exists at the shared boundary. Running them as one growing conversation is what makes them hit.

Speculative decoding, the lever I dismissed

A drafter proposes a block of tokens, the target verifies the whole block in one pass, and every accepted token in that block therefore shares a single read of the weights. Nothing about the 102.5 GB/s ceiling changes; the tokens produced per byte moved does. z-lab/Qwen3.5-9B-DFlash is the drafter that fits, at 2.4 GiB of BF16, and it fits because of one tensor: fc.weight is [4096, 32768], fusing hidden states from 8 of the target’s layers, 1, 5, 9, 13, 17, 21, 25 and 29. bind_target_model asserts no dimensions at all and reads only embed_scale, so that concatenation width is the entire contract between the two models. The drafter ships no embed_tokens and no lm_head and borrows the target’s, which means draft logits come out of this abliterated head rather than a stock one.

Measured on the same prompt throughout, 100 tokens out, with 86% of drafted tokens accepted:

configurationdecode (tok/s)against target onlybit identical to target
target only, mlx-lm, greedy21.41.00xreference
DFlash, default kernels36.41.70xyes
DFlash, verify kernels, qmm on42.92.01xno, relative 4.3e-3
DFlash, verify kernels, qmm off36.41.70xyes

Losslessness was checked rather than assumed: at 36.4 tok/s the output is byte identical to mlx_lm generate --temp 0 on the same prompt. The 20.1 to 21.4 tok/s and 226.9 tok/s that mlx-lm reports for the target alone also reproduce both roofs above on a second runtime, which is the closest thing to an independent check either number has. Peak memory was 5.99 GB against 5.20 GB for the target alone, so the drafter cost 0.79 GB rather than 2.4: mlx-lm loads lazily and quantises per tensor, so the BF16 copy never exists whole, which is what keeps both models inside the 11.5 GB budget. The speedup also held with context, 1.51x at 1,865 prompt tokens against 1.48x at 23, where the published M5 Max curve falls from 4.37x to 2.22x because it starts far higher.

The 2.01x row is a numerics trade rather than free speed. dflash-mlx gates its hand written Metal verify kernels on num_layers >= 40, which excludes a 32 layer model on layer count alone and not on capability: the per linear gate accepts 248 of the 249 QuantizedLinear modules here, rejecting only lm_head on a deliberate N < 100_000 rule, and _PROJ_TAGS already carries explicit tags for the gated DeltaNet projections. Swapping all 248 buys nothing on its own, at 36.4 tok/s either way. The whole 18% comes from one kernel, the M equals 16 qmm path, which accumulates in fp16 and is precisely the one that is not bit exact. Isolated on mlp.gate_proj, the stock and verify linears agree at n of 1, 8 and 16 with that path off, and at n of 1 and 8 with it on, diverging only at 16, which is the DFlash block size and therefore perturbs the verifier on every block. Enabling it means the verifier is no longer numerically the model it verifies against.

None of this became the default server, because leaving mlx-vlm costs four things at once. dflash-mlx builds on mlx-lm, so the vision tower goes, and with it continuous batching: 38.18 tok/s aggregate across four requests still beats 36.4 on one, so the switch wins for a single interactive stream and loses for concurrent agent traffic. The two caching repairs do not come along either, which returns prefill to its cold cost and gives back a 57x that is worth more than a 1.70x on most real traffic. Thinking is also on by default in this runtime, so a real task spends tokens before answering.

The multi token prediction head is still unmeasured, and it is now the smaller question. The source checkpoint carries all 15 mtp.* tensors, 0.49 GB at BF16, which the MLX conversions deliberately exclude. mistral.rs implements the head but reaches it only through PagedAttention, which fails on Metal for this architecture. Hybrid models request zero KV blocks for their recurrent layers, so the cache allocator asks Metal for a zero length buffer, and Metal refuses where CUDA does not. Clamping that allocation to one element got past the failure and into a duplicate function constant index while compiling the paged attention kernels at runtime. Since DFlash already carries decode past the roof losslessly, what MTP would add is an increment on a solved axis. The larger drafters do not port at all, because both the 27B MTP head and incoai/Qwen3.8-27B-DFlash2 are shaped to a 5,120 wide residual stream, with fc.weight at [5120, 25600] fusing 5 of that model’s 64 layers, and nothing but the vocabulary lines up with 4096, 12288 and 32.

What remains open is prefill, which none of this touches. A drafted block still has to be verified against a prompt the target has already read, so the 4.5 s per 1,000 tokens is untouched by anything measured here, and the only two moves against it are still the cache and sending fewer tokens.

Continue reading

Complete index →