Running Qwen3.8 9B on a 16 GB M4: 36.4 tok/s Decode and a 57x Prefill Cut
Which quantisation fits 16 GB, why prefill is already at a hardware limit, and the two defects that stop prompt caching from working. Prefill on a 4,312 token prompt falls …

Tag archive
Notes connected by the Caching tag.
Which quantisation fits 16 GB, why prefill is already at a hardware limit, and the two defects that stop prompt caching from working. Prefill on a 4,312 token prompt falls …
A cold cargo check of rust-analyzer goes from 23.77s to 0.47s. Three changes across cargo and rustc, what each one bought, and how two of them ship as a separate tool that …
No entries are filed here yet.
Browse all notes →