The OCR engine reads text. The understanding model turns that text into a document record.
For Kenyan ID extraction, I wanted a small local model that does one thing: receive noisy OCR lines and return this JSON shape:
{
"serial_number": "...",
"id_number": "...",
"full_names": "...",
"date_of_birth": "...",
"sex": "...",
"district_of_birth": "...",
"place_of_issue": "...",
"date_of_issue": "..."
}
I published the code as drmhse/paddle-paddle-v6-ocr-rust
. The important directory is training/.
This is the pipeline:
synthetic OCR lines
-> MLX LoRA fine-tune
-> held-out evaluation
-> fused weights
-> Candle probe
-> Rust OCR API
Generate The Data
The generator is training/gen_id.py.
It does not generate perfect labels beside perfect values. That would train the wrong skill. The real OCR output from this document type is messier:
- values can appear before labels
- labels can cluster together
- headers and labels can be garbled
- spaces can disappear
- dates can lose punctuation
- junk lines can appear near signatures or stamps
The training target is field association under that noise. Names and numbers are copied verbatim from OCR. Dates are normalized to DD.MM.YYYY. Missing fields become null.
Start in the training directory:
cd ~/src/paddle-paddle-v6-ocr-rust/training
python3 -m venv .venv
./.venv/bin/pip install mlx-lm huggingface_hub boto3 fastapi "uvicorn[standard]" python-multipart requests
./.venv/bin/python gen_id.py
That writes:
data_id/train.jsonl 1600 examples
data_id/valid.jsonl 160 examples
The strongest privacy property is here: the model is trained on synthetic ID-like OCR examples, not on real identity documents.
Patch The Base Model For MLX
This part is the training side of the workflow. I use MLX on Apple Silicon to produce the adapter. The served OCR API does not require MLX; it consumes fused weights through Candle in Rust.
The base model is SupraLabs/Supra-1.5-50M-Instruct-exp, a small Llama-style model.
The upstream tokenizer metadata uses TokenizersBackend, which transformers and mlx_lm do not load directly. I patch the local model copy by snapshotting it into ./supra_local, keeping the safetensors/config/tokenizer files, then rewriting tokenizer_config.json to use PreTrainedTokenizerFast and remove the backend key.
The point is not to change the model. It is to make the tokenizer load through the standard fast-tokenizer path that MLX expects.
Prepare MLX Text Records
Supra does not ship a chat template that MLX can use here, so the training examples are written as plain text:
{system}
{user}
{completion}</s>
Generate the data_id_text/ directory from the chat-style JSONL:
mkdir -p data_id_text
./.venv/bin/python - <<'PY'
import json, os
os.makedirs("data_id_text", exist_ok=True)
for split in ("train", "valid"):
with open(f"data_id/{split}.jsonl") as fin, open(f"data_id_text/{split}.jsonl", "w") as fout:
for line in fin:
messages = json.loads(line)["messages"]
system = messages[0]["content"]
user = messages[1]["content"]
assistant = messages[2]["content"]
text = f"{system}\n\n{user}\n{assistant}</s>"
fout.write(json.dumps({"text": text}, ensure_ascii=False) + "\n")
PY
I also keep conv_pc.py for prompt/completion experiments, but the command below uses the raw text format.
Train The LoRA
Train the adapter:
./.venv/bin/python -m mlx_lm lora \
--model ./supra_local \
--train \
--data ./data_id_text \
--iters 700 \
--batch-size 8 \
--num-layers 8 \
--adapter-path ./adapters_supra \
--steps-per-eval 350
This trains about 0.6M parameters, roughly 1.16% of the model. The adapter is about 2.4 MB. In my run, validation loss settled around 0.50.
That size is the reason the approach is practical. The OCR models do the visual work. The small model only learns the document contract.
Evaluate Held-Out OCR
Do not judge this with one pretty sample.
training/eval_any.py imports the same generator with an unseen random seed and asks the adapter to extract fields from fresh examples:
./.venv/bin/python eval_any.py gen_id ./supra_local ./adapters_supra 60
My current held-out run reports:
valid JSON: 60/60
micro-average field accuracy: about 92%
The evaluator also prints per-field hits and full-record exact matches. That matters because one field can be systematically weak while the headline average still looks fine.
Fuse The Adapter
The server does not load MLX adapters at request time. The adapter is fused into the base model first:
./.venv/bin/python -m mlx_lm fuse \
--model ./supra_local \
--adapter-path ./adapters_supra \
--save-path ./supra_id_fused
The deployed artifact is an f16 safetensors model. I keep the small LoRA adapter for reproducibility, but the Rust API downloads the fused weights from the model CDN and verifies the checksum before loading.
When publishing a new version, bump the model prefix and update the manifest checksum in src/remote.rs.
Cross The MLX To Candle Boundary
This is the part worth testing separately.
Training happens with MLX. Serving happens inside the Rust OCR API with Candle. Before wiring the model into the server, use training/candle-extract/ as a small inference probe.
After preparing a Candle-readable model directory with model.safetensors, config.json, and tokenizer.json, run:
cd ~/src/paddle-paddle-v6-ocr-rust/training/candle-extract
printf '%s\n' \
'SERIAL NUMBER: 880045612' \
'IDNUMBER: 40781236' \
'FULL NAMES' \
'AISHA MWAKIO JUMA' \
'DATE OF BIRTH' \
'03.08.1991' \
'SEX' \
'FEMALE' \
'DISTRICT OF BIRTH' \
'MOMBASA' \
'PLACE OF ISSUE' \
'LIKONI' \
'DATE OF ISSUE' \
'22.04.2014' \
| cargo run --release -- ../supra_candle f16
The probe loads the tokenizer, config, and safetensors with Candle, builds the same extraction prompt, and greedily generates JSON. If this fails, fix the conversion before touching the web API.
training/sidecar.py is useful for development experiments, but it is not the production path. The production path is fused Supra weights loaded by the Rust server.
Ship The API Mode
The API keeps this document behavior behind an explicit mode:
mode=kenya_id&understand=true
The response includes OCR lines, detection boxes, and extracted fields. Field confidence is the understanding model’s weakest token probability while emitting the field value. It is not a PP-OCR line confidence and it is not a substitute for evaluation.
That gives you three debugging layers:
annotated image did detection find the text?
raw OCR lines did recognition read the text?
JSON fields did the model associate values correctly?
What To Reuse
The reusable pattern is not “train one model for every Kenyan document.”
The reusable pattern is narrower and more useful:
one document family
one OCR mode
one schema
one synthetic generator
one held-out evaluator
one serving artifact
For another layout, start again at the generator. If the document is regular enough, a parser may beat a model. If the layout is broad or the sensitivity is low, a hosted multimodal API may be the better first version.
For this ID workflow, the small model is worth it because it keeps the image local, makes the output contract explicit, and turns OCR from a text dump into an application field record.
