Software EngineeringMachine Learning

From Noisy Kenyan ID OCR to JSON with MLX and Candle

Fine-tune a 50M-parameter model on synthetic Kenyan ID OCR, measure field-level accuracy, and package the fused F16 weights for Candle inference in Rust.

Hand-drawn diagram of a document photo moving through OCR and document mode into a LoRA-tuned Supra-50M model that emits JSON.
Lead image Hand-drawn diagram of a document photo moving through OCR and document mode into a LoRA-tuned Supra-50M model that emits JSON.
On this page

Audio edition

0:00 / 5:53

Ready

This work starts at the boundary where OCR stops. I already have a list of noisy text lines; the remaining job is to associate each value with the right field and emit one predictable JSON object:

{
  "serial_number": "...",
  "id_number": "...",
  "full_names": "...",
  "date_of_birth": "...",
  "sex": "...",
  "district_of_birth": "...",
  "place_of_issue": "...",
  "date_of_issue": "..."
}

For this experiment, I fine-tuned the model with MLX on an Apple silicon Mac, fused the adapter into the base weights, and loaded the result with Candle inside a Rust service:

synthetic OCR records
  -> MLX LoRA training
  -> unseen-seed synthetic evaluation
  -> fused F16 weights
  -> Candle inference probe
  -> Rust OCR service

The complete implementation is in training/ within drmhse/paddle-paddle-v6-ocr-rust . The commands below document the Apple silicon training path I used. The Candle probe and final Rust service do not depend on that training machine.

Set up the training environment

Clone the project on an Apple silicon Mac and create a virtual environment:

git clone https://github.com/drmhse/paddle-paddle-v6-ocr-rust.git
cd paddle-paddle-v6-ocr-rust/training

python3 -m venv .venv
./.venv/bin/pip install "mlx-lm[train]" huggingface_hub

The training commands follow the data and LoRA interfaces documented by mlx-lm .

Generate OCR-shaped data

gen_id.py creates synthetic OCR-to-JSON pairs. Its noise patterns are grounded in the shape of one real OCR dump, but the names, numbers, dates, and places in the generated records are fabricated. The input side reproduces problems the OCR service actually exposed:

  • values before their labels
  • adjacent or clustered labels
  • missing spaces and garbled headings
  • dates with missing punctuation
  • unrelated lines around signatures and stamps
  • optional missing fields

The target keeps names, numbers, sex, and place names exactly as the synthetic OCR input presents them. It only normalizes dates to DD.MM.YYYY. I wanted this 50M-parameter model to learn field association, not to guess corrections to people’s names.

Generate the dataset:

./.venv/bin/python gen_id.py

The fixed generator seed produces:

data_id/train.jsonl   1600 records
data_id/valid.jsonl    160 records

No real identity values are copied into these training files. That makes the generator useful for iteration without turning identity documents into a training corpus. It does not make the resulting accuracy representative of production scans; that requires a separate, access-controlled acceptance set.

Prepare the base model

I used SupraLabs/Supra-1.5-50M-Instruct-exp , a small Llama-style model. SupraLabs labels the checkpoint experimental and advises against production use, so this is a compact base for reproducing the pipeline rather than a production recommendation.

Its tokenizer metadata names TokenizersBackend, which mlx-lm does not load through the standard Transformers path. Copy the model locally and change only that metadata:

./.venv/bin/python - <<'PY'
import json
import os
import shutil
from huggingface_hub import snapshot_download

source = snapshot_download("SupraLabs/Supra-1.5-50M-Instruct-exp")
os.makedirs("supra_local", exist_ok=True)

for name in (
    "model.safetensors",
    "config.json",
    "generation_config.json",
    "tokenizer.json",
):
    shutil.copy2(os.path.join(source, name), os.path.join("supra_local", name))

with open(os.path.join(source, "tokenizer_config.json")) as file:
    tokenizer_config = json.load(file)

tokenizer_config["tokenizer_class"] = "PreTrainedTokenizerFast"
tokenizer_config.pop("backend", None)

with open("supra_local/tokenizer_config.json", "w") as file:
    json.dump(tokenizer_config, file, indent=2)
PY

This leaves the weights and tokenizer vocabulary unchanged. It only gives the existing fast tokenizer a class name that the loading path recognizes.

Convert the records to plain text

The base model does not provide a chat template. Rather than silently introducing a generic one, convert each example to the exact plain-text prompt used by the Rust probe:

./.venv/bin/python - <<'PY'
import json
import os

os.makedirs("data_id_text", exist_ok=True)

for split in ("train", "valid"):
    with open(f"data_id/{split}.jsonl") as source, \
         open(f"data_id_text/{split}.jsonl", "w") as target:
        for line in source:
            messages = json.loads(line)["messages"]
            text = (
                f'{messages[0]["content"]}\n\n'
                f'{messages[1]["content"]}\n'
                f'{messages[2]["content"]}</s>'
            )
            target.write(json.dumps({"text": text}, ensure_ascii=False) + "\n")
PY

The end-of-sequence token belongs after the target JSON, not between the prompt and completion. Keeping this format explicit also makes the serving prompt easier to compare with the training input.

Train the LoRA adapter

Run the fine-tune:

./.venv/bin/python -m mlx_lm lora \
  --model ./supra_local \
  --train \
  --data ./data_id_text \
  --iters 700 \
  --batch-size 8 \
  --num-layers 8 \
  --adapter-path ./adapters_supra \
  --steps-per-eval 350

In my recorded run, LoRA trained about 600,000 parameters—roughly 1.16% of the base model—and produced a 2.4 MB adapter. Validation loss settled near 0.50. These numbers describe that run; they are useful as a rough comparison, not as acceptance criteria.

Measure synthetic field accuracy

eval_any.py changes the random seed, generates 60 new synthetic records, runs deterministic inference, and compares every predicted field with its target:

./.venv/bin/python eval_any.py \
  gen_id ./supra_local ./adapters_supra 60

That recorded run produced valid JSON for all 60 samples and about 92% micro-average field accuracy. The evaluator also reports full-record exact matches and accuracy for each field. I care about those per-field results because an aggregate can hide one field that fails systematically.

This is an unseen-seed test, not an independent dataset. The evaluator calls the same generator used for training, so it remains in-distribution and synthetic. It is useful for iteration and regression detection. It cannot answer how the model handles a different card layout, OCR failure mode, or scan population; approved real OCR output must be evaluated separately before deployment.

Fuse and package the model for Candle

Fuse the adapter into the base weights:

./.venv/bin/python -m mlx_lm fuse \
  --model ./supra_local \
  --adapter-path ./adapters_supra \
  --save-path ./supra_id_fused

The base weights are BF16 and tie the output head to the token embeddings. This project’s Candle loader expects F16 tensors and an explicit lm_head.weight. Create that serving artifact with MLX:

mkdir -p supra_candle
cp supra_id_fused/config.json supra_candle/
cp supra_id_fused/tokenizer.json supra_candle/

./.venv/bin/python - <<'PY'
import glob
import mlx.core as mx

weights = {}
for path in glob.glob("supra_id_fused/*.safetensors"):
    weights.update(mx.load(path))

weights["lm_head.weight"] = weights["model.embed_tokens.weight"]
weights = {name: tensor.astype(mx.float16) for name, tensor in weights.items()}
mx.save_safetensors("supra_candle/model.safetensors", weights)
PY

supra_candle/ now contains the three files needed at the MLX-to-Candle boundary:

config.json
model.safetensors
tokenizer.json

Test the Rust inference boundary

Test the fused artifact before adding HTTP handling or OCR code:

cd candle-extract

printf '%s\n' \
  'SERIAL NUMBER: 880045612' \
  'IDNUMBER: 40781236' \
  'FULL NAMES' \
  'AISHA MWAKIO JUMA' \
  'DATE OF BIRTH' \
  '03.08.1991' \
  'SEX' \
  'FEMALE' \
  'DISTRICT OF BIRTH' \
  'MOMBASA' \
  'PLACE OF ISSUE' \
  'LIKONI' \
  'DATE OF ISSUE' \
  '22.04.2014' \
  | cargo run --release -- ../supra_candle f16

The probe loads the tokenizer, Llama configuration, and safetensors with Candle, then greedily generates the JSON response. Running it before adding HTTP or OCR code keeps packaging failures, tokenizer mismatches, and inference bugs in a small test boundary.

Passing this probe shows that Candle can load and run the artifact; one successful record says nothing about production field accuracy. With that boundary working, the same artifact can move into the OCR service. The companion article shows the complete local API and its three debugging layers: annotated detections, raw OCR lines, and structured fields.

Continue reading

Complete index →