zorro-ocr

Zorro OCR — Hindi/English OCR (MiniCPM-V 4.7 1B LoRA)

A LoRA adapter that turns openbmb/MiniCPM-V-4.7-1B (open-weight ~1.5B VLM, Apache-2.0) into a focused document-OCR model for Devanagari (Hindi) + English text, including the conjuncts and mixed-script layouts where the base model is weakest.

• Base model: openbmb/MiniCPM-V-4.7-1B
• Adapter: PEFT LoRA r=16, alpha=32 on q/k/v/o/gate/up/down projections of the LLM (no vision-tower retraining)
• Training data: 20,000 synthetic document images (Table 1) generated deterministically (seed 20261013) — pure-Devanagari, mixed Hindi+English+digits, conjunct-heavy vocabulary (क्षेत्र, ज्ञान, त्रिभुज, …)
• Recipe: bf16, 2× RTX 5060 Ti (DDP), LR 1e-4 cosine w/ 3% warmup, 2 epochs, length-batched, gradient checkpointing, NaViT variable-shape slicing (4× downsample, ≤36 slices)

Results

Measured with eval/run_eval.py (NF4, seed 0) + eval/metrics.py (NFC-normalized, whitespace-stripped Levenshtein CER).

Per-bucket mean CER on the 50-image set (base → LoRA):

The 50-image set is genuinely out-of-distribution (real fonts/layouts vs synthetic training), so the honest headline there is word accuracy 60.1% → 72.6%; mixed-script improves most, pure-Devanagari is already the base model's strong bucket. The 300-image val set shares distribution with training — it measures fit, not generalization.

Usage

from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

base = "openbmb/MiniCPM-V-4.7-1B"
adapter = "zosmaai/zorro-ocr"

proc = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base,
        torch_dtype="bfloat16", device_map="auto").eval()
from peft import PeftModel
model = PeftModel.from_pretrained(model, adapter)

msgs = [{"role": "user", "content": [
    {"type": "image", "image": Image.open("doc.jpg").convert("RGB")},
    {"type": "text", "text": "Extract all the text in this image exactly as it appears."}]}]
inputs = proc.apply_chat_template(msgs, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt",
    downsample_mode="4x", max_slice_nums=36, enable_thinking=False)
out = model.generate(**inputs.to(model.device), max_new_tokens=1024)
text = proc.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)

Notes: - Keep downsample_mode="4x" and enable_thinking=False to match training. - Works with any HF transformers ≥ the MiniCPM-V 4.7 release; flash-linear-attention is used automatically for the gated delta-net layers.

Training details

Limitations

• Trained on synthetic rendered documents only (no scans, no photos, no low-quality print); real-world performance should be validated on your own data before relying on it.
• Optimized for horizontal document text; handwritten Devanagari, vertical text, and Indic scripts other than Devanagari are not covered.
• Audio modalities and general VQA performance of the base model are out of scope for this adapter.

License

Adapter weights: Apache-2.0 (matching the base model). Synthetic training data: generated in-house.