Baidu's Unlimited-OCR (DeepSeek-OCR architecture: SAM-B + CLIP-L page
encoder, 12-layer DeepseekV2 MoE decoder, 3B parameters / ~0.5B active) for Apple silicon, including its
multi-page mode: up to 16 pages go in as one prompt and the whole document comes out of a single generation as
markdown, <PAGE>-separated, with <|det|>label [x1, y1, x2, y2]<|/det|> layout boxes (0–999 page coordinates).
Offline.
Two runtimes share the Core ML page encoder:
- MLX decoder (
mlx/, recommended): 8-bit, upstream's R-SWA cache, ~200–260 tokens/s on an M5 Pro. Baidu's
14-page paper in one pass: 13,420 tokens in ~70 s.
- Core ML decoder (
UnlimitedOcrDecoder.mlpackage): stateful 8-bit, ~50 tokens/s.
Runtime: UnlimitedOcrFastManager (MLX) and UnlimitedOcrManager (Core ML) in
FluidUse (Swift, macOS 15+), with a SwiftUI demo (UnlimitedOcrDemo):
drop a PDF, every page is read in one pass while the page thumbnails highlight the block being written.
Packages
| File |
Role |
Size |
Precision |
UnlimitedOcrPageEncoder.mlpackage |
DeepEncoder + projector: one 1024 × 1024 page → 273 embeddings (16 rows of 16 + row newlines + view separator) |
0.8 GB |
fp16 |
UnlimitedOcrDecoder.mlpackage |
decoder with a stateful K/V cache; functions prefill_P{384,640,1152,2304} (128 tokens per call) and decode_P… (1 token), weights shared |
2.7 GB |
8-bit weights (per block of 32), fp16 compute |
mlx/decoder.safetensors, mlx/config.json |
MLX decoder (embeddings, 12 layers, SwitchGLU experts, head) |
3.1 GB |
8-bit affine, group 64 |
embeddings.f16 |
input token embeddings for the Core ML decoder (host-side gather) |
0.3 GB |
fp16 |
tokenizer.json, config.json |
tokenizer; shapes, buckets, prompt token ids |
Upstream's R-SWA attention keeps the whole prefix (prompt + page tokens) plus a ring of the last 128 generated tokens,
so the cache has a fixed size per prefix bucket: k_cache / v_cache are Core ML state, [12 × (P + 128), 1280].
P = 384 / 640 / 1152 / 2304 holds 1 / 2 / 4 / 8 pages (each page is 273 tokens). Decoder inputs: x [1, C, 1280]
embeddings, cos / sin [C, 128], additive mask [1, 1, C, P + 128 + C] (cached slots, then the C new tokens),
write_idx [C] cache slots, out_index [1]; outputs the top-16 logits of each vocab half. The host (Swift runtime is
the reference) assembles the prefix, places generated token t in slot P + t % 128, and applies upstream's 35-gram
no-repeat ban.
Quality
Reference: upstream's own code (trust_remote_code), fp32 on CPU, greedy, same pixels.
|
vs upstream fp32 |
| 1 page (Baidu paper p. 4), 865 tokens |
identical |
| 3 pages in one generation (p. 3–5), 2,461 tokens |
identical text; 2 layout-box coordinates differ by 1 |
| 4-bit decoder (not shipped) |
diverges at token 329 |
The page encoder must run on the GPU: on the Neural Engine its output is wrong (row cosine down to 0.66).
Speed (M5 Pro, 24 GB, GPU, Swift)
|
|
| page encoder |
~100 ms / page |
| prefill, 8 pages (2,189 tokens) incl. encoding |
3.0 s |
| decode |
~19 ms / token at 1 page, ~23 ms at 8 pages; flat over the whole read |
| Baidu paper, first 8 pages in one read: 6,727 tokens |
160 s (first release; ~17% less decode time per token since) |
| memory |
1.9 GB footprint + 2.7 GB mapped decoder weights |
The decoder computes all 64 experts per token (routing weights zero the unselected ones), which keeps the graph static
but reads every expert's weights on each step.
Usage (Swift)
import FluidUse
import PDFKit
import UnlimitedOcrMLX
let manager = try await UnlimitedOcrFastManager.load(from: bundleDirectory) // MLX; UnlimitedOcrManager = Core ML
let pdf = PDFDocument(url: pdfURL)!
let pages = try (0..<min(pdf.pageCount, manager.maxPages)).map {
try UnlimitedOcrPagePreprocessor.page(from: pdf.page(at: $0)!)
}
var text = ""
for try await event in manager.read(pages: pages) {
if case .text(let delta) = event { text += delta }
}
print(UnlimitedOcrDocument(parsing: text).markdown)
License and credits
MIT, same as baidu/Unlimited-OCR (LICENSE is Baidu's). All model
weights are Baidu's; this repository only changes their format and precision. Conversion by
Fluid Inference.
