IndicOCR

1. Overview

IndicOCR (IndicPixel) is a large-scale multilingual Optical Character Recognition (OCR) and document dataset covering the South Asian linguistic landscape. The dataset provides dense document coverage across 23 official and literary languages representing 12 distinct writing systems.

Dataset Specifications:

• Scale & Scope: Over 12,000,000+ verified OCR pairs organized into 2,400+ standardized POSIX WebDataset shards (.tar), enabling distributed multi-GPU streaming without local disk bottlenecks.
• 4-Tier Document Hierarchy: Encompasses complete structural representation from isolated tokens to complex multi-column documents:
• Tier 1: Full-Page Document Spreads (2%) — Multi-column administrative, literary, and broadsheet layouts with word-level bounding box ground truth.
• Tier 2: Multi-Line Paragraph Blocks (13%) — Wrapped multi-line natural reading sequences.
• Tier 3: Reading Lines (55%) — Core continuous text sentences with token-level bounding coordinates.
• Tier 4: Isolated Tokens & Complex Conjuncts (30%) — Rare samyuktaksharas, ligatures, numerals, and vocabulary words.
• Complex Text Layout (CTL) Fidelity: Native HarfBuzz OpenType shaping and FreeType rasterization with font-level metrics to prevent diacritic truncation and glyph clipping across all scripts.
• Realistic Document Degradation: High-fidelity simulation of scan wear, bleed-through, paper textures, non-uniform illumination, folds, and compression artifacts for real-world document generalization.

2. Dataset Portfolio & Linguistic Coverage

3. Structural Specification & Annotation Format

Each sample is packaged inside standardized WebDataset .tar archives containing paired lossless image files (.webp) and comprehensive metadata descriptors (.json):

{
  "sample_key": "kan_0008644",
  "tier": "Tier_2_Paragraph",
  "lang": "kan",
  "script": "Knda",
  "font_name": "BalooTamma2-Regular.ttf",
  "is_clean": false,
  "canvas_size": [669, 213],
  "text": "ಅದನ್ನು ತಂದು ಶ್ರೀನಿವಾಸನಿಗೆ ಕೊಡುತ್ತಾಳೆ. ಅಂಗಡಿಗೆ ಹಿಂದಿರುಗಿ\nಬಂದ ಶ್ರೀನಿವಾಸ ಡಬ್ಬಿಯನ್ನು",
  "tokens": [
    {"text": "ಅದನ್ನು", "bbox": [24, 16, 110, 68]},
    {"text": "ತಂದು", "bbox": [122, 16, 185, 68]},
    {"text": "ಶ್ರೀನಿವಾಸನಿಗೆ", "bbox": [198, 16, 360, 68]}
  ],
  "token_count": 22,
  "applied_augmentations": [
    "#04 Substrate: notebook_ruled",
    "#40 Keystone Tilt"
  ]
}

Metadata Fields:

• sample_key (string): Unique globally indexed identifier across shards.
• tier (string): Structural tier (Tier_1_FullPage, Tier_2_Paragraph, Tier_3_Line, Tier_4_Word).
• lang (string): ISO 639-3 language identifier.
• script (string): ISO 15924 script tag.
• font_name (string): Typographic font family utilized for CTL rendering.
• canvas_size (list of int): [width, height] in pixels.
• text (string): Canonical ground truth text in Unicode NFC normalization.
• tokens (list of dict): Word-level bounding boxes [x0, y0, x1, y1] tightly enclosing ink boundaries.
• token_count (int): Total word/token count.
• applied_augmentations (list of string): Stochastic physical degradation operators applied.

4. Distributed Streaming Quickstart (PyTorch / WebDataset)

The dataset is formatted for direct streaming directly into GPU memory without prior disk extraction:

import webdataset as wds
from torch.utils.data import DataLoader

# Example: Stream Kannada shards directly from Hugging Face Hub
shard_url = "https://huggingface.co/datasets/Faizaniqbal/IndicOCR/resolve/main/data/kannada/kan_train_{00000..00099}.tar"

dataset = (
    wds.WebDataset(shard_url, resampled=False, shardshuffle=True)
    .shuffle(1000)
    .decode("pil")
    .to_tuple("webp", "json")
)

dataloader = DataLoader(dataset, batch_size=32, num_workers=4)

for images, metadata in dataloader:
    # Forward pass to Vision-Language Model / TrOCR / Donut / LayoutLMv3
    pass

5. Licensing & Citation

IndicPixel is released under the Apache 2.0 License for open commercial and academic research.

@dataset{indicpixel_2026,
  author    = {Faizan Iqbal},
  title     = {IndicPixel: A Large-Scale Foundational Multilingual Document AI and OCR Benchmark for Pan-Indic Languages},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/Faizaniqbal/IndicOCR}
}