advspot-public

AdvSpot

AdvSpot is the first grounded adversarial OCR benchmark, introduced in the paper ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation. It targets visual text patterns that are readable to humans but challenging for multimodal large language models (MLLMs) to localize and recognize, and evaluates models with region-level annotations: bounding boxes, transcriptions, perception-type labels, and region-grounded VQA pairs.

This repository hosts the public, desensitized release of AdvSpot. See the Provenance, Desensitization & Content Safety section for an explanation of how this release differs from the evaluation set described in the paper, and how to report residual sensitive samples.

Table of Contents

• Dataset Summary
• Taxonomy & Distribution
• Data Fields
• Example
• Evaluation
• Provenance, Desensitization & Content Safety
• Intended Use & Limitations
• License
• Citation

Dataset Summary

AdvSpot formulates adversarial OCR as a grounded OCR perception task. For each image, one or more regions are annotated with a bounding box, a transcription, one or more perception-type labels, and a region-grounded VQA pair whose answer is exactly that region's transcription. A model is expected to identify the relevant text region and recognize the adversarial text content under challenging visual conditions — not merely infer an answer from global context.

This public release contains:

• 383 images and 389 grounded VQA pairs.
• 5 primary categories and 13 fine-grained subtypes (see Taxonomy & Distribution).
• Every subtype is kept at 30–50 samples, consistent with the evaluation set used in the paper; counts are reported at the adversarial-pattern level — see Taxonomy & Distribution.

The dataset is designed to be used together with the ArmorOCR code and model: - Code / evaluation script: advspot_infer.py reads this data_public.jsonl directly and produces metrics (#metrics.json) and badcase files (#badcases.csv). - Model weights: inclusionAI/ArmorOCR.

Taxonomy & Distribution

AdvSpot groups adversarial OCR patterns into 5 primary categories and 13 fine-grained subtypes, organized by the underlying OCR failure mechanism. The labels field stores the Chinese subtype names; the table below maps English ↔ Chinese, defines each subtype, and gives the per-subtype count. Every subtype is kept at 30–50 samples.

Counts are computed at the adversarial-pattern level: a single VQA pair may carry multiple subtype labels (30 of the 389 pairs do), so the per-subtype counts sum to more than the 389 unique VQA pairs (383 unique images; 6 images each contribute 2 region-pairs).

Overall: 383 images · 389 VQA pairs.

Data Fields

Each line of data_public.jsonl is one grounded VQA sample (one region) with the following fields:

conversations — a single grounded-VQA turn pair: - human — the image (via the <image> token) followed by a region-specific question that asks the model to recognize the hidden text and output it together with its bounding box as JSON ({"text":..., "bbox":[...]}, coordinates in 0–1000). - gpt — the ground-truth recognized text, as a JSON-encoded array string (e.g. "[\"Banker\"] → ["Banker"] after json.loads).

Both thegpt answer and the bbox are stored as JSON-encoded strings (not native JSON arrays) — json.loads them when parsing in Python.

Example

A single sample from data_public.jsonl (AIGC Fusion Text subtype), shown as stored with indentation added for readability:

{
  "id": "f4fa1bb566e2b3460103e9a1f28b5f88#0",
  "image": [
    "images/f4fa1bb566e2b3460103e9a1f28b5f88.png"
  ],
  "labels": [
    "背景同化对抗-AIGC合成"
  ],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\n图中隐藏的英文单词是什么？请用JSON格式输出答案和对应文本的边界框，格式为：{\"text\":\"答案内容\",\"bbox\":[x_min,y_min,x_max,y_max]}，坐标值范围0-1000，不要输出多余解释。"
    },
    {
      "from": "gpt",
      "value": "[\"Banker\"]"
    }
  ],
  "bbox": "[123, 193, 597, 419]"
}

Evaluation

AdvSpot is evaluated with two region-grounded metrics:

• Grounded VQA accuracy — whether the predicted recognized text matches the ground-truth transcription(s) for the target region. Following the paper, the answer is correct when the ground-truth text set is covered by the model's predicted text set (R_vqa = I[a* ⊆ â]).
• Average localization IoU — the Intersection-over-Union between the predicted bounding box and the ground-truth box, averaged over samples.

The official evaluation is performed by advspot_infer.py in the code repository.

Results

Base = Qwen3-VL-8B-Instruct; ArmorOCR = the model trained with the ArmorOCR framework. Results are measured on this public release (383 images / 389 grounded VQA pairs). All values are in %.

Provenance, Desensitization & Content Safety

This public release has been desensitized: sensitive samples were removed, privacy-related information was masked, and replacement samples were synthesized to keep the benchmark balanced. The resulting set passed multiple rounds of manual review.

Residual-risk notice. Sensitive or rights-infringing samples that we missed may still remain in this release. If you find any sample that infringes privacy, copyright, or other rights, or is otherwise inappropriate, please contact us so we can remove it:

• Open an issue: https://github.com/ant-research/ArmorOCR/issues
• Please include the sample id.

We ask downstream users not to redistribute flagged samples, and to respect applicable laws and platform policies when using the dataset.

Intended Use & Limitations

Intended use

• Evaluating and benchmarking adversarial / grounded OCR perception in multimodal LLMs.
• Research on visual-text robustness, region grounding, and VQA under challenging visual conditions.
• Driving model improvement (e.g. the ArmorOCR training framework).

Out of scope

• Training or deploying systems to detect, surface, or distribute sensitive content.
• Any use that violates applicable law or platform content-safety policies.
• Treating the dataset as representative of general OCR scenarios — AdvSpot is deliberately narrow: it targets adversarial perception patterns, not natural / document OCR.

Limitations

• Scale: 383 images / 389 VQA pairs. AdvSpot is a focused, high-quality benchmark, not a large-scale training set.
• Coverage: 5 categories / 13 subtypes designed around OCR failure mechanisms; it does not exhaust all possible adversarial text patterns.
• VQA generation: region-grounded questions are model-generated (Qwen3-VL-235B-A22B) and human-verified, but may carry minor phrasing quirks.
• This is a processed release: the public set differs slightly from the original evaluation set used in the paper (see above).

License

This dataset is released under the Apache License 2.0.

The accompanying ArmorOCR model is built on Qwen/Qwen3-VL-8B-Instruct; any use of that model is additionally subject to the license and acceptable-use policy of the base model.

Because the images in this dataset may contain third-party material, the rights in such third-party content remain with their respective owners. The dataset is provided for research and benchmarking purposes. If you believe any item infringes your rights, please follow the residual-risk notice and we will remove it.

Citation

If you use AdvSpot or ArmorOCR, please cite the paper:

@misc{cao2026armorocrgroundedadversarialvisual,
      title={ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation},
      author={Linhan Cao and Siyuan Li and Jun Lan and Liangbo He and Guannan Li and Xiaolei Huang and Jun Jia and Shuheng Zhou and Huijia Zhu and Weiqiang Wang and Wei Sun},
      year={2026},
      eprint={2608.20122},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}