pii-skills-ablation

PII Skills: Dehydrated Evaluation Benchmark

Benchmark for Evaluating Tool-Augmented PII Detection in Small Language Models, a technical report (not peer reviewed).

Annotation coverage

The 2,000 benchmark rows are unchanged. All canonical label counts agree with the primary scoring gold after the report's normalization. The benchmark contains labels and source indices; it does not contain a complete span-annotated dataset. Earlier descriptions of this release as uniformly span-annotated were inaccurate.

There are 1,132 AI4Privacy, 581 Gretel and 287 NVIDIA examples. The scoring gold used for the historical report has spans for the AI4Privacy subset and type fallback for the other 868 examples. NVIDIA source annotations do have offsets; the historical hydration parser lost them when reading Python-representation strings as JSON. Corrected hydration code fixes that defect for future use, but it does not change which offsets existed in archived scoring.

The benchmark has 29 base categories and 27 after DATE/DATE_TIME and ADDRESS/LOCATION alignment. Sampling and source order constrain generalization; this is not a representative sample of all PII. Six AI4Privacy source examples contain text/offset-slice disagreements. Excluding them preserves the primary ranking in the analysis.

Fields and hydration

• id: stable sample ID.
• source: source dataset identifier.
• original_index: row index after the compilation's source filter. Apply English-language filtering for AI4Privacy and US-locale filtering for NVIDIA before looking up these indices. Gretel uses its supplied English dataset.
• ground_truth: JSON list of source type labels; not complete entity spans.
• pii_codex_ground_truth: JSON list of canonical type labels. The report applies additional explicitly documented alignment rules.

Raw input text is not included. Source datasets are AI4Privacy PII-Masking-300K, NVIDIA Nemotron-PII and Gretel PII Masking (English). Users must follow each source's licensing, access and citation terms. Full source revisions were not recorded in the historical release; hydration across a changed upstream dataset may not reproduce the original text.

config/ and config/prompts/v1/ preserve historical configuration and prompt evidence. They are not rewritten to imply corrected code generated the archived predictions. See the research repository for repaired preparation and analysis code, and the results repository for corrected scores and explicit scoring gold.

The observed comparison concerns retained parsed outputs of specific pipelines. It does not isolate a causal capability effect or establish deployment safety.

Citations

If you use this benchmark or config, please cite the paper and, where applicable, the dataset:

@article{rosado2026context,
  title   = {PII-Codex–Aligned Agent Skills Ablation Benchmark},
  author  = {Rosado, Eidan J.},
  year    = {2026},
  note    = {Benchmark: \url{https://huggingface.co/datasets/EdyVision/pii-skills-ablation}}
}

Source Dataset Citations

This benchmark compiles and aligns samples from the following datasets. If you use this benchmark, please also cite the original datasets:

AI4Privacy PII-Masking-300K

@misc{ai4privacy_2024,
    author       = { Ai4Privacy },
    title        = { pii-masking-300k (Revision 86db63b) },
    year         = 2024,
    url          = { https://huggingface.co/datasets/ai4privacy/pii-masking-300k },
    doi          = { 10.57967/hf/1995 },
    publisher    = { Hugging Face }
}

NVIDIA Nemotron-PII

@dataset{nemotron-pii,
  author = {Amy Steier and Andre Manoel and Alexa Haushalter and Maarten Van Segbroeck},
  title = {Nemotron-PII: Synthesized Data for Privacy-Preserving AI},
  year = {2025},
  publisher = {NVIDIA},
  url = {https://huggingface.co/datasets/nvidia/Nemotron-PII}
}

Gretel PII Masking (English)

@dataset{gretel-pii-docs-en-v1,
  author       = {Gretel AI},
  title        = {GLiNER Models for PII Detection through Fine-Tuning on Gretel-Generated Synthetic Documents},
  year         = {2024},
  month        = {10},
  publisher    = {Gretel},
}

Limitations

• Ground truth is derived from source datasets and aligned to PII-Codex; alignment choices can affect edge cases and rare types.
• The benchmark is sampled for experimental control (stratification by PII types and text length); it is not a general-purpose PII benchmark for all domains.
• Prompt and config versions (e.g. v1) are fixed for the reported study; other versions may be added later.

License

See the dataset repository and the linked paper/code repository for license terms. Source datasets (AI4Privacy, NVIDIA, Gretel) have their own licenses; compilers must comply with those when redistributing or deriving data.

Related Resources

• Scored results (predictions and metrics): EdyVision/pii-skills-ablation-results — splits for the main and pilot ablations plus the few-shot/chain-of-thought baselines, standalone detector, 14B scale control, and full-precision control.
• Code and notebooks: EdyVision/pii-skills-ablation-study

Technical-report citation

@techreport{rosado2026context,
  title = {Evaluating Tool-Augmented PII Detection in Small Language Models},
  author = {Rosado, Eidan J.},
  year = {2026},
  note = {Technical report; not peer reviewed}
}