pii-skills-ablation-results

PII Skills: Corrected Archived Predictions and Scores

Results accompanying Evaluating Tool-Augmented PII Detection in Small Language Models, a technical report (not peer reviewed).

Scoring and data notes

The current files replace ambiguous historical score fields with an explicit report scoring contract. No model inference was rerun. All 66,000 prediction records remain; labels, offsets and record identities are preserved from the prior Hub release. Detector prediction order is preserved as published. Entity text is redacted, including text previously present in the detector split. Prior repository revisions remain historical and are superseded for evaluation.

The standalone detector previously lacked fallback gold for 868 examples in local scoring. Correct gold is now included by joining the primary run by sample ID. Its corrected mean type F1 is .502178 and mixed span/type F1 is .453214. This removes the former large model-plus-tool advantage. Primary means remain:

See CORRECTIONS.json for record counts, score changes and file hashes. Corrections to diagnostics, aggregation and uncertainty are documented in the research repository.

Scoring schema

One row represents (sample_id, model, condition). All six existing splits are retained: main_v1 (32,000), pilot_v1 (3,200), baselines_v1 (16,000), detector_v1 (2,000), fp16_v1 (4,800), scaling_v1 (8,000). The v1 suffix identifies historical prompting; scoring_version identifies the corrected analysis independently.

• predictions: JSON entity list with original labels and offsets; text redacted.
• ground_truth: JSON scoring gold from the archived local runs; detector gold joined from the primary run. Text redacted.
• scores: JSON precision, recall and F1 for aligned type matching. These values supersede the older generic scores values, which used different conventions. Type matching measures category/count agreement, not correct localization or entity text.
• mixed_span_type_f1: F1 requiring same aligned type and span IoU >= .5 where gold offsets exist, with type fallback otherwise. The primary benchmark has 1,132 span-bearing examples and 868 type-only examples in historical scoring. This is not pure span F1.
• scoring_version: report_v0.3_aligned_type_and_mixed.
• gold_provenance: archived local scoring gold, or primary gold joined by sample ID for the detector.
• Other columns retain source, model, condition, run/prompt identifiers, invocation flags, error, elapsed time and turn count. Invocation flags do not prove successful calls or equivalent document exposure. Batched elapsed time is amortized, not isolated latency.

Apply the same normalization to both sides. The exact contract is lib/ablation_harness/scoring/saved_analysis.py in the code repository. Average row F1 over examples within a model/condition; average model means for the four-model aggregate. Repeated conditions do not provide independent samples for pooled confidence intervals.

Limits

These are archived parsed predictions, not raw model responses. Historical parser failures cannot be counted or repaired because raw responses were not saved. Context delivery differed across model adapters, so this is a comparison of implemented pipelines, not an isolated causal effect of tools or skills. NVIDIA source offsets were lost during historical hydration; those missing historical offsets are not retrospectively added to change the report's evaluation contract. Six source examples have text/offset disagreements; excluding them preserves the primary ranking. Model revisions and full generation provenance were not archived.

The benchmark contains sample indices and labels, not source text. Comply with its source dataset licenses and terms. Redaction does not establish deployment safety. Historical revisions may contain unredacted detector entity text; use the corrected current release.

Technical-report citation

@techreport{rosado2026context,
  title = {Evaluating Tool-Augmented PII Detection in Small Language Models},
  author = {Rosado, Eidan J.},
  year = {2026},
  note = {Technical report; not peer reviewed}
}