HalluVision fine-tuned CyNER (xlm-roberta-base)
Token-classification checkpoint used to produce the reference entities in
Song, Lee, Kim, Choi, Kim. Measuring Hallucination in Large Language Models for Cyber
Threat Intelligence: An Exploratory Study. Computers and Electrical Engineering, 2026 (in press).
It is CyNER (Alam et al., 2022) fine-tuned on 50 MITRE
ATT&CK software articles annotated with five STIX-aligned labels. Every entity-based number
in the paper (answer-matching precision/recall, frequency-based entity similarity, the
Threatpost and The Hacker News ground truth, the Llama-3 summary NER) was produced by this
exact checkpoint, so it is released to make those numbers regenerable.
Labels
BIO tagging over five entity types:
| id |
tag |
|
id |
tag |
| 0 |
B-Malware |
6 |
B-Organization |
| 1 |
O |
7 |
I-Malware |
| 2 |
B-Indicator |
8 |
I-Organization |
| 3 |
I-Indicator |
9 |
B-Vulnerability |
| 4 |
B-System |
10 |
I-Vulnerability |
| 5 |
I-System |
Label definitions follow Table 3 of the paper (Malware, System, Indicator, Vulnerability,
Organization).
Training
Fine-tuned with the CyNER training loop (cyner.TransformersNER) from xlm-roberta-base.
Hyperparameters, verbatim from parameter.json:
| parameter |
value |
| dataset |
dataset_new (MITRE, CoNLL BIO; released as ner_finetune/ on Mendeley Data) |
| transformers_model |
xlm-roberta-base |
| random_seed |
1 |
| lr |
5e-6 |
| epochs |
30 |
| warmup_step |
0 |
| weight_decay |
1e-7 |
| batch_size |
32 |
| max_seq_length |
128 |
| fp16 |
false |
| max_grad_norm |
1 |
| lower_case |
false |
Held-out test split: precision 74.69, recall 77.84, F1 76.23 (paper Section 4.2).
A manual audit of the checkpoint on 70 Threatpost articles (204 entity judgements, two raters,
Cohen's kappa 0.726) is reported in Section 4.8 and released as ner_audit/ on Mendeley Data.
Vulnerability is the weakest label (strict precision 0.0 on the audited sample); see the paper
before relying on that label.
Usage
The paper's extraction recipe (sentence split, CyNER heuristics on, flair NER on, keep the
five target types, order-preserving dedup, drop spans of two characters or fewer) is
implemented in scripts/run_cyner_ner.py of the code release:
pip install git+https://github.com/aiforsec/CyNER nltk
huggingface-cli download Ricky-dev/halluvision-cyner-xlmr-base --local-dir cyner_ckpt
CYNER_CKPT=cyner_ckpt python scripts/run_cyner_ner.py \
--input llm_outputs/llm_results_gpt4o_mini_2503.json --text-key summary_text \
--output summary_ner_gpt4o_mini.json
Minimal direct use:
import cyner
model = cyner.CyNER(transformer_model="cyner_ckpt", use_heuristic=True, flair_model="ner")
for ent in model.get_entities("Evil Corp pivoted to LockBit ransomware after OFAC sanctions."):
print(ent.entity_type, ent.text)
Related resources
- Data: Mendeley Data, doi:10.17632/z8sfdnch2c (corpus, LLM outputs, reference entities,
per-document metrics, human validation labels, NER audit sheets, fine-tuning data).
- Code: https://github.com/SeungHo0422/HalluVision
License
MIT, matching CyNER and xlm-roberta-base. The MITRE fine-tuning data is derived from
publicly available MITRE ATT&CK pages.
Citation
@article{song2026halluvision,
title = {Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study},
author = {Song, Seungho and Lee, Donggyu and Kim, Jaehan and Choi, Jiyoung and Kim, Jinwoo},
journal = {Computers and Electrical Engineering},
year = {2026},
note = {in press}
}
