JEV-like email spam classifier (prototype)
A fine-tuned DistilBERT sequence classifier for binary email triage: legitimate and spam. It is a prototype trained and evaluated on a historical SpamAssassin corpus. It is not a phishing or malware detector and should not be treated as a complete email security system.
Intended use
Use as an experimental text-only aid to sort historical or otherwise appropriately reviewed email into ham/spam queues. Route uncertain, empty, or unsupported inputs to human review. Do not use the output as the sole basis for blocking senders, taking account actions, or making other consequential decisions.
Out of scope
This model does not authenticate senders, inspect attachments, resolve or visit links, verify SPF/DKIM/DMARC, or detect all phishing and malware. It can miss new campaigns, adversarial examples, non-English messages, and content unlike its training data. Confidence scores are model estimates, not calibrated probabilities or proof of safety.
Model
- Architecture: DistilBERT sequence classification (distilbert/distilbert-base-uncased)
- Task: binary sequence classification
- Labels: 0 = legitimate, 1 = spam
- Input: normalized subject/header/body text and URL strings; maximum sequence length 512 tokens
- Output: label scores and a routing action; low confidence or empty input may be sent to review
- Fine-tuning dataset: Apache SpamAssassin public corpus (easy_ham, hard_ham, spam, spam_2), mapped to ham/spam
- Training data size after exact duplicate removal: 4,675 messages (2,781 legitimate; 1,894 spam)
- Base model revision: 12040accade4e8a0f71eabdb258fecc2e7e948be
Evaluation
Held-out group-stratified test split from the same historical corpus (n=890). These results describe this split only; they do not establish current real-world effectiveness or performance on independent sources.
| Label |
Precision |
Recall |
F1 |
Support |
| legitimate |
0.989 |
0.989 |
0.989 |
528 |
| spam |
0.983 |
0.983 |
0.983 |
362 |
| macro average |
0.986 |
0.986 |
0.986 |
890 |
Accuracy: 0.987. Spam PR-AUC: 0.998. The test split had 6 spam false negatives. The confidence threshold in the training artifacts is a prototype policy choice, not a guarantee of calibration.
Quick start
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "akumar0205/phishing_spam_jev_like_classifier"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
text = "Subject: Example subject\nBody: Example email text"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
scores = model(**inputs).logits.softmax(dim=-1)[0]
print({"legitimate": scores[0].item(), "spam": scores[1].item()})
For the notebook's {label, scores, action} policy interface, see inference.py.
Data and attribution
The model was fine-tuned using the SpamAssassin public corpus. The corpus documentation states that copyright for message text remains with the original senders. See ATTRIBUTION.md and the linked corpus documentation for source details. No raw email messages are included in this repository.
License
This repository declares Apache-2.0 for the model artifact. See the repository license metadata and base model attribution. Dataset terms and original message copyrights are described separately in ATTRIBUTION.md.
Limitations and risks
This is a research prototype. The corpus is old and may not represent modern mail, languages, or threat patterns. Data-source artifacts can make held-out results optimistic. Do not deploy as an autonomous security control; evaluate on independently sourced, appropriately authorized, current data and use human review.
