phishing-scam-detector-base-ONNX

This is an ONNX version of Horizon-Labs/phishing-scam-detector-base. It was automatically converted and uploaded using this Hugging Face Space.

Usage with Transformers.js

See the pipeline documentation for text-classification: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextClassificationPipeline

Phishing & Scam Detector (base, 307M): multilingual

Flags phishing, scams and spam in emails, SMS, chat and social-media messages in many languages. Three labels:

• fraud: phishing (fake bank, delivery, account or tax messages with links), scams (prizes, investment and crypto schemes, fake jobs, "hi mum, new number", romance, advance fee), requests for passwords, one-time codes or payment details, malware lures;
• spam: unwanted advertising and bulk marketing that is not trying to steal anything;
• legitimate: everything else, including genuine notifications that look similar (real one-time codes, delivery updates, bank alerts).

Built on mmBERT-base, Apache-2.0, ONNX files for CPU and the browser (transformers.js) included. Useful for message filtering and moderation, and for AI agents that read email or chats (see also our prompt-injection guard for attacks on the agent itself).

• Score: P(fraud) + P(spam) for "unwanted", or P(fraud) alone if marketing is acceptable. The default threshold is 0.5; raise it to cut false positives (see the false-positive rates below).
• onnx/model_quantized.onnx (int8 embeddings, 641 MB) gives the same top label as fp32 for 99.8% of 520 benchmark texts.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/phishing-scam-detector-base", top_k=None)
print(clf("Your parcel could not be delivered. Pay the 1.99 EUR customs fee within 24h: dhl-redelivery.example.com"))
# [[{'label': 'fraud', 'score': ...}, {'label': 'legitimate', 'score': ...}, {'label': 'spam', 'score': ...}]]

transformers.js:

import { pipeline } from "@huggingface/transformers";
const clf = await pipeline("text-classification", "Horizon-Labs/phishing-scam-detector-base", { dtype: "q8" });
console.log(await clf("Ihr Konto wurde gesperrt. Bestätigen Sie Ihre Daten: sparkasse-sicher.example.de", { top_k: null }));

Evaluation

Public spam / phishing / smishing test sets, used only for evaluation. Every model is scored the same way (code/): AUC of its "unwanted" probability (for this model P(fraud) + P(spam)), and F1 and false-positive rate at 0.5.

• Real smishing in other languages (the main test): Bengali / Banglish / code-mixed SMS (shariul-islam/bengali-sms-smishing-dataset, test split; smishing and promotions count as unwanted), Mozambican Portuguese SMS and Facebook messages (MOZNLP/MOZ-Smishing), Korean SMS (jmjmjm3/kor-smishing-message), up to 1,000 each.
• English emails: zefang-liu/phishing-email-dataset and SetFit/enron_spam test; English SMS: the UCI SMS Spam Collection; and its machine translation into 21 languages (dbarbedillo/SMS_Spam_Multilingual_Collection_Dataset). These classic sets are in or near the training data of several English baselines (marked in the note column), which explains their near-perfect English scores.

• The table shows the released checkpoint. Two training seeds: real-smishing AUC 0.916 / 0.925, false-positive rate 0.122 / 0.110.
• On real smishing in Bengali, Portuguese and Korean this model ranks messages much better (AUC) and flags far fewer normal messages than the English models (their false-positive rates there: 0.33-0.96). Its F1 at 0.5 is lower than roberta-spam's because it is conservative at 0.5: on the Portuguese set it catches only 32% of smishing messages at 0.5 (Bengali 95%, Korean 93%). Lower the threshold if missed scams cost more than false alarms.
• On the classic English sets it is slightly behind models trained on them.
• Machine-translated SMS by language (AUC): ar 0.952, bn 0.924, de 0.967, es 0.974, fr 0.975, hi 0.940, id 0.965, ja 0.949, jv 0.911, ko 0.937, mr 0.924, no 0.968, pa 0.881, pt 0.971, ru 0.958, sv 0.972, tr 0.962, uk 0.957, ur 0.947, zh 0.959.
• The spam / fraud split comes from our teacher; the public sets only distinguish wanted from unwanted, so only that is measured.

Per set:

Training

• Messages: 98,379 synthetic SMS, emails, chat, social-media and push messages in about 70 languages written by Qwen3.8-27B (Apache-2.0): per prompt 4 legitimate (half of them genuine notifications that resemble phishing), 2 spam and 4 fraud of one of 20 scam types (parcel fees, bank verification, tax refunds, prizes, fake jobs, crypto, romance, "new number", CEO fraud, fake login alerts, OTP theft, toll fines, ...). Plus 59,403 ordinary web snippets in 92 languages from FineWeb-2 / FineWeb (ODC-BY), so normal text stays normal. No test-set messages were used.
• Labels: Qwen3.8-27B (prompted) scored every text as legitimate / spam / fraud; the model learns the teacher's probabilities (soft labels). Checkpoint chosen on held-out teacher-labelled data.
• mmBERT-base, max length 384 tokens, learning rate 3e-5. Code: code/ in this repository.

Limitations

• Not a complete security product: it sees only the text, not sender reputation, links, attachments or headers. Use it as one signal among others, and review decisions that matter.
• False positives remain (about 11% of normal messages on the smishing test sets at 0.5), especially for genuine notifications with links and for promotions; tune the threshold on your own traffic.
• Trained on synthetic messages: new scam scripts, very long emails (truncated at 384 tokens) and rare languages are weaker. The Portuguese smishing set is the hardest here (AUC 0.840; at 0.5 it misses most of those messages).