multilingual-intent-router

Multilingual intent router (v1, shipped)

This is v1, the shipped model (branch main): a 12-class intent classifier for logistics chat messages, fine-tuned from intfloat/multilingual-e5-base, with an optional abstention flag for messages unlike the training data. A robustness variant (v3) is on branch robust-v3.

Model fingerprint (sha256 of fp32 state dict): 7ca22e16d2ace345d9dcf7737887e85fedf400a1dfc863678c2fc2663bf102bd

Label map

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "gauravgandhi2411/multilingual-intent-router"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
text = "query: " + "where is my shipment?"   # the E5 prefix is required
with torch.no_grad():
    logits = model(**tok(text, return_tensors="pt", truncation=True, max_length=64)).logits
print(model.config.id2label[int(logits.argmax(-1))])

Abstention (the out-of-distribution flag) is NOT part of the plain model: it needs predict.py and ood_bank.safetensors from the same repository, used as follows. Confidences are temperature-scaled (T = 1.0228); messages whose Mahalanobis score falls below the threshold (-128.06 v1 / -178.56 v3; this model's threshold = -128.06, set to retain 95.0% of validation rows) are flagged abstained.

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("gauravgandhi2411/multilingual-intent-router", revision="main")
sys.path.insert(0, path)
from predict import IntentRouter          # needs only torch, transformers, safetensors, numpy
router = IntentRouter.from_pretrained(path)
router.predict("...")   # {label, confidence, ood_score, abstained, top3}

Training data

A 500-row synthetic logistics intent dataset (12 classes, per-class counts 30 to 55) provided for an assessment. It is not redistributed and no dataset text is included. Fixed splits: 352 training rows, 74 validation rows (epoch, temperature and threshold selection), 74 test rows (one logged evaluation); model seed 42.

Intended and out-of-scope use

Intended: routing short, single-turn logistics chat messages (shipment status, analytics, disruptions, appointments, orders, documents, yard operations, support, knowledge-base, chit-chat) to one of the 12 intents; research and portfolio demonstration of the evaluation methodology.

Out of scope: safety or security controls; regulated or high-stakes decisions; languages other than English, Spanish, French, German and Chinese; messages longer than a short chat turn; relying on the abstention flag to catch new intents (it is a weak detector, see below); any claim of generalisation beyond a 500-row synthetic dataset.

Track A: closed-set classification

Test split, n = 74, one logged evaluation; 95% percentile-bootstrap intervals over rows.

Per-class test F1:

Track B: open-set rejection

Whole classes are held out (3 model seeds: 42, 43, 44; 62 known and 80 unknown evaluation rows). The threshold keeps 95% of known rows.

Open-set rejection is weak: most unseen-intent messages are not rejected at that operating point. A supervised oracle probe (it sees the held-out class labels, so it is an upper bound) reaches AUROC 0.969 on unseen-known rows against 0.870 for the unsupervised score used here.

v1 vs v3

Source: results/tradeoff_v1_v3.json (one logged test evaluation each; paired delta is v3 minus v1).

Limitations

• Weak open-set rejection (measured): strict rejection recall at 95% retention is 34.2% on the headline holdout (CI 26.2% to 42.5%). Do not rely on abstained to catch new intents.
• Identifier-prefix shortcut: the shipped model keys on identifier prefixes: the cross-validated neutral-swap flip rate is 0.238 (the robustness variant v3 reduces it to 0.052); on the test split the swaps flip 0.400 of 25 pairs.
• Small test set: n = 74; differences between v1 and v3 on Track A are inside the noise.
• Synthetic data: 500 synthetic rows from one source; real-world performance is unmeasured.
• Language coverage: 12.4% of messages have a non-English primary language, so non-English behaviour is thinly measured. Under machine translation the prediction agreement with the English prediction is 0.853 for v1 and 0.900 for v3 (synthetic translations, not native text).
• LLM baselines: only local 8B-class open-weight models (qwen3:8b, llama3.1:8b) were tested; no frontier or hosted LLM was evaluated.

Links

• Report: report.pdf
• GitHub: https://github.com/gaurav-gandhi-2411/multilingual-intent-router
• W&B report: https://wandb.ai/gauravgandhi429-gaurav-gandhi/multilingual-intent-router/reports/Submission-summary--VmlldzoxODA1MTA3Mw==
• Colab: notebook