multilingual-zeroshot-base

Multilingual Zero-Shot Classifier (base, 308M)

Classify text in 30+ languages into any labels you choose, with no training. Use it with the transformers zero-shot-classification pipeline, like facebook/bart-large-mnli, but multilingual, smaller, and with an 8k-token context window (fine-tuned at up to 1,024 tokens).

• Multilingual: the text can be in any of the languages below; labels and the hypothesis template stay in English.
• Commercially clean: Apache-2.0, trained only on data that allows commercial use (no XNLI, ANLI or other non-commercial sets). See Training.
• Small and fast: 308M parameters, ModernBERT architecture (mmBERT), ONNX included for CPU and the browser.
• Honest numbers: all models below were run by us with the same script and templates.

Try it in the browser: Horizon-Labs/multilingual-zeroshot demo.

Part of Horizon Labs' open models (collection). Source code: github.com/horizon-ai-labs/agent-io-guards.

Quick start

from transformers import pipeline

clf = pipeline("zero-shot-classification", model="Horizon-Labs/multilingual-zeroshot-base")
clf("Mi pedido llegó roto y quiero que me devuelvan el dinero.",
    candidate_labels=["refund request", "shipping question", "product praise", "account problem"])
# {'labels': ['refund request', ...], 'scores': [...]}

# several labels can apply at once
clf("The camera is great but the battery dies by noon.", ["camera", "battery", "screen", "price"], multi_label=True)

# a task-specific template often helps
clf("¿Me pones una alarma a las siete?", ["set an alarm", "play music", "weather"], hypothesis_template="The user wants to {}.")

Labels: not_entailment (0) and entailment (1). For each candidate label the model scores whether the text entails "This example is {label}." (or your hypothesis_template). Any NLI-style use works too: pass text and text_pair to a text-classification pipeline.

Evaluation

Accuracy, single-label (multi_label=False: the label with the highest entailment score wins). English templates and labels for every language; the same template for every model (e.g. "This text is about {}." for SIB-200). No model saw these datasets' training splits, except where marked. ‡ = trained partly on data with non-commercial licenses (their -c variants are the commercially usable ones). Script: zeroshot/evaluate_zs.py.

§ = label names seen in our synthetic training data. Since v1.1 our training data includes generic label taxonomies (topics, news sections, Q&A question topics, emotions, sentiment) whose label names overlap these benchmarks' label sets; for Yahoo Answers and AG News almost exactly. No benchmark texts were used, but on these rows our models are not zero-shot with respect to the label names, so compare with care. MASSIVE, Banking77 and XNLI label sets were not used.

Multilingual

Per language, mean of MASSIVE and SIB-200:

English

v1.0 → v1.1

v1.1 adds data with broad, reusable label taxonomies, so it is better on common categories (topics, emotions, sentiment, aspects) — the § rows, where the label names are familiar to it. On label sets it has not seen it stays within about ±0.015 of v1.0 (slightly lower on some). To pin the previous model, load it with revision="v1.0".

NLI

† XNLI is included for reference only: xlm-roberta-large-xnli and mDeBERTa-xnli were trained on XNLI data (the first scores 0.99, which suggests it saw the test sentences). Our models never saw XNLI.

Limitations

• English-only models trained with more (partly non-commercial) classification data are better on English topic and emotion benchmarks (e.g. deberta-v3-base-zeroshot-v2.0 on Emotion and Yahoo). If you only need English, compare them on your data.
• bge-m3-zeroshot-v2.0 (568M, trained partly on non-commercial data) scores higher on MASSIVE and SIB-200.
• Zero-shot accuracy depends a lot on label wording and the template. Use descriptive labels ("request a refund" rather than "refund_req") and try a template that fits your task. The model links explicit wording better than implied categories. Example (small model, multi-label, a gym review not like our training domains): "The machines are always taken after 5pm and half the treadmills are broken, but the coaches really know their stuff. For 60 euros a month I expected cleaner showers." gives equipment 0.99, trainers 0.97, membership cost 0.82, but hygiene only 0.28 and crowding 0.03 (v1.0: trainers 0.56, membership cost 0.58).
• Broad labels (e.g. "world news", "education") tend to win over specific ones. Emotions close in meaning (joy / love / surprise) are often confused.
• With multi_label=True, scores are independent; tune the threshold on a few examples of your own.
• Lower-resource languages (e.g. Swahili) score clearly lower than high-resource ones.
• Much of the training data is synthetic (Qwen3.8-27B) or machine-translated.

Training

• Backbone: jhu-clsp/mmBERT-base (MIT), sequence-pair classification, bf16, max length 1024.
• Data (label = does the text entail the hypothesis):
• English NLI: MultiNLI (OANC and CC-BY-SA-3.0 parts), SNLI (CC-BY-SA-4.0), WANLI (CC-BY-4.0).
• 120k MultiNLI/WANLI pairs machine-translated by Qwen3.8-27B into 24 languages, with native and English hypotheses.
• Synthetic zero-shot tasks by Qwen3.8-27B: FineWeb-Edu / FineWeb-2 passages (ODC-BY) labelled by topic, genre, audience, tone and purpose with near-miss wrong labels, and ~90k short texts (requests, reviews, tickets, posts, headlines) over 26 task types, 32 domains and 33 languages, each with an invented label set and hypothesis template.
• (v1.1) Generic taxonomies by Qwen3.8-27B: 24k new FineWeb / FineWeb-2 passages labelled for topic, text type, sentiment, audience, purpose and news section; ~130k short texts written for fixed label sets (emotion, sentiment, Q&A question topic, news section, customer-message topic, urgency, formality, spam) without using the label words; ~25k reviews in 8 domains mentioning aspects (e.g. "internet", "food") without naming them.
• Not used: XNLI, ANLI, FEVER-NLI, any benchmark above.