This model is a conversion of MoritzLaurer/roberta-base-zeroshot-v2.0-c to ONNX format using the Optimum library.
Open-weight model · Zero-shot classification
xtremedistil-l6-h256-zeroshot-v1.1-all-33
by Moritz Borrett-Laurer (formerly Laurer) MoritzLaurer/xtremedistil-l6-h256-zeroshot-v1.1-all-33
This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper. The foundation model is microsoft/xtremedistil-l6-h256-uncased.
Parameters13M
Context512
Weights115.3 MB
Licensemit
AccessOpen weights
Monthly Downloads7.8k
Runs On
What it takes to serve xtremedistil-l6-h256-zeroshot-v1.1-all-33 (13M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 0.0 GB | 0.0 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 0.0 GB | 0.0 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.0 GB | 0.0 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.
Model Card
By Moritz Borrett-Laurer (formerly Laurer), published under mit, revision c07f66d9cbf7.
This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper. The foundation model is microsoft/xtremedistil-l6-h256-uncased. The model only has 22 million backbone parameters and 30 million vocabulary parameters. The backbone parameters are the main parameters active during inference, providing a significant speedup over larger models. The model is 25 MB small. This model was trained to provide a very small and highly efficient zeroshot option, especially for edge devices or in-browser use-cases with transformers.js. For usage instructions and other details refer to this model card…
Read Moritz Borrett-Laurer (formerly Laurer)'s full model card
This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper.
The foundation model is microsoft/xtremedistil-l6-h256-uncased. The model only has 22 million backbone parameters and 30 million vocabulary parameters. The backbone parameters are the main parameters active during inference, providing a significant speedup over larger models. The model is 25 MB small.
This model was trained to provide a very small and highly efficient zeroshot option, especially for edge devices or in-browser use-cases with transformers.js.
Usage and other details
For usage instructions and other details refer to this model card MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and this paper.
Metrics:
I didn't not do zeroshot evaluation for this model to save time and compute. The table below shows standard accuracy for all datasets the model was trained on (note that the NLI datasets are binary).
General takeaway: the model is much more efficient than its larger sisters, but it performs less well.
| Datasets | mnli_m | mnli_mm | fevernli | anli_r1 | anli_r2 | anli_r3 | wanli | lingnli | wellformedquery | rottentomatoes | amazonpolarity | imdb | yelpreviews | hatexplain | massive | banking77 | emotiondair | emocontext | empathetic | agnews | yahootopics | biasframes_sex | biasframes_offensive | biasframes_intent | financialphrasebank | appreviews | hateoffensive | trueteacher | spam | wikitoxic_toxicaggregated | wikitoxic_obscene | wikitoxic_identityhate | wikitoxic_threat | wikitoxic_insult | manifesto | capsotu |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy | 0.894 | 0.895 | 0.854 | 0.629 | 0.582 | 0.618 | 0.772 | 0.826 | 0.684 | 0.794 | 0.91 | 0.879 | 0.935 | 0.676 | 0.651 | 0.521 | 0.654 | 0.707 | 0.369 | 0.858 | 0.649 | 0.876 | 0.836 | 0.839 | 0.849 | 0.892 | 0.894 | 0.525 | 0.976 | 0.88 | 0.901 | 0.874 | 0.903 | 0.886 | 0.433 | 0.619 |
| Inference text/sec (A10G GPU, batch=128) | 4117.0 | 4093.0 | 1935.0 | 2984.0 | 3094.0 | 2683.0 | 5788.0 | 4926.0 | 9701.0 | 6359.0 | 1843.0 | 692.0 | 756.0 | 5561.0 | 10172.0 | 9070.0 | 7511.0 | 7480.0 | 2256.0 | 3942.0 | 1020.0 | 4362.0 | 4034.0 | 4185.0 | 5449.0 | 2606.0 | 6343.0 | 931.0 | 5550.0 | 864.0 | 839.0 | 837.0 | 832.0 | 857.0 | 4418.0 | 4845.0 |
Configuration
- Architecture
- BertForSequenceClassification
- Context length (tokens)
- 512
- Layers
- 6
- Hidden size
- 256
- Feed-forward size
- 1,024
- Attention heads
- 8
- Vocabulary size
- 30,522
- Stored precision
- float16
- Model type
- bert
Identity and Version
- Repository
- MoritzLaurer/xtremedistil-l6-h256-zeroshot-v1.1-all-33
- Publisher
- Moritz Borrett-Laurer (formerly Laurer)
- Task
- Zero-shot classification
- Modality
- Text
- Library
- transformers
- Parameters
- 13M parameters
- Languages
- en
- Revision
- c07f66d9cbf781191bee66edfe8ad7856f045781
- First published
- 2024-01-10
- Last updated
- 2025-01-16
Files and Weights
12 files, 116.3 MB in total. The weights are 5 files totalling 115.3 MB in bin, onnx, safetensors.
Weights5 files · 115.3 MB
Configuration2 files · 1.0 KB
Tokenizer3 files · 943.3 KB
Documentation1 file · 2.9 KB
Repository1 file · 1.5 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 25.5 MB | bf2d83426aa6 |
| onnx/model.onnx | Weights | 51.1 MB | 89ca1f968301 |
| onnx/model_quantized.onnx | Weights | 13.1 MB | 534df3a88f34 |
| pytorch_model.bin | Weights | 25.5 MB | 98b31cbc8fab |
| training_args.bin | Weights | 4.7 KB | e9fdf5b1ada2 |
| config.json | Configuration | 882 B | — |
| special_tokens_map.json | Configuration | 125 B | — |
| README.md | Documentation | 2.9 KB | — |
| .gitattributes | Repository | 1.5 KB | — |
| tokenizer.json | Tokenizer | 711.4 KB | — |
| tokenizer_config.json | Tokenizer | 366 B | — |
| vocab.txt | Tokenizer | 231.5 KB | — |
License and Download
- License
- mit
- Access
- Open weights, no gate
- Download size
- 115.3 MB
Download from Moritz Borrett-Laurer (formerly Laurer)
Released by Moritz Borrett-Laurer (formerly Laurer) through its official repository on Hugging Face. Read the license.
Built From
- Derived from microsoft/xtremedistil-l6-h256-uncased
- Described by arXiv:2312.17543
- Quantized from microsoft/xtremedistil-l6-h256-uncased
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 115.3 MB |
| 16-bit | 0.0 GB |
| 8-bit | 0.0 GB |
| 4-bit | 0.0 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About xtremedistil-l6-h256-zeroshot-v1.1-all-33
How much GPU memory does xtremedistil-l6-h256-zeroshot-v1.1-all-33 need?
About 0 GB at 16-bit and 0 GB at 4-bit: the weights (13M parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run xtremedistil-l6-h256-zeroshot-v1.1-all-33 on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use xtremedistil-l6-h256-zeroshot-v1.1-all-33 commercially?
Yes. xtremedistil-l6-h256-zeroshot-v1.1-all-33 is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.
What is xtremedistil-l6-h256-zeroshot-v1.1-all-33's context length?
512 tokens, from the maximum position embeddings in its published configuration.
Similar Models
https://huggingface.co/facebook/bart-large-mnli with ONNX weights to be compatible with Transformers.js. If you haven't already, you can install the Transformers.js JavaScript library from NPM using: Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).
distilbart-mnli is the distilled version of bart-large-mnli created using the No Teacher Distillation technique proposed for BART summarisation by Huggingface, here. We just copy alternating layers from bart-large-mnli and finetune more on the same data. This is a very simple and effective technique, as we can see the performance drop is very little. Detailed performace trade-offs will be posted in this sheet. If you want to train these models yourself, clone the distillbart-mnli repo and follow the steps below Clone and install transformers from source Download MNLI data Create student model Start fine-tuning You can find the logs of these trained models in this wandb project.
distilbart-mnli is the distilled version of bart-large-mnli created using the No Teacher Distillation technique proposed for BART summarisation by Huggingface, here. We just copy alternating layers from bart-large-mnli and finetune more on the same data. This is a very simple and effective technique, as we can see the performance drop is very little. Detailed performace trade-offs will be posted in this sheet. If you want to train these models yourself, clone the distillbart-mnli repo and follow the steps below Clone and install transformers from source Download MNLI data Create student model Start fine-tuning You can find the logs of these trained models in this wandb project.
https://huggingface.co/typeform/mobilebert-uncased-mnli with ONNX weights to be compatible with Transformers.js. If you haven't already, you can install the Transformers.js JavaScript library from NPM using: Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).
https://huggingface.co/cross-encoder/nli-deberta-v3-xsmall with ONNX weights to be compatible with Transformers.js. If you haven't already, you can install the Transformers.js JavaScript library from NPM using: Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).