SAVRN
Search Contact SAVRN

Open-weight model · Zero-shot classification

deberta-v3-xsmall-zeroshot-v1.1-all-33

by Moritz Borrett-Laurer (formerly Laurer) MoritzLaurer/deberta-v3-xsmall-zeroshot-v1.1-all-33

This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper. The foundation model is microsoft/deberta-v3-xsmall.

Parameters71M
Context512
Weights654.9 MB
Licensemit
AccessOpen weights
Monthly Downloads65.9k

Runs On

What it takes to serve deberta-v3-xsmall-zeroshot-v1.1-all-33 (71M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.1 GB 0.2 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.1 GB 0.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.0 GB 0.0 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Moritz Borrett-Laurer (formerly Laurer), published under mit, revision 262ae02f2917.

This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper. The foundation model is microsoft/deberta-v3-xsmall. The model only has 22 million backbone parameters and 128 million vocabulary parameters. The backbone parameters are the main parameters active during inference, providing a significant speedup over larger models. The model is 142 MB small. This model was trained to provide a small and highly efficient zeroshot option, especially for edge devices or in-browser use-cases with transformers.js. For usage instructions and other details refer to this model card…

Read Moritz Borrett-Laurer (formerly Laurer)'s full model card

This model was fine-tuned using the same pipeline as described in the model card for MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and in this paper.

The foundation model is microsoft/deberta-v3-xsmall. The model only has 22 million backbone parameters and 128 million vocabulary parameters. The backbone parameters are the main parameters active during inference, providing a significant speedup over larger models. The model is 142 MB small.

This model was trained to provide a small and highly efficient zeroshot option, especially for edge devices or in-browser use-cases with transformers.js.

Usage and other details

For usage instructions and other details refer to this model card MoritzLaurer/deberta-v3-large-zeroshot-v1.1-all-33 and this paper.

Metrics:

I didn't not do zeroshot evaluation for this model to save time and compute. The table below shows standard accuracy for all datasets the model was trained on (note that the NLI datasets are binary).

General takeaway: the model is much more efficient than its larger sisters, but it performs less well.

Datasets mnli_m mnli_mm fevernli anli_r1 anli_r2 anli_r3 wanli lingnli wellformedquery rottentomatoes amazonpolarity imdb yelpreviews hatexplain massive banking77 emotiondair emocontext empathetic agnews yahootopics biasframes_sex biasframes_offensive biasframes_intent financialphrasebank appreviews hateoffensive trueteacher spam wikitoxic_toxicaggregated wikitoxic_obscene wikitoxic_identityhate wikitoxic_threat wikitoxic_insult manifesto capsotu
Accuracy 0.925 0.923 0.886 0.732 0.633 0.661 0.814 0.887 0.722 0.872 0.944 0.925 0.967 0.774 0.734 0.627 0.762 0.745 0.465 0.888 0.702 0.94 0.853 0.863 0.914 0.926 0.921 0.635 0.968 0.897 0.918 0.915 0.935 0.9 0.505 0.701
Inference text/sec (A10G, batch=128) 1573.0 1630.0 683.0 1282.0 1352.0 1072.0 2325.0 2008.0 4781.0 2743.0 677.0 228.0 238.0 2357.0 5027.0 4323.0 3247.0 3129.0 941.0 1643.0 335.0 1517.0 1452.0 1498.0 2367.0 974.0 2634.0 353.0 2284.0 260.0 252.0 256.0 254.0 259.0 1941.0 2080.0

Configuration

Architecture
DebertaV2ForSequenceClassification
Context length (tokens)
512
Layers
12
Hidden size
384
Feed-forward size
1,536
Attention heads
6
Vocabulary size
128,100
Stored precision
float16
Model type
deberta-v2

Identity and Version

Repository
MoritzLaurer/deberta-v3-xsmall-zeroshot-v1.1-all-33
Publisher
Moritz Borrett-Laurer (formerly Laurer)
Task
Zero-shot classification
Modality
Text
Library
transformers
Parameters
71M parameters
Languages
en
Revision
262ae02f29173eec1c250f90804dc7edc677dcff
First published
2024-01-10
Last updated
2024-05-07

Files and Weights

13 files, 666.0 MB in total. The weights are 5 files totalling 654.9 MB in bin, onnx, safetensors.

Weights5 files · 654.9 MB
Configuration3 files · 1.2 KB
Tokenizer2 files · 8.7 MB
Documentation1 file · 2.9 KB
Other1 file · 2.5 MB
Repository1 file · 1.5 KB
Every file
FileTypeSizeSHA-256
model.safetensorsWeights141.7 MB 56b89067a4e6
onnx/model.onnxWeights284.2 MB ee238235df56
onnx/model_quantized.onnxWeights87.2 MB 1fbf78872cea
pytorch_model.binWeights141.7 MB 3567df375819
training_args.binWeights4.7 KB 9421725264eb
added_tokens.jsonConfiguration23 B —
config.jsonConfiguration1.0 KB —
special_tokens_map.jsonConfiguration173 B —
README.mdDocumentation2.9 KB —
spm.modelOther2.5 MB c679fbf93643
.gitattributesRepository1.5 KB —
tokenizer.jsonTokenizer8.7 MB —
tokenizer_config.jsonTokenizer384 B —

License and Download

License
mit
Access
Open weights, no gate
Download size
654.9 MB
Download from Moritz Borrett-Laurer (formerly Laurer)

Released by Moritz Borrett-Laurer (formerly Laurer) through its official repository on Hugging Face. Read the license.

Built From

  • Derived from microsoft/deberta-v3-xsmall
  • Described by arXiv:2312.17543
  • Quantized from microsoft/deberta-v3-xsmall

Memory Requirements

PrecisionWeights in memory
As published654.9 MB
16-bit0.1 GB
8-bit0.1 GB
4-bit0.0 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About deberta-v3-xsmall-zeroshot-v1.1-all-33

How much GPU memory does deberta-v3-xsmall-zeroshot-v1.1-all-33 need?

About 0.2 GB at 16-bit and 0 GB at 4-bit: the weights (71M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run deberta-v3-xsmall-zeroshot-v1.1-all-33 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use deberta-v3-xsmall-zeroshot-v1.1-all-33 commercially?

Yes. deberta-v3-xsmall-zeroshot-v1.1-all-33 is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

What is deberta-v3-xsmall-zeroshot-v1.1-all-33's context length?

512 tokens, from the maximum position embeddings in its published configuration.

Similar Models

This model was trained on 782 357 hypothesis-premise pairs from 4 NLI datasets: MultiNLI, Fever-NLI, LingNLI and ANLI. Note that the model was trained on binary NLI to predict either "entailment" or "not-entailment". This is specifically designed for zero-shot classification, where the difference between "neutral" and "contradiction" is irrelevant. The base model is DeBERTa-v3-xsmall from Microsoft. The v3 variant of DeBERTa substantially outperforms previous versions of the model by including a different pre-training objective, see the DeBERTa-V3 paper. For highest performance (but less speed), I recommend using…

Open weights mit 71M parameters 512 tokens transformers
View model

This model was trained using SentenceTransformers Cross-Encoder class. This model is based on microsoft/deberta-v3-xsmall The model was trained on the SNLI and MultiNLI datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral. For futher evaluation results, see SBERT.net - Pretrained Cross-Encoder. Pre-trained models can be used like this: You can use the model also directly with Transformers library (without SentenceTransformers library): This model can also be used for zero-shot-classification

Open weights apache-2.0 71M parameters 512 tokens sentence-transformers
View model

Model · Zero-shot classification

distilbert-base-uncased-mnli

Typeform

This is the uncased DistilBERT model fine-tuned on Multi-Genre Natural Language Inference (MNLI) dataset for the zero-shot classification task. This model can be used for text classification tasks. CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes. Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). This model of DistilBERT-uncased is pretrained on the Multi-Genre Natural Language Inference (MultiNLI) corpus. It is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment…

Open weights 67M parameters 512 tokens transformers
View model

This model was trained using SentenceTransformers Cross-Encoder class. The model was trained on the SNLI and MultiNLI datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral. For evaluation results, see SBERT.net - Pretrained Cross-Encoder. Pre-trained models can be used like this: You can use the model also directly with Transformers library (without SentenceTransformers library): This model can also be used for zero-shot-classification

Open weights apache-2.0 82M parameters 514 tokens sentence-transformers
View model

This model was trained using SentenceTransformers Cross-Encoder class. The model was trained on the SNLI and MultiNLI datasets. For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral. For evaluation results, see SBERT.net - Pretrained Cross-Encoder. Pre-trained models can be used like this: You can use the model also directly with Transformers library (without SentenceTransformers library): This model can also be used for zero-shot-classification

Open weights apache-2.0 82M parameters 514 tokens sentence-transformers
View model

multilingual - zero-shot-classification - text-classification - nli - pytorch - accuracy - multinli - xnli pipelinetag: zero-shot-classification candidatelabels: "politics, economy, entertainment, environment" This multilingual model can perform natural language inference (NLI) on 100+ languages and is therefore also suitable for multilingual zero-shot classification. The underlying multilingual-MiniLM-L6 model was created by Microsoft and was distilled from XLM-RoBERTa-large (see details in the original paper and newer information in this repo). The model was then fine-tuned on the XNLI dataset, which contains hypothesis-premise pairs from 15 languages, as well as the English MNLI dataset.…

Open weights mit 107M parameters 514 tokens transformers
View model