SAVRN
Search Contact SAVRN

Open-weight model · Question answering

mdeberta-v3-base-squad2

by Tim timpal0l/mdeberta-v3-base-squad2

It has been finetuned for 3 epochs on SQuAD2.0. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.

Parameters278M
Context512
Weights2.2 GB
Licensemit
AccessOpen weights
Monthly Downloads47.8k

Runs On

What it takes to serve mdeberta-v3-base-squad2 (278M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.6 GB 0.7 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.3 GB 0.3 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.1 GB 0.2 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Tim, published under mit, revision 08d6e89c7a65.

This model can be used for Extractive QA

It has been finetuned for 3 epochs on SQuAD2.0.

Usage

from transformers import pipeline

qa_model = pipeline("question-answering", "timpal0l/mdeberta-v3-base-squad2")
question = "Where do I live?"
context = "My name is Tim and I live in Sweden."
qa_model(question = question, context = context)
# {'score': 0.975547730922699, 'start': 28, 'end': 36, 'answer': ' Sweden.'}

Evaluation on SQuAD2.0 dev set

{
    "epoch": 3.0,
    "eval_HasAns_exact": 79.65587044534414,
    "eval_HasAns_f1": 85.91387795001529,
    "eval_HasAns_total": 5928,
    "eval_NoAns_exact": 82.10260723296888,
    "eval_NoAns_f1": 82.10260723296888,
    "eval_NoAns_total": 5945,
    "eval_best_exact": 80.8809904826076,
    "eval_best_exact_thresh": 0.0,
    "eval_best_f1": 84.00551406448994,
    "eval_best_f1_thresh": 0.0,
    "eval_exact": 80.8809904826076,
    "eval_f1": 84.00551406449004,
    "eval_samples": 12508,
    "eval_total": 11873,
    "train_loss": 0.7729689576483615,
    "train_runtime": 9118.953,
    "train_samples": 134891,
    "train_samples_per_second": 44.377,
    "train_steps_per_second": 0.925
}

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.

Read the full model card (272 words)

Configuration

Architecture
DebertaV2ForQuestionAnswering
Context length (tokens)
512
Layers
12
Hidden size
768
Feed-forward size
3,072
Attention heads
12
Vocabulary size
251,000
Stored precision
float32
Model type
deberta-v2

Identity and Version

Repository
timpal0l/mdeberta-v3-base-squad2
Publisher
Tim
Task
Question answering
Modality
Text
Library
transformers
Parameters
278M parameters
Languages
af, am, ar, as, az, be, bg, bn
Revision
08d6e89c7a6557f967db2e1021f7f640483400ed
First published
2022-11-30
Last updated
2024-11-26

Files and Weights

9 files, 2.2 GB in total. The weights are 2 files totalling 2.2 GB in bin, safetensors.

Weights2 files · 2.2 GB
Configuration3 files · 1.1 KB
Tokenizer2 files · 16.3 MB
Documentation1 file · 3.2 KB
Repository1 file · 1.5 KB
Every file
FileTypeSizeSHA-256
model.safetensorsWeights1.1 GB 16675f4d0b1d
pytorch_model.binWeights1.1 GB 91d05e57e35a
added_tokens.jsonConfiguration23 B —
config.jsonConfiguration879 B —
special_tokens_map.jsonConfiguration173 B —
README.mdDocumentation3.2 KB —
.gitattributesRepository1.5 KB —
tokenizer.jsonTokenizer16.3 MB c6b52ff7043b
tokenizer_config.jsonTokenizer453 B —

License and Download

License
mit
Access
Open weights, no gate
Download size
2.2 GB
Download from Tim

Released by Tim through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published2.2 GB
16-bit0.6 GB
8-bit0.3 GB
4-bit0.1 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About mdeberta-v3-base-squad2

How much GPU memory does mdeberta-v3-base-squad2 need?

About 0.7 GB at 16-bit and 0.2 GB at 4-bit: the weights (278M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run mdeberta-v3-base-squad2 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use mdeberta-v3-base-squad2 commercially?

Yes. mdeberta-v3-base-squad2 is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

What is mdeberta-v3-base-squad2's context length?

512 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Question answering

xlm-roberta-base-squad2

Deepset

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set with the official eval script. "exact": 33.67279167589108 "total": 4517 "exact": 48.739495798319325 "total": 1190 Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Open weights cc-by-4.0 277M parameters 514 tokens transformers
View model

Model · Question answering

xlm-roberta-base-squad2-distilled

Deepset

Haystack's distillation feature was used for training. deepset/xlm-roberta-large-squad2 was used as the teacher model. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set Timo Möller: [email protected] deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Open weights mit 277M parameters 514 tokens transformers
View model

Model · Question answering

flan-t5-base-squad2

Sebastian Husch Lee

This is the flan-t5-base model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering. UPDATE: With transformers version 4.31.0 the useremotecode=True is no longer necessary. NOTE: The token must be manually added to the beginning of the question for this model to work properly. It uses the token to be able to make "no answer" predictions. The t5 tokenizer does not automatically add this special token which is why it is added manually. The following hyperparameters were used during training: - learningrate: 2e-05 - trainbatchsize: 16 - evalbatchsize: 8 - gradientaccumulationsteps: 6…

Open weights mit 223M parameters transformers
View model

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is cased: it makes a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.…

Open weights apache-2.0 334M parameters 512 tokens transformers
View model

Model · Question answering

gelectra-large-germanquad

Deepset

We trained a German question answering model with a gelectra-large model as its basis. - The dataset is GermanQuAD, a new, German language dataset, which we hand-annotated and published online. - The training dataset is one-way annotated and contains 11518 questions and 11518 answers, while the test dataset is three-way annotated so that there are 2204 questions and with 2204·3−76 = 6536 answers, because we removed 76 wrong answers. See https://deepset.ai/germanquad for more details and dataset download in SQuAD format. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on…

Open weights mit 335M parameters 512 tokens transformers
View model

This is a berta-large model, fine-tuned using the SQuAD2.0 dataset for the task of question answering. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone!

Open weights cc-by-4.0 335M parameters 512 tokens transformers
View model