SAVRN
Search Contact SAVRN

Open-weight model · Question answering

roberta-large-squad2

by Deepset deepset/roberta-large-squad2

roberta-large-squad2 is an open-weight model for question answering from Deepset, released under Creative Commons Attribution 4.0. It has 354M parameters and a 514-token context. At 16-bit it needs about 0.9 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 9.4k downloads a month.

This is the roberta-large model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering.

Parameters354M
Context514
Weights4.3 GB
Licensecc-by-4.0
AccessOpen weights
Monthly Downloads9.4k

Runs On

What it takes to serve roberta-large-squad2 (354M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.7 GB 0.9 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.4 GB 0.4 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.2 GB 0.2 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 13, 2026.

roberta-large-squad2 on every accelerator the SAVRN Index prices, at every precision

Model Card

By Deepset, published under cc-by-4.0, revision 78fb38a59ea3.

roberta-large for Extractive QA

This is the roberta-large model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering.

Overview

Language model: roberta-large
Language: English
Downstream-task: Extractive QA
Training data: SQuAD 2.0
Eval data: SQuAD 2.0
Code: See an example extractive QA pipeline built with Haystack
Infrastructure: 4x Tesla v100

Hyperparameters

base_LM_model = "roberta-large"

Using a distilled model instead

Please note that we have also released a distilled version of this model called deepset/roberta-base-squad2-distilled. The distilled model has a comparable prediction quality and runs at twice the speed of the large model.

Usage

In Haystack

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack:

Read the full model card (393 words)

Configuration

Architecture
RobertaForQuestionAnswering
Context length (tokens)
514
Layers
24
Hidden size
1,024
Feed-forward size
4,096
Attention heads
16
Vocabulary size
50,265
Model type
roberta

Identity and Version

Repository
deepset/roberta-large-squad2
Publisher
Deepset
Task
Question answering
Modality
Text
Library
transformers
Parameters
354M parameters
Languages
en
Revision
78fb38a59ea3cb6902e04d96da93efc87aeeff76
First published
2022-03-02
Last updated
2024-09-26

Files and Weights

10 files, 4.3 GB in total. The weights are 3 files totalling 4.3 GB in bin, msgpack, safetensors.

Weights3 files · 4.3 GB
Configuration2 files · 935 B
Tokenizer3 files · 1.3 MB
Documentation1 file · 7.7 KB
Repository1 file · 790 B
Every file
FileTypeSizeSHA-256
flax_model.msgpackWeights1.4 GB 17d19dd0f991
model.safetensorsWeights1.4 GB 63ff07ecdc76
pytorch_model.binWeights1.4 GB f0a98cec67b9
config.jsonConfiguration696 B —
special_tokens_map.jsonConfiguration239 B —
README.mdDocumentation7.7 KB —
.gitattributesRepository790 B —
merges.txtTokenizer456.4 KB —
tokenizer_config.jsonTokenizer1.2 KB —
vocab.jsonTokenizer798.3 KB —

License and Download

License
cc-by-4.0
Access
Open weights, no gate
Download size
4.3 GB
Download from Deepset

Released by Deepset through its official repository on Hugging Face. Read the license.

Built From

Evaluations

Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical.

BenchmarkConditionsResultReported byRevisionDate
adversarial_qa Configuration adversarialQATask Question AnsweringMetric Exact MatchComparison conditions not established 35.9 deepset
Publisher reported		 Evaluated revision not stated —
adversarial_qa Configuration adversarialQATask Question AnsweringMetric F1Comparison conditions not established 48.923 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric Exact MatchComparison conditions not established 87.162 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric F1Comparison conditions not established 93.603 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric Exact MatchComparison conditions not established 81.142 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric F1Comparison conditions not established 87.099 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric Exact MatchComparison conditions not established 85.168 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric F1Comparison conditions not established 88.349 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric Exact MatchComparison conditions not established 72.453 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric F1Comparison conditions not established 86.325 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric Exact MatchComparison conditions not established 82.338 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric F1Comparison conditions not established 91.974 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric Exact MatchComparison conditions not established 84.352 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric F1Comparison conditions not established 92.645 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric Exact MatchComparison conditions not established 74.722 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric F1Comparison conditions not established 86.86 deepset
Publisher reported		 Evaluated revision not stated —

Memory Requirements

PrecisionWeights in memory
As published4.3 GB
16-bit0.7 GB
8-bit0.4 GB
4-bit0.2 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About roberta-large-squad2

How much GPU memory does roberta-large-squad2 need?

About 0.9 GB at 16-bit and 0.2 GB at 4-bit: the weights (354M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run roberta-large-squad2 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use roberta-large-squad2 commercially?

Yes. roberta-large-squad2 is released under Creative Commons Attribution 4.0. CC BY 4.0 permits sharing and adapting the work, including commercially, provided the creator is credited and changes are indicated.

What is roberta-large-squad2's context length?

514 tokens, from the maximum position embeddings in its published configuration.

Similar Models

This is a berta-large model, fine-tuned using the SQuAD2.0 dataset for the task of question answering. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone!

Open weights cc-by-4.0 335M parameters 512 tokens transformers
View model

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is uncased: it does not make a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this…

Open weights apache-2.0 335M parameters 512 tokens transformers
View model

Model · Question answering

gelectra-large-germanquad

Deepset

We trained a German question answering model with a gelectra-large model as its basis. - The dataset is GermanQuAD, a new, German language dataset, which we hand-annotated and published online. - The training dataset is one-way annotated and contains 11518 questions and 11518 answers, while the test dataset is three-way annotated so that there are 2204 questions and with 2204·3−76 = 6536 answers, because we removed 76 wrong answers. See https://deepset.ai/germanquad for more details and dataset download in SQuAD format. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on…

Open weights mit 335M parameters 512 tokens transformers
View model

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is cased: it makes a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.…

Open weights apache-2.0 334M parameters 512 tokens transformers
View model

Model · Question answering

mdeberta-v3-base-squad2

Tim

It has been finetuned for 3 epochs on SQuAD2.0. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa…

Open weights mit 278M parameters 512 tokens transformers
View model

Model · Question answering

xlm-roberta-base-squad2

Deepset

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set with the official eval script. "exact": 33.67279167589108 "total": 4517 "exact": 48.739495798319325 "total": 1190 Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Open weights cc-by-4.0 277M parameters 514 tokens transformers
View model