SAVRN
Search Contact SAVRN

Open-weight model · Question answering

roberta-base-squad2

by Deepset deepset/roberta-base-squad2

This is the roberta-base model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering.

Parameters124M
Context514
Weights2.5 GB
Licensecc-by-4.0
AccessOpen weights
Monthly Downloads559.7k

Runs On

What it takes to serve roberta-base-squad2 (124M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.2 GB 0.3 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.1 GB 0.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.1 GB 0.1 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

SAVRN's Notes on roberta-base-squad2

Extractive question answering is the job: give it a passage and a question and it points at the answering span, or reports that there is none, since Deepset trained it on SQuAD2.0 pairs that include unanswerable questions. At 124M parameters it needs 0.2 GB of weights and 0.3 GB of memory at 16-bit. The cheapest setup in our Index, one MI300X with 192 GB at $1.85 per hour, sits almost entirely idle on this model. Nobody should buy a GPU for it.

CC BY 4.0 allows commercial use and adaptation provided Deepset is credited and changes are indicated, so a fine-tune on your own documents ships with an attribution line and a change note. Check the 514-token context first, since longer documents must be chunked ahead of the model. It derives from FacebookAI/roberta-base, and Deepset also released deepset/tinyroberta-squad2, described as twice as fast, worth a side-by-side test.

SAVRN Research, 2026-09-18

Model Card

By Deepset, published under cc-by-4.0, revision adc3b06f79f7.

roberta-base for Extractive QA

This is the roberta-base model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering. We have also released a distilled version of this model called deepset/tinyroberta-squad2. It has a comparable prediction quality and runs at twice the speed of deepset/roberta-base-squad2.

Overview

Language model: roberta-base
Language: English
Downstream-task: Extractive QA
Training data: SQuAD 2.0
Eval data: SQuAD 2.0
Code: See an example extractive QA pipeline built with Haystack
Infrastructure: 4x Tesla v100

Hyperparameters

batch_size = 96
n_epochs = 2
base_LM_model = "roberta-base"
max_seq_len = 386
learning_rate = 3e-5
lr_schedule = LinearWarmup
warmup_proportion = 0.2
doc_stride=128
max_query_length=64

Usage

In Haystack

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack:

Read the full model card (435 words)

Configuration

Architecture
RobertaForQuestionAnswering
Context length (tokens)
514
Layers
12
Hidden size
768
Feed-forward size
3,072
Attention heads
12
Vocabulary size
50,265
Model type
roberta

Identity and Version

Repository
deepset/roberta-base-squad2
Publisher
Deepset
Task
Question answering
Modality
Text
Library
transformers
Parameters
124M parameters
Languages
en
Revision
adc3b06f79f797d1c575d5479d6f5efe54a9e3b4
First published
2022-03-02
Last updated
2024-09-24

Files and Weights

12 files, 2.5 GB in total. The weights are 5 files totalling 2.5 GB in bin, h5, msgpack, ot, safetensors.

Weights5 files · 2.5 GB
Configuration2 files · 1.3 KB
Tokenizer3 files · 1.4 MB
Documentation1 file · 9.2 KB
Repository1 file · 445 B
Every file
FileTypeSizeSHA-256
flax_model.msgpackWeights496.2 MB 6a8d759d881d
model.safetensorsWeights496.3 MB ac5db66fdcfe
pytorch_model.binWeights496.3 MB e0b64ccefc1b
rust_model.otWeights498.6 MB 5a16ed126bbc
tf_model.h5Weights496.5 MB 9b672dd16f09
config.jsonConfiguration571 B —
special_tokens_map.jsonConfiguration772 B —
README.mdDocumentation9.2 KB —
.gitattributesRepository445 B —
merges.txtTokenizer456.3 KB —
tokenizer_config.jsonTokenizer79 B —
vocab.jsonTokenizer898.8 KB —

License and Download

License
cc-by-4.0
Access
Open weights, no gate
Download size
2.5 GB
Download from Deepset

Released by Deepset through its official repository on Hugging Face. Read the license.

Built From

Evaluations

Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical.

BenchmarkConditionsResultReported byRevisionDate
adversarial_qa Configuration adversarialQATask Question AnsweringMetric Exact MatchComparison conditions not established 29.5 deepset
Publisher reported		 Evaluated revision not stated —
adversarial_qa Configuration adversarialQATask Question AnsweringMetric F1Comparison conditions not established 40.367 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric Exact MatchComparison conditions not established 85.289 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric F1Comparison conditions not established 91.841 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric Exact MatchComparison conditions not established 78.567 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric F1Comparison conditions not established 84.469 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric Exact MatchComparison conditions not established 79.9309 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric F1Comparison conditions not established 82.9501 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric totalComparison conditions not established 11869 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric Exact MatchComparison conditions not established 69.924 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric F1Comparison conditions not established 83.284 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric Exact MatchComparison conditions not established 81.204 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric F1Comparison conditions not established 90.595 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric Exact MatchComparison conditions not established 82.931 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric F1Comparison conditions not established 90.756 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric Exact MatchComparison conditions not established 71.55 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric F1Comparison conditions not established 82.939 deepset
Publisher reported		 Evaluated revision not stated —

Memory Requirements

PrecisionWeights in memory
As published2.5 GB
16-bit0.2 GB
8-bit0.1 GB
4-bit0.1 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About roberta-base-squad2

How much GPU memory does roberta-base-squad2 need?

About 0.3 GB at 16-bit and 0.1 GB at 4-bit: the weights (124M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run roberta-base-squad2 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use roberta-base-squad2 commercially?

Yes. roberta-base-squad2 is released under Creative Commons Attribution 4.0. CC BY 4.0 permits sharing and adapting the work, including commercially, provided the creator is credited and changes are indicated.

What is roberta-base-squad2's context length?

514 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Question answering

roberta-base-squad2-distilled

Deepset

haystack's distillation feature was used for training. deepset/roberta-large-squad2 was used as the teacher model. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Timo Möller: [email protected] deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone!

Open weights mit 124M parameters 514 tokens transformers
View model

Model · Question answering

roberta-base-cuad

Taner Akdeniz

This model is the fine-tuned version of "RoBERTa Base" using CUAD dataset https://huggingface.co/datasets/cuad For the use of the model with CUAD: https://github.com/marshmellow77/cuad-demo and https://huggingface.co/spaces/akdeniz27/contract-understanding-atticus-dataset-demo

Open weights 124M parameters 514 tokens transformers
View model

Model · Question answering

bert-base-uncased-squad-v1

Qingqing Cao

This model was fine-tuned from the HuggingFace BERT base uncased checkpoint on SQuAD1.1. CPU: Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz Memory: 32 GiB GPUs: 2 GeForce GTX 1070, each with 8GiB memory GPU driver: 418.87.01, CUDA: 10.1 It took about 2 hours to finish. Note that the above results didn't involve any hyperparameter search.

Open weights mit 109M parameters 512 tokens transformers
View model

Model · Question answering

gelectra-base-germanquad

Deepset

We trained a German question answering model with a gelectra-base model as its basis. - The dataset is GermanQuAD, a new, German language dataset, which we hand-annotated and published online. - The training dataset is one-way annotated and contains 11518 questions and 11518 answers, while the test dataset is three-way annotated so that there are 2204 questions and with 2204·3−76 = 6536answers, because we removed 76 wrong answers. See https://deepset.ai/germanquad for more details and dataset download in SQuAD format. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on…

Open weights mit 109M parameters 512 tokens transformers
View model

Model · Question answering

bert-base-uncased-squad2

Deepset

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. - Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone!

Open weights cc-by-4.0 109M parameters 512 tokens transformers
View model