xlm-roberta-base-squad2 by Deepset: Open-Weight Model

Runs On

What it takes to serve xlm-roberta-base-squad2 (277M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Weights	Memory needed	Cheapest setup	Per hour	Also fits
16-bit	0.6 GB	0.7 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	0.3 GB	0.3 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	0.1 GB	0.2 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Deepset, published under cc-by-4.0, revision a5fab9908c8d.

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set with the official eval script. "exact": 33.67279167589108 "total": 4517 "exact": 48.739495798319325 "total": 1190 Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Read Deepset's full model card

Multilingual XLM-RoBERTa base for Extractive QA on various languages

Overview

Language model: xlm-roberta-base
Language: Multilingual
Downstream-task: Extractive QA
Training data: SQuAD 2.0
Eval data: SQuAD 2.0 dev set - German MLQA - German XQuAD
Code: See an example extractive QA pipeline built with Haystack
Infrastructure: 4x Tesla v100

Hyperparameters

batch_size = 22*4
n_epochs = 2
max_seq_len=256,
doc_stride=128,
learning_rate=2e-5,

Corresponding experiment logs in mlflow: link

Usage

In Haystack

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack:

# After running pip install haystack-ai "transformers[torch,sentencepiece]"

from haystack import Document
from haystack.components.readers import ExtractiveReader

docs = [
    Document(content="Python is a popular programming language"),
    Document(content="python ist eine beliebte Programmiersprache"),
]

reader = ExtractiveReader(model="deepset/xlm-roberta-base-squad2")
reader.warm_up()

question = "What is a popular programming language?"
result = reader.run(query=question, documents=docs)
# {'answers': [ExtractedAnswer(query='What is a popular programming language?', score=0.5740374326705933, data='python', document=Document(id=..., content: '...'), context=None, document_offset=ExtractedAnswer.Span(start=0, end=6),...)]}

For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial.

In Transformers

from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline

model_name = "deepset/xlm-roberta-base-squad2"

# a) Get predictions
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
QA_input = {
    'question': 'Why is model conversion important?',
    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
}
res = nlp(QA_input)

# b) Load model & tokenizer
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Performance

Evaluated on the SQuAD 2.0 dev set with the official eval script.

"exact": 73.91560683904657
"f1": 77.14103746689592

Evaluated on German MLQA: test-context-de-question-de.json "exact": 33.67279167589108 "f1": 44.34437105434842 "total": 4517

Evaluated on German XQuAD: xquad.de.json "exact": 48.739495798319325 "f1": 62.552615701071495 "total": 1190

Authors

Branden Chan: branden.chan [at] deepset.ai Timo Möller: timo.moeller [at] deepset.ai Malte Pietsch: malte.pietsch [at] deepset.ai Tanay Soni: tanay.soni [at] deepset.ai

About us

deepset is the company behind the production-ready open-source AI framework Haystack.

Some of our other work: - Distilled roberta-base-squad2 (aka "tinyroberta-squad2") - German BERT, GermanQuAD and GermanDPR, German embedding model - deepset Cloud, deepset Studio

Get in touch and join the Haystack community

For more info on Haystack, visit our GitHub repo and Documentation. We also have a Discord community open to everyone!

By the way: we're hiring!

Configuration

Architecture: XLMRobertaForQuestionAnswering
Context length (tokens): 514
Layers: 12
Hidden size: 768
Feed-forward size: 3,072
Attention heads: 12
Vocabulary size: 250,002
Model type: xlm-roberta

Identity and Version

Repository: deepset/xlm-roberta-base-squad2
Publisher: Deepset
Task: Question answering
Modality: Text
Library: transformers
Parameters: 277M parameters
Languages: xlm-roberta
Revision: a5fab9908c8d856e8c583fd41ba6d92444e46477
First published: 2022-03-02
Last updated: 2024-09-26

Files and Weights

8 files, 2.2 GB in total. The weights are 2 files totalling 2.2 GB in bin, safetensors.

Weights2 files · 2.2 GB

Configuration2 files · 755 B

Tokenizer1 file · 79 B

Documentation1 file · 6.2 KB

Other1 file · 5.1 MB

Repository1 file · 513 B

Every file

File	Type	Size	SHA-256
model.safetensors	Weights	1.1 GB	b3b3953828b8
pytorch_model.bin	Weights	1.1 GB	faeaac92ea0e
config.json	Configuration	605 B	—
special_tokens_map.json	Configuration	150 B	—
README.md	Documentation	6.2 KB	—
sentencepiece.bpe.model	Other	5.1 MB	cfc8146abe2a
.gitattributes	Repository	513 B	—
tokenizer_config.json	Tokenizer	79 B	—

License and Download

License: cc-by-4.0
Access: Open weights, no gate
Download size: 2.2 GB

Download from Deepset

Released by Deepset through its official repository on Hugging Face. Read the license.

Built From

Trained on (disclosed) squad_v2

Evaluations

Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical.

Benchmark	Conditions	Result	Reported by	Revision	Date
squad_v2	Configuration squad_v2Task Question AnsweringMetric Exact MatchComparison conditions not established	74.0354	deepset Publisher reported	Evaluated revision not stated	—
squad_v2	Configuration squad_v2Task Question AnsweringMetric F1Comparison conditions not established	77.1833	deepset Publisher reported	Evaluated revision not stated	—

Memory Requirements

Precision	Weights in memory
As published	2.2 GB
16-bit	0.6 GB
8-bit	0.3 GB
4-bit	0.1 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About xlm-roberta-base-squad2

How much GPU memory does xlm-roberta-base-squad2 need?

About 0.7 GB at 16-bit and 0.2 GB at 4-bit: the weights (277M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run xlm-roberta-base-squad2 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use xlm-roberta-base-squad2 commercially?

Yes. xlm-roberta-base-squad2 is released under Creative Commons Attribution 4.0. CC BY 4.0 permits sharing and adapting the work, including commercially, provided the creator is credited and changes are indicated.

What is xlm-roberta-base-squad2's context length?

514 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Question answering

xlm-roberta-base-squad2-distilled

Deepset

Haystack's distillation feature was used for training. deepset/xlm-roberta-large-squad2 was used as the teacher model. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set Timo Möller: [email protected] deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Open weights mit 277M parameters 514 tokens transformers

View model

Model · Question answering

mdeberta-v3-base-squad2

Tim

It has been finetuned for 3 epochs on SQuAD2.0. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa…

Open weights mit 278M parameters 512 tokens transformers

View model

Model · Question answering

flan-t5-base-squad2

Sebastian Husch Lee

This is the flan-t5-base model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering. UPDATE: With transformers version 4.31.0 the useremotecode=True is no longer necessary. NOTE: The token must be manually added to the beginning of the question for this model to work properly. It uses the token to be able to make "no answer" predictions. The t5 tokenizer does not automatically add this special token which is why it is added manually. The following hyperparameters were used during training: - learningrate: 2e-05 - trainbatchsize: 16 - evalbatchsize: 8 - gradientaccumulationsteps: 6…

Open weights mit 223M parameters transformers

View model

Model · Question answering

bert-large-cased-whole-word-masking-finetuned-squad

BERT community

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is cased: it makes a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.…

Open weights apache-2.0 334M parameters 512 tokens transformers

View model

Model · Question answering

gelectra-large-germanquad

Deepset

We trained a German question answering model with a gelectra-large model as its basis. - The dataset is GermanQuAD, a new, German language dataset, which we hand-annotated and published online. - The training dataset is one-way annotated and contains 11518 questions and 11518 answers, while the test dataset is three-way annotated so that there are 2204 questions and with 2204·3−76 = 6536 answers, because we removed 76 wrong answers. See https://deepset.ai/germanquad for more details and dataset download in SQuAD format. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on…

Open weights mit 335M parameters 512 tokens transformers

View model

Model · Question answering

bert-large-uncased-whole-word-masking-squad2

Deepset

This is a berta-large model, fine-tuned using the SQuAD2.0 dataset for the task of question answering. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone!

Open weights cc-by-4.0 335M parameters 512 tokens transformers

View model