SAVRN
Search Contact SAVRN

Open-weight model · Question answering

bert-large-uncased-whole-word-masking-squad2

by Deepset deepset/bert-large-uncased-whole-word-masking-squad2

This is a berta-large model, fine-tuned using the SQuAD2.0 dataset for the task of question answering. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications.

Parameters335M
Context512
Weights5.4 GB
Licensecc-by-4.0
AccessOpen weights
Monthly Downloads206.1k

Runs On

What it takes to serve bert-large-uncased-whole-word-masking-squad2 (335M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.7 GB 0.8 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.3 GB 0.4 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.2 GB 0.2 GB 1x MI300X (192 GB)
Vultr		 $1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Deepset, published under cc-by-4.0, revision 0a489d6caaf6.

bert-large-uncased-whole-word-masking-squad2 for Extractive QA

This is a berta-large model, fine-tuned using the SQuAD2.0 dataset for the task of question answering.

Overview

Language model: bert-large
Language: English
Downstream-task: Extractive QA
Training data: SQuAD 2.0
Eval data: SQuAD 2.0
Code: See an example extractive QA pipeline built with Haystack

Usage

In Haystack

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack:

# After running pip install haystack-ai "transformers[torch,sentencepiece]"

from haystack import Document
from haystack.components.readers import ExtractiveReader

docs = [
    Document(content="Python is a popular programming language"),
    Document(content="python ist eine beliebte Programmiersprache"),
]

reader = ExtractiveReader(model="deepset/bert-large-uncased-whole-word-masking-squad2")
reader.warm_up()

question = "What is a popular programming language?"
result = reader.run(query=question, documents=docs)
# {'answers': [ExtractedAnswer(query='What is a popular programming language?', score=0.5740374326705933, data='python', document=Document(id=..., content: '...'), context=None, document_offset=ExtractedAnswer.Span(start=0, end=6),...)]}

Read the full model card (323 words)

Configuration

Architecture
BertForQuestionAnswering
Context length (tokens)
512
Layers
24
Hidden size
1,024
Feed-forward size
4,096
Attention heads
16
Vocabulary size
30,522
Model type
bert

Identity and Version

Repository
deepset/bert-large-uncased-whole-word-masking-squad2
Publisher
Deepset
Task
Question answering
Modality
Text
Library
transformers
Parameters
335M parameters
Languages
en
Revision
0a489d6caaf651c1983e8ecd8a418c7f65993d72
First published
2022-03-02
Last updated
2024-09-24

Files and Weights

12 files, 6.6 GB in total. The weights are 4 files totalling 5.4 GB in bin, h5, msgpack, safetensors.

Weights4 files · 5.4 GB
Configuration3 files · 654 B
Tokenizer2 files · 231.5 KB
Documentation1 file · 7.5 KB
Other1 file · 1.2 GB
Repository1 file · 445 B
Every file
FileTypeSizeSHA-256
flax_model.msgpackWeights1.3 GB 7a06e722d755
model.safetensorsWeights1.3 GB f40d65b33b6c
pytorch_model.binWeights1.3 GB bb9ea214f1b4
tf_model.h5Weights1.3 GB 69184a2ab8f5
added_tokens.jsonConfiguration2 B —
config.jsonConfiguration540 B —
special_tokens_map.jsonConfiguration112 B —
README.mdDocumentation7.5 KB —
saved_model.tar.gzOther1.2 GB 905fc7ee48ca
.gitattributesRepository445 B —
tokenizer_config.jsonTokenizer19 B —
vocab.txtTokenizer231.5 KB —

License and Download

License
cc-by-4.0
Access
Open weights, no gate
Download size
5.4 GB
Download from Deepset

Released by Deepset through its official repository on Hugging Face. Read the license.

Built From

  • Trained on (disclosed) squad_v2

Evaluations

Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical.

BenchmarkConditionsResultReported byRevisionDate
adversarial_qa Configuration adversarialQATask Question AnsweringMetric Exact MatchComparison conditions not established 28.233 deepset
Publisher reported		 Evaluated revision not stated —
adversarial_qa Configuration adversarialQATask Question AnsweringMetric F1Comparison conditions not established 41.17 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric Exact MatchComparison conditions not established 85.904 deepset
Publisher reported		 Evaluated revision not stated —
squad Configuration plain_textTask Question AnsweringMetric F1Comparison conditions not established 92.586 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric Exact MatchComparison conditions not established 78.064 deepset
Publisher reported		 Evaluated revision not stated —
squad_adversarial Configuration AddOneSentTask Question AnsweringMetric F1Comparison conditions not established 83.591 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric Exact MatchComparison conditions not established 80.8846 deepset
Publisher reported		 Evaluated revision not stated —
squad_v2 Configuration squad_v2Task Question AnsweringMetric F1Comparison conditions not established 83.8765 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric Exact MatchComparison conditions not established 65.615 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts amazon Configuration amazonTask Question AnsweringMetric F1Comparison conditions not established 80.733 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric Exact MatchComparison conditions not established 81.57 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts new_wiki Configuration new_wikiTask Question AnsweringMetric F1Comparison conditions not established 91.199 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric Exact MatchComparison conditions not established 83.279 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts nyt Configuration nytTask Question AnsweringMetric F1Comparison conditions not established 91.09 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric Exact MatchComparison conditions not established 69.305 deepset
Publisher reported		 Evaluated revision not stated —
squadshifts reddit Configuration redditTask Question AnsweringMetric F1Comparison conditions not established 82.405 deepset
Publisher reported		 Evaluated revision not stated —

Memory Requirements

PrecisionWeights in memory
As published5.4 GB
16-bit0.7 GB
8-bit0.3 GB
4-bit0.2 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About bert-large-uncased-whole-word-masking-squad2

How much GPU memory does bert-large-uncased-whole-word-masking-squad2 need?

About 0.8 GB at 16-bit and 0.2 GB at 4-bit: the weights (335M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run bert-large-uncased-whole-word-masking-squad2 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use bert-large-uncased-whole-word-masking-squad2 commercially?

Yes. bert-large-uncased-whole-word-masking-squad2 is released under Creative Commons Attribution 4.0. CC BY 4.0 permits sharing and adapting the work, including commercially, provided the creator is credited and changes are indicated.

What is bert-large-uncased-whole-word-masking-squad2's context length?

512 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is uncased: it does not make a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this…

Open weights apache-2.0 335M parameters 512 tokens transformers
View model

Model · Question answering

gelectra-large-germanquad

Deepset

We trained a German question answering model with a gelectra-large model as its basis. - The dataset is GermanQuAD, a new, German language dataset, which we hand-annotated and published online. - The training dataset is one-way annotated and contains 11518 questions and 11518 answers, while the test dataset is three-way annotated so that there are 2204 questions and with 2204·3−76 = 6536 answers, because we removed 76 wrong answers. See https://deepset.ai/germanquad for more details and dataset download in SQuAD format. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on…

Open weights mit 335M parameters 512 tokens transformers
View model

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is cased: it makes a difference between english and English. Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same. The training is identical -- each masked WordPiece token is predicted independently. After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.…

Open weights apache-2.0 334M parameters 512 tokens transformers
View model

Model · Question answering

roberta-large-squad2

Deepset

This is the roberta-large model, fine-tuned using the SQuAD2.0 dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Question Answering. Please note that we have also released a distilled version of this model called deepset/roberta-base-squad2-distilled. The distilled model has a comparable prediction quality and runs at twice the speed of the large model. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering…

Open weights cc-by-4.0 354M parameters 514 tokens transformers
View model

Model · Question answering

mdeberta-v3-base-squad2

Tim

It has been finetuned for 3 epochs on SQuAD2.0. DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa…

Open weights mit 278M parameters 512 tokens transformers
View model

Model · Question answering

xlm-roberta-base-squad2

Deepset

Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set with the official eval script. "exact": 33.67279167589108 "total": 4517 "exact": 48.739495798319325 "total": 1190 Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to…

Open weights cc-by-4.0 277M parameters 514 tokens transformers
View model