# distilbert-base-cased-distilled-squad by DistilBERT community Source: https://savrn.com/models/distilbert-base-cased-distilled-squad Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve distilbert-base-cased-distilled-squad (65M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 0.1 GB | 0.2 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 0.1 GB | 0.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 0.0 GB | 0.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 24, 2026. [distilbert-base-cased-distilled-squad on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/distilbert-base-cased-distilled-squad/gpus) ## SAVRN's Notes on distilbert-base-cased-distilled-squad This is DistilBERT fine-tuned for extractive question answering on SQuAD: hand it a passage and a question, and it returns the span that answers it. At 65M parameters it is small enough to answer in milliseconds on a CPU, which is its whole appeal. Apache 2.0 allows commercial use. It was trained on SQuAD 1.1, so it always returns some span even when the passage has no answer; if your questions may be unanswerable, use a SQuAD 2.0 model such as deepset's DeBERTa-v3 or tinyroberta versions instead. SAVRN Research, 2026-09-18 ## Model Card By DistilBERT community, published under apache-2.0, revision 564e9b582944. ### Table of Contents - Model Details - How To Get Started With the Model - Uses - Risks, Limitations and Biases - Training - Evaluation - Environmental Impact - Technical Specifications - Citation Information - Model Card Authors ### Model Details Model Description: The DistilBERT model was proposed in the blog post [Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5), and the paper [DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter](https://savrn.com/papers/distilbert-a-distilled-version-of-bert-smaller-faster-cheaper-and-lighter). DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark. This model is a fine-tune checkpoint of [DistilBERT-base-cased](https://huggingface.co/distilbert-base-cased), fine-tuned using (a second step of) knowledge distillation on [SQuAD v1.1](https://huggingface.co/datasets/squad). [Read the full model card (895 words)](https://savrn.com/models/distilbert-base-cased-distilled-squad/card) ## Configuration Architecture DistilBertForQuestionAnswering Context length (tokens) 512 Vocabulary size 28,996 Model type distilbert ## Identity and Version Repository distilbert/distilbert-base-cased-distilled-squad Publisher DistilBERT community Task Question answering Modality Text Library transformers Parameters 65M parameters Languages en Revision 564e9b582944a57a3e586bbb98fd6f0a4118db7f First published 2022-03-02 Last updated 2024-05-06 ## Files and Weights 14 files, 1.8 GB in total. The weights are 5 files totalling 1.3 GB in bin, h5, ot, safetensors. Weights5 files · 1.3 GB Configuration1 file · 473 B Tokenizer3 files · 649.3 KB Documentation1 file · 9.5 KB Other3 files · 483.1 MB Repository1 file · 399 B Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | model.safetensors | Weights | 260.8 MB | f198de8ef6e4 | | openvino_model.bin | Weights | 260.8 MB | fbe35d48b0d7 | | pytorch_model.bin | Weights | 260.8 MB | 4e10bdbc83fd | | rust_model.ot | Weights | 260.8 MB | 8a9f9b2f153a | | tf_model.h5 | Weights | 260.9 MB | 63ee5a014206 | | config.json | Configuration | 473 B | — | | README.md | Documentation | 9.5 KB | — | | openvino_model.xml | Other | 512.9 KB | — | | saved_model.tar.gz | Other | 241.5 MB | f7e26fe22fde | | tfjs.tar.gz | Other | 241.1 MB | 2e966858819f | | .gitattributes | Repository | 399 B | — | | tokenizer.json | Tokenizer | 435.8 KB | — | | tokenizer_config.json | Tokenizer | 49 B | — | | vocab.txt | Tokenizer | 213.4 KB | — | ## License and Download License apache-2.0 Access Open weights, no gate Download size 1.3 GB [Download from DistilBERT community](https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad) Released by DistilBERT community through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0). ## Built From - Described by [arXiv:1910.01108](https://savrn.com/papers/distilbert-a-distilled-version-of-bert-smaller-faster-cheaper-and-lighter) - Described by [arXiv:1910.09700](https://savrn.com/papers/quantifying-the-carbon-emissions-of-machine-learning) - Trained on (disclosed) squad ## Evaluations Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical. | Benchmark | Conditions | Result | Reported by | Revision | Date | | --- | --- | --- | --- | --- | --- | | squad | Configuration plain_textTask Question AnsweringMetric Exact MatchComparison conditions not established | 79.5998 | [distilbert](https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad) Publisher reported | Evaluated revision not stated | — | | squad | Configuration plain_textTask Question AnsweringMetric F1Comparison conditions not established | 86.9965 | [distilbert](https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad) Publisher reported | Evaluated revision not stated | — | ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 1.3 GB | | 16-bit | 0.1 GB | | 8-bit | 0.1 GB | | 4-bit | 0.0 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Built on This Model - Quantized from[distilbert-base-cased-distilled-squad](https://savrn.com/models/distilbert-base-cased-distilled-squad-2) - Derived from[distilbert-base-cased-distilled-squad](https://savrn.com/models/distilbert-base-cased-distilled-squad-2) ## Questions About distilbert-base-cased-distilled-squad ### How much GPU memory does distilbert-base-cased-distilled-squad need? About 0.2 GB at 16-bit and 0 GB at 4-bit: the weights (65M parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run distilbert-base-cased-distilled-squad on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use distilbert-base-cased-distilled-squad commercially? Yes. distilbert-base-cased-distilled-squad is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ### What is distilbert-base-cased-distilled-squad's context length? 512 tokens, from the maximum position embeddings in its published configuration. ## Similar Models Model · Question answering ### [distilbert-base-uncased-distilled-squad](https://savrn.com/models/distilbert-base-uncased-distilled-squad) [DistilBERT community](https://savrn.com/model-publishers/distilbert) The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark. This model is a fine-tune checkpoint of DistilBERT-base-uncased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1. - See this repository for more about Distil\ (a class of… Open weights apache-2.0 66M parameters 512 tokens transformers [View model](https://savrn.com/models/distilbert-base-uncased-distilled-squad) Model · Question answering ### [tinyroberta-squad2](https://savrn.com/models/tinyroberta-squad2) [Deepset](https://savrn.com/model-publishers/deepset) This is the distilled version of the deepset/roberta-base-squad2 model. This model has a comparable prediction quality and runs at twice the speed of the base model. This model was distilled using the TinyBERT approach described in this paper and implemented in haystack. Firstly, we have performed intermediate layer distillation with roberta-base as the teacher which resulted in deepset/tinyroberta-6l-768d. Secondly, we have performed task-specific distillation with deepset/roberta-base-squad2 as the teacher for further intermediate layer distillation on an augmented version of SQuADv2 and then with deepset/roberta-large-squad2 as the teacher for prediction layer distillation. Haystack is… Open weights cc-by-4.0 82M parameters 514 tokens transformers [View model](https://savrn.com/models/tinyroberta-squad2) Model · Question answering ### [bert-medium-squad2-distilled](https://savrn.com/models/bert-medium-squad2-distilled) [Deepset](https://savrn.com/model-publishers/deepset) Haystack version 1.x distillation feature was used for training. deepset/bert-large-uncased-whole-word-masking-squad2 was used as the teacher model. Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. - Timo Möller: timo.moeller [at] deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to… Open weights mit 41M parameters 512 tokens transformers [View model](https://savrn.com/models/bert-medium-squad2-distilled) Model · Question answering ### [minilm-uncased-squad2](https://savrn.com/models/minilm-uncased-squad2) [Deepset](https://savrn.com/model-publishers/deepset) Haystack is an AI orchestration framework to build customizable, production-ready LLM applications. You can use this model in Haystack to do extractive question answering on documents. To load and run the model with Haystack: For a complete example with an extractive question answering pipeline that scales over many documents, check out the corresponding Haystack tutorial. Evaluated on the SQuAD 2.0 dev set with the official eval script. Timo Möller: timo.moeller@deepset.ai deepset is the company behind the production-ready open-source AI framework Haystack. We also have a Discord community open to everyone! Open weights cc-by-4.0 33M parameters 512 tokens transformers [View model](https://savrn.com/models/minilm-uncased-squad2) Model · Question answering ### [rai](https://savrn.com/models/rai) [Prayas Abhinav](https://savrn.com/model-publishers/prayasabhinav) What it is. An extractive question-answering model. Given a question and a short typed answer, it returns the phrase in the answer that answers the question, or nothing. It cannot generate text. It is the reading part of rai, a tool that says only whether a description is complete under a notion of completeness written down in advance. Every decision after the reading is made in plain code. Trained from. deepset/minilm-uncased-squad2 (CC-BY-4.0), deepset's fine-tune of Microsoft's MiniLM-L12-H384-uncased (MIT) on SQuAD 2.0. Same architecture, 33M parameters, nothing added. Credit to deepset and Microsoft. 0.3.7 trains on 0.3.6's rows and 5,800 more written in Claude Code. In 0.3.6's rows… Open weights cc-by-4.0 33M parameters 512 tokens transformers [View model](https://savrn.com/models/rai) Model · Question answering ### [basic-chat-model](https://savrn.com/models/basic-chat-model) [Lenin Villanueva](https://savrn.com/model-publishers/leninangelov) This modelcard aims to be a base template for new models. It has been generated using this raw template. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Use the code below to get started with the model. Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). Open weights apache-2.0 31M parameters [View model](https://savrn.com/models/basic-chat-model) ## DistilBERT community [All models and datasets](https://savrn.com/model-publishers/distilbert) ## Versions - [564e9b582944](https://savrn.com/models/distilbert-base-cased-distilled-squad/versions/564e9b582944) · current 2026-09-18 ## Explore More - [All question answering models](https://savrn.com/models/tasks/question-answering) - [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-20. - [Hugging Face record](https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad) - [How the hub is built](https://savrn.com/model-hub/methodology)