# deberta-v3-large by Microsoft: Open-Weight Model Source: https://savrn.com/models/deberta-v3-large Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## SAVRN's Notes on deberta-v3-large The 3.2 GB repository carries three copies of the same weights: an 873.7 MB PyTorch bin, a 571.3 MB generator bin and a 1.7 GB TensorFlow h5 — download one, not all. The card describes 304M backbone parameters plus a 131M-parameter embedding layer from its 128K vocabulary, with 24 layers, a 1,024 hidden size and 512 tokens of context. The publisher never stated a consolidated parameter count to the hub, so our precision table shows only the published size. This is a fine-tuning base, not a service: fill-mask pretraining under MIT, with ELECTRA-style gradient-disentangled embedding sharing per the DeBERTa V3 paper, arXiv 2111.09543. The hub record shows the ecosystem it anchors — zero-shot and NLI fine-tunes both derive from it. At 1.1M downloads a month it still anchors NLU work, but new projects needing long context or instruction behavior should weigh the neighbors; 512 tokens is a hard ceiling here. SAVRN Research ## Model Card By Microsoft, published under mit, revision 64a8c8eab3e3. ### DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing [DeBERTa](https://savrn.com/papers/deberta-decoding-enhanced-bert-with-disentangled-attention) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In [DeBERTa V3](https://savrn.com/papers/debertav3-improving-deberta-using-electra-style-pre-training-with-gradient-disentangled-em), we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our [paper](https://savrn.com/papers/debertav3-improving-deberta-using-electra-style-pre-training-with-gradient-disentangled-em). Please check the [official repository](https://github.com/microsoft/DeBERTa) for more implementation details and updates. The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M backbone parameters with a vocabulary containing 128K tokens which introduces 131M parameters in the Embedding layer. This model was trained using the 160GB data as DeBERTa V2. #### Fine-tuning on NLU tasks We present the dev results on SQuAD 2.0 and MNLI tasks. [Read the full model card (312 words)](https://savrn.com/models/deberta-v3-large/card) ## Configuration Context length (tokens) 512 Layers 24 Hidden size 1,024 Feed-forward size 4,096 Attention heads 16 Vocabulary size 128,100 Model type deberta-v2 ## Identity and Version Repository microsoft/deberta-v3-large Publisher Microsoft Task Fill mask Modality Text Library transformers Parameters Not stated by the source Languages en Revision 64a8c8eab3e352a784c658aef62be1662607476f First published 2022-03-02 Last updated 2023-03-19 ## Files and Weights 9 files, 3.2 GB in total. The weights are 3 files totalling 3.2 GB in bin, h5. Weights3 files · 3.2 GB Configuration2 files · 1.1 KB Tokenizer1 file · 52 B Documentation1 file · 3.3 KB Other1 file · 2.5 MB Repository1 file · 1.2 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | pytorch_model.bin | Weights | 873.7 MB | dd5b5d93e2db | | pytorch_model.generator.bin | Weights | 571.3 MB | ff85455c5628 | | tf_model.h5 | Weights | 1.7 GB | 20462c6c7699 | | config.json | Configuration | 580 B | — | | generator_config.json | Configuration | 560 B | — | | README.md | Documentation | 3.3 KB | — | | spm.model | Other | 2.5 MB | c679fbf93643 | | .gitattributes | Repository | 1.2 KB | — | | tokenizer_config.json | Tokenizer | 52 B | — | ## License and Download License mit Access Open weights, no gate Download size 3.2 GB [Download from Microsoft](https://huggingface.co/microsoft/deberta-v3-large) Released by Microsoft through its official repository on Hugging Face. [Read the license](https://opensource.org/license/mit). ## Built From - Described by [arXiv:2006.03654](https://savrn.com/papers/deberta-decoding-enhanced-bert-with-disentangled-attention) - Described by [arXiv:2111.09543](https://savrn.com/papers/debertav3-improving-deberta-using-electra-style-pre-training-with-gradient-disentangled-em) ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 3.2 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Built on This Model - Quantized from[deberta-v3-large-zeroshot-v2.0](https://savrn.com/models/deberta-v3-large-zeroshot-v2-0) - Derived from[deberta-v3-large-zeroshot-v2.0](https://savrn.com/models/deberta-v3-large-zeroshot-v2-0) - Quantized from[nli-deberta-v3-large](https://savrn.com/models/nli-deberta-v3-large) - Derived from[nli-deberta-v3-large](https://savrn.com/models/nli-deberta-v3-large) - Derived from[deberta_MP_dynamic](https://savrn.com/models/deberta-mp-dynamic) - Derived from[belief-engine-argument-quality-deberta-v3-large](https://savrn.com/models/belief-engine-argument-quality-deberta-v3-large) - Derived from[disaster-interpretation-deberta-v3-large](https://savrn.com/models/disaster-interpretation-deberta-v3-large) ## Questions About deberta-v3-large ### Can I use deberta-v3-large commercially? Yes. deberta-v3-large is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included. ### What is deberta-v3-large's context length? 512 tokens, from the maximum position embeddings in its published configuration. ## Similar Models Model · Fill mask ### [mdeberta-v3-base](https://savrn.com/models/mdeberta-v3-base) [Microsoft](https://savrn.com/model-publishers/microsoft) DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/mdeberta-v3-base) Model · Fill mask ### [deberta-v3-base](https://savrn.com/models/deberta-v3-base) [Microsoft](https://savrn.com/model-publishers/microsoft) DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/deberta-v3-base) Model · Fill mask ### [Bio_ClinicalBERT](https://savrn.com/models/bio-clinicalbert) [Emily Alsentzer](https://savrn.com/model-publishers/emilyalsentzer) The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (casedL-12H-768A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes. The BioClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Each note in MIMIC was first split… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/bio-clinicalbert) Model · Fill mask ### [BiomedNLP-BiomedBERT-base-uncased-abstract](https://savrn.com/models/biomednlp-biomedbert-base-uncased-abstract) [Microsoft](https://savrn.com/model-publishers/microsoft) This model was previously named "PubMedBERT (abstracts)". You can either adopt the new model name "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract" or update your transformers library to version 4.22+ if you need to refer to the old name. Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. Recent work shows that for domains with abundant unlabeled text, such as biomedicine, pretraining… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/biomednlp-biomedbert-base-uncased-abstract) Model · Fill mask ### [legal-bert-base-uncased](https://savrn.com/models/legal-bert-base-uncased) [AUEB NLP Group](https://savrn.com/model-publishers/nlpaueb) LEGAL-BERT is a family of BERT models for the legal domain, intended to assist legal NLP research, computational law, and legal technology applications. To pre-train the different variations of LEGAL-BERT, we collected 12 GB of diverse English legal text from several fields (e.g., legislation, court cases, contracts) scraped from publicly available resources. Sub-domain variants (CONTRACTS-, EURLEX-, ECHR-) and/or general LEGAL-BERT perform better than using BERT out of the box for domain-specific tasks. A light-weight model (33% the size of BERT-BASE) pre-trained from scratch on legal data with competitive performance is also available. I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N.… Open weights cc-by-sa-4.0 512 tokens transformers [View model](https://savrn.com/models/legal-bert-base-uncased) Model · Fill mask ### [bert-large-portuguese-cased](https://savrn.com/models/bert-large-portuguese-cased) [NeuralMind Inteligência Artificial](https://savrn.com/model-publishers/neuralmind) BERTimbau Large is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large. For further information or requests, please go to BERTimbau repository. If you use our work, please cite Open weights mit 512 tokens transformers [View model](https://savrn.com/models/bert-large-portuguese-cased) ## Microsoft [All models and datasets](https://savrn.com/model-publishers/microsoft) ## Versions - [64a8c8eab3e3](https://savrn.com/models/deberta-v3-large/versions/64a8c8eab3e3) · current 2026-10-24 ## Explore More - [All fill mask models](https://savrn.com/models/tasks/fill-mask) - [All models under mit](https://savrn.com/models/licenses/mit) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-19. - [Microsoft website](https://github.com/microsoft) - [Hugging Face record](https://huggingface.co/microsoft/deberta-v3-large) - [How the hub is built](https://savrn.com/model-hub/methodology)