SAVRN
Search Contact SAVRN

Open-weight model · Fill mask

deberta-base

by Microsoft microsoft/deberta-base

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data. Please check the official repository for more details and updates.

Parameters—
Context512
Weights1.7 GB
Licensemit
AccessOpen weights
Monthly Downloads282.9k

Model Card

By Microsoft, published under mit, revision 0d1b43ccf21b.

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data. Please check the official repository for more details and updates. We present the dev results on SQuAD 1.1/2.0 and MNLI tasks. If you find DeBERTa useful for your work, please cite the following paper

Read Microsoft's full model card

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.

Please check the official repository for more details and updates.

Fine-tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.

Model SQuAD 1.1 SQuAD 2.0 MNLI-m
RoBERTa-base 91.5/84.6 83.7/80.5 87.6
XLNet-Large -/- -/80.2 86.8
DeBERTa-base 93.1/87.2 86.2/83.1 88.8

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

Configuration

Context length (tokens)
512
Layers
12
Hidden size
768
Feed-forward size
3,072
Attention heads
12
Vocabulary size
50,265
Model type
deberta

Identity and Version

Repository
microsoft/deberta-base
Publisher
Microsoft
Task
Fill mask
Modality
Text
Library
transformers
Parameters
Not stated by the source
Languages
en
Revision
0d1b43ccf21b5acd9f4e5f7b077fa698f05cf195
First published
2022-03-02
Last updated
2022-09-26

Files and Weights

10 files, 1.7 GB in total. The weights are 4 files totalling 1.7 GB in bin, h5, ot.

Weights4 files · 1.7 GB
Configuration1 file · 474 B
Tokenizer3 files · 1.4 MB
Documentation1 file · 1.3 KB
Repository1 file · 345 B
Every file
FileTypeSizeSHA-256
bpe_encoder.binWeights3.9 MB e7c6f9eecb46
pytorch_model.binWeights558.6 MB 81ab61c5dc91
rust_model.otWeights558.6 MB 42a8937bf8d3
tf_model.h5Weights554.7 MB 16442fa0caaa
config.jsonConfiguration474 B —
README.mdDocumentation1.3 KB —
.gitattributesRepository345 B —
merges.txtTokenizer456.3 KB —
tokenizer_config.jsonTokenizer52 B —
vocab.jsonTokenizer898.8 KB —

License and Download

License
mit
Access
Open weights, no gate
Download size
1.7 GB
Download from Microsoft

Released by Microsoft through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published1.7 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Built on This Model

Questions About deberta-base

Can I use deberta-base commercially?

Yes. deberta-base is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

What is deberta-base's context length?

512 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Fill mask

mdeberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was…

Open weights mit 512 tokens transformers
View model

Model · Fill mask

deberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M…

Open weights mit 512 tokens transformers
View model

Model · Fill mask

Bio_ClinicalBERT

Emily Alsentzer

The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (casedL-12H-768A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes. The BioClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Each note in MIMIC was first split…

Open weights mit 512 tokens transformers
View model

BERTimbau Large is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large. For further information or requests, please go to BERTimbau repository. If you use our work, please cite

Open weights mit 512 tokens transformers
View model

This model was previously named "PubMedBERT (abstracts)". You can either adopt the new model name "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract" or update your transformers library to version 4.22+ if you need to refer to the old name. Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. Recent work shows that for domains with abundant unlabeled text, such as biomedicine, pretraining…

Open weights mit 512 tokens transformers
View model

Model · Fill mask

deberta-v3-large

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M…

Open weights mit 512 tokens transformers
View model