bertweet-base by VinAI Research: Open-Weight Model

Model Card

By VinAI Research, published under mit, revision b349c1243407.

BERTweet is the first public large-scale language model pre-trained for English Tweets. BERTweet is trained based on the RoBERTa pre-training procedure. The corpus used to pre-train BERTweet consists of 850M English Tweets (16B word tokens ~ 80GB), containing 845M Tweets streamed from 01/2012 to 08/2019 and 5M Tweets related to the COVID-19 pandemic. The general architecture and experimental results of BERTweet can be found in our paper: author = {Dat Quoc Nguyen and Thanh Vu and Anh Tuan Nguyen}, pages = {9--14}, year = {2020} Please CITE our paper when BERTweet is used to help produce published results or is incorporated into other software. For further information or requests, please go…

Read VinAI Research's full model card

BERTweet: A pre-trained language model for English Tweets

BERTweet is the first public large-scale language model pre-trained for English Tweets. BERTweet is trained based on the RoBERTa pre-training procedure. The corpus used to pre-train BERTweet consists of 850M English Tweets (16B word tokens ~ 80GB), containing 845M Tweets streamed from 01/2012 to 08/2019 and 5M Tweets related to the COVID-19 pandemic. The general architecture and experimental results of BERTweet can be found in our paper:

@inproceedings{bertweet,
title     = {{BERTweet: A pre-trained language model for English Tweets}},
author    = {Dat Quoc Nguyen and Thanh Vu and Anh Tuan Nguyen},
booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
pages     = {9--14},
year      = {2020}
}

Please CITE our paper when BERTweet is used to help produce published results or is incorporated into other software.

For further information or requests, please go to BERTweet's homepage!

Main results

Configuration

Architecture: RobertaForMaskedLM
Context length (tokens): 130
Layers: 12
Hidden size: 768
Feed-forward size: 3,072
Attention heads: 12
Vocabulary size: 64,001
Model type: roberta

Identity and Version

Repository: vinai/bertweet-base
Publisher: VinAI Research
Task: Fill mask
Modality: Text
Library: transformers
Parameters: Not stated by the source
Languages: tf, jax
Revision: b349c1243407b0dcffeabb2337497477286e27ab
First published: 2022-03-02
Last updated: 2024-08-20

Files and Weights

12 files, 1.8 GB in total. The weights are 3 files totalling 1.8 GB in bin, h5, msgpack.

Weights3 files · 1.8 GB

Configuration4 files · 41.3 KB

Tokenizer2 files · 3.8 MB

Documentation1 file · 1.9 KB

Other1 file · 1.1 MB

Repository1 file · 391 B

Every file

File	Type	Size	SHA-256
flax_model.msgpack	Weights	539.9 MB	2b81d613bab4
pytorch_model.bin	Weights	542.5 MB	8831d06dd76f
tf_model.h5	Weights	739.5 MB	1a71e4df2580
__init__.py	Configuration		—
config.json	Configuration	558 B	—
tokenization_bertweet.py	Configuration	28.4 KB	—
tokenization_bertweet_fast.py	Configuration	12.3 KB	—
README.md	Documentation	1.9 KB	—
bpe.codes	Other	1.1 MB	—
.gitattributes	Repository	391 B	—
tokenizer.json	Tokenizer	2.9 MB	—
vocab.txt	Tokenizer	843.4 KB	—

License and Download

License: mit
Access: Open weights, no gate
Download size: 1.8 GB

Download from VinAI Research

Released by VinAI Research through its official repository on Hugging Face. Read the license.

Memory Requirements

Precision	Weights in memory
As published	1.8 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About bertweet-base

Can I use bertweet-base commercially?

Yes. bertweet-base is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

What is bertweet-base's context length?

130 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Fill mask

mdeberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

deberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

Bio_ClinicalBERT

Emily Alsentzer

The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (casedL-12H-768A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes. The BioClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Each note in MIMIC was first split…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

bert-large-portuguese-cased

NeuralMind Inteligência Artificial

BERTimbau Large is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large. For further information or requests, please go to BERTimbau repository. If you use our work, please cite

Open weights mit 512 tokens transformers

View model

Model · Fill mask

BiomedNLP-BiomedBERT-base-uncased-abstract

Microsoft

This model was previously named "PubMedBERT (abstracts)". You can either adopt the new model name "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract" or update your transformers library to version 4.22+ if you need to refer to the old name. Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. Recent work shows that for domains with abundant unlabeled text, such as biomedicine, pretraining…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

deberta-v3-large

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M…

Open weights mit 512 tokens transformers

View model