barthez by Moussa Kamal Eddine: Open-Weight Model

Model Card

By Moussa Kamal Eddine, published under apache-2.0, revision 8fe7a6d417fe.

A french sequence to sequence pretrained model based on BART. BARThez is pretrained by learning to reconstruct a corrupted input sentence. A corpus of 66GB of french raw text is used to carry out the pretraining. Unlike already existing BERT-based French language models such as CamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks (such as abstractive summarization), since not only its encoder but also its decoder is pretrained. In addition to BARThez that is pretrained from scratch, we continue the pretraining of a multilingual BART mBART which boosted its performance in both discriminative and generative tasks. We call the french adapted version mBARThez.

Read Moussa Kamal Eddine's full model card

A french sequence to sequence pretrained model based on BART.
BARThez is pretrained by learning to reconstruct a corrupted input sentence. A corpus of 66GB of french raw text is used to carry out the pretraining.
Unlike already existing BERT-based French language models such as CamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks (such as abstractive summarization), since not only its encoder but also its decoder is pretrained.

In addition to BARThez that is pretrained from scratch, we continue the pretraining of a multilingual BART mBART which boosted its performance in both discriminative and generative tasks. We call the french adapted version mBARThez.

Model	Architecture	#layers	#params
BARThez	BASE	12	165M
mBARThez	LARGE	24	458M

paper: https://arxiv.org/abs/2010.12321 \ github: https://github.com/moussaKam/BARThez

@article{eddine2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Eddine, Moussa Kamal and Tixier, Antoine J-P and Vazirgiannis, Michalis},
  journal={arXiv preprint arXiv:2010.12321},
  year={2020}
}

Configuration

Architecture: MBartForConditionalGeneration
Context length (tokens): 1,024
Layers: 6
Vocabulary size: 50,002
Model type: mbart

Identity and Version

Repository: moussaKam/barthez
Publisher: Moussa Kamal Eddine
Task: Fill mask
Modality: Text
Library: transformers
Parameters: Not stated by the source
Languages: fr
Revision: 8fe7a6d417fe6b5f6faca942fd9969a2b4b3eeae
First published: 2022-03-02
Last updated: 2024-02-20

Files and Weights

7 files, 560.7 MB in total. The weights are 1 file totalling 557.0 MB in bin.

Weights1 file · 557.0 MB

Configuration1 file · 1.3 KB

Tokenizer2 files · 2.6 MB

Documentation1 file · 1.7 KB

Other1 file · 1.1 MB

Repository1 file · 345 B

Every file

File	Type	Size	SHA-256
pytorch_model.bin	Weights	557.0 MB	5b5ce8af0cc3
config.json	Configuration	1.3 KB	—
README.md	Documentation	1.7 KB	—
sentencepiece.bpe.model	Other	1.1 MB	—
.gitattributes	Repository	345 B	—
tokenizer.json	Tokenizer	2.6 MB	—
tokenizer_config.json	Tokenizer	26 B	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 557.0 MB

Download from Moussa Kamal Eddine

Released by Moussa Kamal Eddine through its official repository on Hugging Face. Read the license.

Built From

Described by arXiv:2010.12321

Memory Requirements

Precision	Weights in memory
As published	557.0 MB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About barthez

Can I use barthez commercially?

Yes. barthez is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is barthez's context length?

1,024 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Fill mask

mdeberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

deberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

Bio_ClinicalBERT

Emily Alsentzer

The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (casedL-12H-768A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries. This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes. The BioClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words). Each note in MIMIC was first split…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

bert-large-portuguese-cased

NeuralMind Inteligência Artificial

BERTimbau Large is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large. For further information or requests, please go to BERTimbau repository. If you use our work, please cite

Open weights mit 512 tokens transformers

View model

Model · Fill mask

BiomedNLP-BiomedBERT-base-uncased-abstract

Microsoft

This model was previously named "PubMedBERT (abstracts)". You can either adopt the new model name "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract" or update your transformers library to version 4.22+ if you need to refer to the old name. Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. Recent work shows that for domains with abundant unlabeled text, such as biomedicine, pretraining…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

deberta-v3-large

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M…

Open weights mit 512 tokens transformers

View model