This model is a distilled version of the BERT base multilingual model. The code for the distillation process can be found here. This model is cased: it does make a difference between english and English. The model is trained on the concatenation of Wikipedia in 104 different languages listed here. The model has 6 layers, 768 dimension and 12 heads, totalizing 134M parameters (compared to 177M parameters for mBERT-base). On average, this model, referred to as DistilmBERT, is twice as fast as mBERT-base. We encourage potential users of this model to check out the BERT base multilingual model card to learn more about usage, limitations and potential biases. You can use the raw model for either…
Open weights
apache-2.0
135M parameters
512 tokens
transformers
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is case-sensitive: it makes a difference between english and English. Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team. RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels…
Open weights
mit
125M parameters
514 tokens
transformers
This is a microsoft/codebert-base-mlm model, trained for 1,000,000 steps (with batchsize=32) on Python code from the codeparrot/github-code-clean dataset, on the masked-language-modeling task. It is intended to be used in CodeBERTScore: https://github.com/neulab/code-bert-score, but can be used for any other model or task. If you use this model for research, please cite
Open weights
125M parameters
514 tokens
transformers
ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as: - Rotary Positional Embeddings (RoPE) for long-context support. - Local-Global Alternating Attention for efficiency on long inputs. - Unpadding and Flash Attention for efficient inference. ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it…
Open weights
apache-2.0
150M parameters
8,192 tokens
transformers
This model is a fine-tuned version of Itau-Unibanco/NorBERTo-base on an unknown dataset. It achieves the following results on the evaluation set: The following hyperparameters were used during training: - learningrate: 5e-05 - trainbatchsize: 2 - evalbatchsize: 4 - gradientaccumulationsteps: 16 - totaltrainbatchsize: 32 - lrschedulertype: linear - lrschedulerwarmupsteps: 500 - numepochs: 1.5 - mixedprecisiontraining: Native AMP - PEFT 0.20.0 - Transformers 5.16.1 - Pytorch 2.11.0+cu128 - Datasets 4.0.0 - Tokenizers 0.23.1
Open weights
cc-by-nc-sa-4.0
150M parameters
8,192 tokens
peft
CamemBERT is a state-of-the-art language model for French based on the RoBERTa model. It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains. CamemBERT was trained and evaluated by Louis Martin\, Benjamin Muller\, Pedro Javier Ortiz Suárez\, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot. If you use our work, please cite
Open weights
mit
111M parameters
514 tokens
transformers
