Minerva-MLM is a genome language model for coevolutionary mining. The checkpoint includes the masked language model head and three interaction heads for base-pairing, repeat, and structure signals. Full documentation, finetuning scripts, notebooks, and setup are on the github. Try it in the browser here. Inference from this repo needs only Transformers and PyTorch: flash-attn is picked up automatically when present, otherwise Minerva falls back to PyTorch SDPA. The optional minerva-dna package adds plotting helpers, GenBank utilities and finetuning wrappers, and lets you load the model without trustremotecode: With minerva-dna installed, import the class directly and drop trustremotecode…
Open weights
apache-2.0
671M parameters
transformers
Minerva-MLM is a genome language model for coevolutionary mining. The checkpoint includes the masked language model head and three interaction heads for base-pairing, repeat, and structure signals. Full documentation, finetuning scripts, notebooks, and setup are on the github. Try it in the browser here. Inference from this repo needs only Transformers and PyTorch: flash-attn is picked up automatically when present, otherwise Minerva falls back to PyTorch SDPA. The optional minerva-dna package adds plotting helpers, GenBank utilities and finetuning wrappers, and lets you load the model without trustremotecode: With minerva-dna installed, import the class directly and drop trustremotecode…
Open weights
apache-2.0
671M parameters
transformers
XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al. and first released in this repository. Disclaimer: The team releasing XLM-RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team. XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. RoBERTa is a transformers model pretrained on a large corpus in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in…
Open weights
mit
561M parameters
514 tokens
transformers
ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as: - Rotary Positional Embeddings (RoPE) for long-context support. - Local-Global Alternating Attention for efficiency on long inputs. - Unpadding and Flash Attention for efficient inference. ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it…
Open weights
apache-2.0
396M parameters
8,192 tokens
transformers
This model is a fine-tuned version of ModernBERT optimized for logical reasoning, deductive analysis, and structure-based token prediction. It relies on the ModernBertForMaskedLM architecture, making it highly effective at filling in missing contextual logic tokens (fill-mask). - As an encoder-based Masked Language Model, it is not designed for long-form generative text (like ChatGPT). It excels at predicting masked tokens inside structured prompts.
Open weights
apache-2.0
396M parameters
8,192 tokens
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is case-sensitive: it makes a difference between english and English. Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team. RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels…
Open weights
mit
355M parameters
514 tokens
transformers
