caduceus-ph_seqlen-131k_d_m…256_n_layer-16 by Kuleshov Group

Runs On

What it takes to serve caduceus-ph_seqlen-131k_d_model-256_n_layer-16 (8M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Cheapest setup	Per hour	Also fits
16-bit	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 9, 2026.

caduceus-ph_seqlen-131k_d_model-256_n_layer-16 on every accelerator the SAVRN Index prices, at every precision

SAVRN's Notes on caduceus-ph_seqlen-131k_d_model-256_n_layer-16

Caduceus is a DNA language model, and the reason it matters is sequence length: it is built to read 131,000 base pairs at once and to treat a strand and its reverse complement consistently. This PH variant from the Kuleshov Group is tiny at 8M parameters, so the cost of experimenting is close to nothing.

Apache 2.0 allows commercial use. This is research infrastructure rather than a finished tool; you will fine-tune it on your own genomic task. Compare it with the PS variant from the same group, which enforces reverse-complement symmetry in a different way, and pick whichever holds up on your data.

SAVRN Research, 2026-09-18

Model Card

By Kuleshov Group, published under apache-2.0, revision b0477522ac5d.

Using Caduceus

To use the pre-trained model for masked language modeling, use the following snippet:

from transformers import AutoModelForMaskedLM, AutoTokenizer

# See the `Caduceus` collection page on the hub for list of available models.
model_name = "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

Alternatively, you can instantiate a model from scratch to train on your own data as follows:

from transformers import AutoConfig, AutoModelForMaskedLM

# Add any config overrides here, see the `config.json` file on the hub for details.
config_overrides = {}
# See the `Caduceus` collection page on the hub for list of available models.
config = AutoConfig.from_pretrained(
 "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16",
 **config_overrides,
) 
model = AutoModelForMaskedLM.from_config(config)

Model Details

Read the full model card (275 words)

Configuration

Architecture: CaduceusForMaskedLM
Vocabulary size: 16
Stored precision: float32
Model type: caduceus

Identity and Version

Repository: kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16
Publisher: Kuleshov Group
Task: Fill mask
Modality: Text
Library: transformers
Parameters: 8M parameters
Languages: Not stated by the source
Revision: b0477522ac5d044ad03578aa724ec8e4bdbd405b
First published: 2024-02-26
Last updated: 2025-10-20

Files and Weights

10 files, 31.0 MB in total. The weights are 1 file totalling 30.9 MB in safetensors.

Weights1 file · 30.9 MB

Configuration6 files · 47.4 KB

Tokenizer1 file · 1.5 KB

Documentation1 file · 2.3 KB

Repository1 file · 1.5 KB

Every file

File	Type	Size	SHA-256
model.safetensors	Weights	30.9 MB	1ab95562b7ee
config.json	Configuration	1.4 KB	—
configuration_caduceus.py	Configuration	2.0 KB	—
modeling_caduceus.py	Configuration	28.9 KB	—
modeling_rcps.py	Configuration	10.0 KB	—
special_tokens_map.json	Configuration	173 B	—
tokenization_caduceus.py	Configuration	5.0 KB	—
README.md	Documentation	2.3 KB	—
.gitattributes	Repository	1.5 KB	—
tokenizer_config.json	Tokenizer	1.5 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 30.9 MB

Download from Kuleshov Group

Released by Kuleshov Group through its official repository on Hugging Face. Read the license.

Built From

Described by arXiv:2403.03234

Memory Requirements

Precision	Weights in memory
As published	30.9 MB
16-bit	0.0 GB
8-bit	0.0 GB
4-bit	0.0 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About caduceus-ph_seqlen-131k_d_model-256_n_layer-16

How much GPU memory does caduceus-ph_seqlen-131k_d_model-256_n_layer-16 need?

About 0 GB at 16-bit and 0 GB at 4-bit: the weights (8M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run caduceus-ph_seqlen-131k_d_model-256_n_layer-16 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use caduceus-ph_seqlen-131k_d_model-256_n_layer-16 commercially?

Yes. caduceus-ph_seqlen-131k_d_model-256_n_layer-16 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Model · Fill mask

caduceus-ps_seqlen-131k_d_model-256_n_layer-16

Kuleshov Group

To use the pre-trained model for masked language modeling, use the following snippet: Alternatively, you can instantiate a model from scratch to train on your own data as follows: This is the Caduceus-PS model with hidden dimension 256 and 16 MambaDNA layers. This model is reverse complement (RC) equivariant and thus no RC data augmentation is required when training this model, either during pre-training or for downstream fine-tuning. Note that the model hidden state will be twice that of a non-RC equivariant counterpart. For downstream task training and inference, and to ensure RC invariant outputs at downstream time, one can either run the downstream model on the hidden state and its RC…

Open weights apache-2.0 8M parameters transformers

View model

Model · Fill mask

esm2_t6_8M_UR50D

AI at Meta

ESM-2 is a state-of-the-art protein model trained on a masked language modelling objective. It is suitable for fine-tuning on a wide range of tasks that take protein sequences as input. For detailed information on the model architecture and training data, please refer to the accompanying paper. You may also be interested in some demo notebooks (PyTorch, TensorFlow) which demonstrate how to fine-tune ESM-2 models on your tasks of interest. Several ESM-2 checkpoints are available in the Hub with varying sizes. Larger sizes generally have somewhat better accuracy, but require much more memory and time to train

Open weights mit 8M parameters 1,026 tokens transformers

View model

Model · Fill mask

albert-base-v2

ALBERT community

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model, as all ALBERT models, is uncased: it does not make a difference between english and English. Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by the Hugging Face team. ALBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate…

Open weights apache-2.0 12M parameters 512 tokens transformers

View model

Model · Fill mask

albert-base-v2-persian-zwnj-shiraz-mlm

Shekar AI

ALBERT-base-v2 adapted to Persian with a freshly trained 38,016-token SentencePiece unigram tokenizer. English Wikipedia supplies additional English text. ZWNJ (U+200C, Persian half-space) is preserved as a standalone token, ID 5. Initialized from albert/albert-base-v2, with word embeddings and the tied vocabulary head reinitialized for the new vocabulary. Trained with 15% masked-language modeling on normalized shekar-ai/Shiraz and English Wikipedia (target 10% of corpus characters). Persian text is normalized with shekar.Normalizer before tokenization; the tokenizer itself only normalizes whitespace. The tokenizer was trained from up to 6 million sampled corpus lines. validation gains…

Open weights mit 12M parameters 512 tokens transformers

View model

Model · Fill mask

mdeberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was…

Open weights mit 512 tokens transformers

View model

Model · Fill mask

deberta-v3-base

Microsoft

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M…

Open weights mit 512 tokens transformers

View model