# caduceus-ph_seqlen-131k_d_model-256_n_layer-16 by Kuleshov Group Source: https://savrn.com/models/caduceus-ph-seqlen-131k-d-model-256-n-layer-16 Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve caduceus-ph_seqlen-131k_d_model-256_n_layer-16 (8M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 0.0 GB | 0.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 0.0 GB | 0.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 0.0 GB | 0.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 23, 2026. [caduceus-ph_seqlen-131k_d_model-256_n_layer-16 on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/caduceus-ph-seqlen-131k-d-model-256-n-layer-16/gpus) ## SAVRN's Notes on caduceus-ph_seqlen-131k_d_model-256_n_layer-16 Caduceus is a DNA language model, and the reason it matters is sequence length: it is built to read 131,000 base pairs at once and to treat a strand and its reverse complement consistently. This PH variant from the Kuleshov Group is tiny at 8M parameters, so the cost of experimenting is close to nothing. Apache 2.0 allows commercial use. This is research infrastructure rather than a finished tool; you will fine-tune it on your own genomic task. Compare it with the PS variant from the same group, which enforces reverse-complement symmetry in a different way, and pick whichever holds up on your data. SAVRN Research, 2026-09-18 ## Model Card By Kuleshov Group, published under apache-2.0, revision b0477522ac5d. ### Using Caduceus To use the pre-trained model for masked language modeling, use the following snippet: ``` from transformers import AutoModelForMaskedLM, AutoTokenizer # See the `Caduceus` collection page on the hub for list of available models. model_name = "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForMaskedLM.from_pretrained(model_name) ``` Alternatively, you can instantiate a model from scratch to train on your own data as follows: ``` from transformers import AutoConfig, AutoModelForMaskedLM # Add any config overrides here, see the `config.json` file on the hub for details. config_overrides = {} # See the `Caduceus` collection page on the hub for list of available models. config = AutoConfig.from_pretrained( "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16", **config_overrides, ) model = AutoModelForMaskedLM.from_config(config) ``` ### Model Details [Read the full model card (275 words)](https://savrn.com/models/caduceus-ph-seqlen-131k-d-model-256-n-layer-16/card) ## Configuration Architecture CaduceusForMaskedLM Vocabulary size 16 Stored precision float32 Model type caduceus ## Identity and Version Repository kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16 Publisher Kuleshov Group Task Fill mask Modality Text Library transformers Parameters 8M parameters Languages Not stated by the source Revision b0477522ac5d044ad03578aa724ec8e4bdbd405b First published 2024-02-26 Last updated 2025-10-20 ## Files and Weights 10 files, 31.0 MB in total. The weights are 1 file totalling 30.9 MB in safetensors. Weights1 file · 30.9 MB Configuration6 files · 47.4 KB Tokenizer1 file · 1.5 KB Documentation1 file · 2.3 KB Repository1 file · 1.5 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | model.safetensors | Weights | 30.9 MB | 1ab95562b7ee | | config.json | Configuration | 1.4 KB | — | | configuration_caduceus.py | Configuration | 2.0 KB | — | | modeling_caduceus.py | Configuration | 28.9 KB | — | | modeling_rcps.py | Configuration | 10.0 KB | — | | special_tokens_map.json | Configuration | 173 B | — | | tokenization_caduceus.py | Configuration | 5.0 KB | — | | README.md | Documentation | 2.3 KB | — | | .gitattributes | Repository | 1.5 KB | — | | tokenizer_config.json | Tokenizer | 1.5 KB | — | ## License and Download License apache-2.0 Access Open weights, no gate Download size 30.9 MB [Download from Kuleshov Group](https://huggingface.co/kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16) Released by Kuleshov Group through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0). ## Built From - Described by [arXiv:2403.03234](https://savrn.com/papers/caduceus-bi-directional-equivariant-long-range-dna-sequence-modeling) ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 30.9 MB | | 16-bit | 0.0 GB | | 8-bit | 0.0 GB | | 4-bit | 0.0 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Questions About caduceus-ph_seqlen-131k_d_model-256_n_layer-16 ### How much GPU memory does caduceus-ph_seqlen-131k_d_model-256_n_layer-16 need? About 0 GB at 16-bit and 0 GB at 4-bit: the weights (8M parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run caduceus-ph_seqlen-131k_d_model-256_n_layer-16 on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use caduceus-ph_seqlen-131k_d_model-256_n_layer-16 commercially? Yes. caduceus-ph_seqlen-131k_d_model-256_n_layer-16 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ## Similar Models Model · Fill mask ### [caduceus-ps_seqlen-131k_d_model-256_n_layer-16](https://savrn.com/models/caduceus-ps-seqlen-131k-d-model-256-n-layer-16) [Kuleshov Group](https://savrn.com/model-publishers/kuleshov-group) To use the pre-trained model for masked language modeling, use the following snippet: Alternatively, you can instantiate a model from scratch to train on your own data as follows: This is the Caduceus-PS model with hidden dimension 256 and 16 MambaDNA layers. This model is reverse complement (RC) equivariant and thus no RC data augmentation is required when training this model, either during pre-training or for downstream fine-tuning. Note that the model hidden state will be twice that of a non-RC equivariant counterpart. For downstream task training and inference, and to ensure RC invariant outputs at downstream time, one can either run the downstream model on the hidden state and its RC… Open weights apache-2.0 8M parameters transformers [View model](https://savrn.com/models/caduceus-ps-seqlen-131k-d-model-256-n-layer-16) Model · Fill mask ### [esm2_t6_8M_UR50D](https://savrn.com/models/esm2-t6-8m-ur50d) [AI at Meta](https://savrn.com/model-publishers/facebook) ESM-2 is a state-of-the-art protein model trained on a masked language modelling objective. It is suitable for fine-tuning on a wide range of tasks that take protein sequences as input. For detailed information on the model architecture and training data, please refer to the accompanying paper. You may also be interested in some demo notebooks (PyTorch, TensorFlow) which demonstrate how to fine-tune ESM-2 models on your tasks of interest. Several ESM-2 checkpoints are available in the Hub with varying sizes. Larger sizes generally have somewhat better accuracy, but require much more memory and time to train Open weights mit 8M parameters 1,026 tokens transformers [View model](https://savrn.com/models/esm2-t6-8m-ur50d) Model · Fill mask ### [albert-base-v2](https://savrn.com/models/albert-base-v2) [ALBERT community](https://savrn.com/model-publishers/albert) Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model, as all ALBERT models, is uncased: it does not make a difference between english and English. Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by the Hugging Face team. ALBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate… Open weights apache-2.0 12M parameters 512 tokens transformers [View model](https://savrn.com/models/albert-base-v2) Model · Fill mask ### [albert-base-v2-persian-zwnj-shiraz-mlm](https://savrn.com/models/albert-base-v2-persian-zwnj-shiraz-mlm) [Shekar AI](https://savrn.com/model-publishers/shekar-ai) ALBERT-base-v2 adapted to Persian with a freshly trained 38,016-token SentencePiece unigram tokenizer. English Wikipedia supplies additional English text. ZWNJ (U+200C, Persian half-space) is preserved as a standalone token, ID 5. Initialized from albert/albert-base-v2, with word embeddings and the tied vocabulary head reinitialized for the new vocabulary. Trained with 15% masked-language modeling on normalized shekar-ai/Shiraz and English Wikipedia (target 10% of corpus characters). Persian text is normalized with shekar.Normalizer before tokenization; the tokenizer itself only normalizes whitespace. The tokenizer was trained from up to 6 million sampled corpus lines. validation gains… Open weights mit 12M parameters 512 tokens transformers [View model](https://savrn.com/models/albert-base-v2-persian-zwnj-shiraz-mlm) Model · Fill mask ### [mdeberta-v3-base](https://savrn.com/models/mdeberta-v3-base) [Microsoft](https://savrn.com/model-publishers/microsoft) DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/mdeberta-v3-base) Model · Fill mask ### [deberta-v3-base](https://savrn.com/models/deberta-v3-base) [Microsoft](https://savrn.com/model-publishers/microsoft) DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data. In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper. Please check the official repository for more implementation details and updates. The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M… Open weights mit 512 tokens transformers [View model](https://savrn.com/models/deberta-v3-base) ## Kuleshov Group [All models and datasets](https://savrn.com/model-publishers/kuleshov-group) ## Versions - [b0477522ac5d](https://savrn.com/models/caduceus-ph-seqlen-131k-d-model-256-n-layer-16/versions/b0477522ac5d) · current 2026-09-18 ## Explore More - [All fill mask models](https://savrn.com/models/tasks/fill-mask) - [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-20. - [Hugging Face record](https://huggingface.co/kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16) - [How the hub is built](https://savrn.com/model-hub/methodology)