Kuleshov Group

Model · Fill mask

caduceus-ph_seqlen-131k_d_model-256_n_layer-16

Kuleshov Group

To use the pre-trained model for masked language modeling, use the following snippet: Alternatively, you can instantiate a model from scratch to train on your own data as follows: This is the Caduceus-Ph model with hidden dimension 256 and 16 MambaDNA layers. This model is not inherently reverse complement (RC) equivariant. Rather, it was pre-trained using RC data augmentation. Its intended usage is as follows: for downstream tasks, the model should be trained with RC data augmentation. At downstream task inference, the model should be run twice: once on a sequence and once on its RC. The output of these two applications should be combined (averaged) to form the downstream task prediction.…

Model · Fill mask

caduceus-ps_seqlen-131k_d_model-256_n_layer-16

Kuleshov Group

To use the pre-trained model for masked language modeling, use the following snippet: Alternatively, you can instantiate a model from scratch to train on your own data as follows: This is the Caduceus-PS model with hidden dimension 256 and 16 MambaDNA layers. This model is reverse complement (RC) equivariant and thus no RC data augmentation is required when training this model, either during pre-training or for downstream fine-tuning. Note that the model hidden state will be twice that of a non-RC equivariant counterpart. For downstream task training and inference, and to ensure RC invariant outputs at downstream time, one can either run the downstream model on the hidden state and its RC…