deberta-v2-large-japanese-char-wwm: Open-Weight Model

Runs On

What it takes to serve deberta-v2-large-japanese-char-wwm (330M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Weights	Memory needed	Cheapest setup	Per hour	Also fits
16-bit	0.7 GB	0.8 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	0.3 GB	0.4 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	0.2 GB	0.2 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Language Media Processing Lab at Kyoto University, published under cc-by-sa-4.0, revision 547b0e8b044f.

Model Card for Japanese character-level DeBERTa V2 large

Model description

This is a Japanese DeBERTa V2 large model pre-trained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR. This model is trained with character-level tokenization and whole word masking.

How to use

You can use this model for masked language modeling as follows:

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained('ku-nlp/deberta-v2-large-japanese-char-wwm')
model = AutoModelForMaskedLM.from_pretrained('ku-nlp/deberta-v2-large-japanese-char-wwm')

sentence = '京都大学で自然言語処理を[MASK][MASK]する。'
encoding = tokenizer(sentence, return_tensors='pt')
...

You can also fine-tune this model on downstream tasks.

Tokenization

There is no need to tokenize texts in advance, and you can give raw texts to the tokenizer. The texts are tokenized into character-level tokens by sentencepiece.

Training data

We used the following corpora for pre-training:

Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents)
Japanese portion of CC-100 (85GB, 619M sentences, 66M documents)
Japanese portion of OSCAR (54GB, 326M sentences, 25M documents)

Read the full model card (386 words)

Configuration

Architecture: DebertaV2ForMaskedLM
Context length (tokens): 512
Layers: 24
Hidden size: 1,024
Feed-forward size: 4,096
Attention heads: 16
Vocabulary size: 22,012
Stored precision: float16
Model type: deberta-v2

Identity and Version

Repository: ku-nlp/deberta-v2-large-japanese-char-wwm
Publisher: Language Media Processing Lab at Kyoto University
Task: Fill mask
Modality: Text
Library: transformers
Parameters: 330M parameters
Languages: ja
Revision: 547b0e8b044fba3f9b84d0ab9f990440bd130c8b
First published: 2023-03-09
Last updated: 2023-09-15

Files and Weights

8 files, 2.6 GB in total. The weights are 2 files totalling 2.6 GB in bin, safetensors.

Weights2 files · 2.6 GB

Configuration2 files · 1.0 KB

Tokenizer2 files · 88.7 KB

Documentation1 file · 3.5 KB

Repository1 file · 1.5 KB

Every file

File	Type	Size	SHA-256
model.safetensors	Weights	1.3 GB	2630f547d018
pytorch_model.bin	Weights	1.3 GB	bf0dab8ad87b
config.json	Configuration	895 B	—
special_tokens_map.json	Configuration	125 B	—
README.md	Documentation	3.5 KB	—
.gitattributes	Repository	1.5 KB	—
tokenizer_config.json	Tokenizer	520 B	—
vocab.txt	Tokenizer	88.2 KB	—

License and Download

License: cc-by-sa-4.0
Access: Open weights, no gate
Download size: 2.6 GB

Download from Language Media Processing Lab at Kyoto University

Released by Language Media Processing Lab at Kyoto University through its official repository on Hugging Face. Read the license.

Built From

Trained on (disclosed) cc100
Trained on (disclosed) oscar
Trained on (disclosed) wikipedia

Memory Requirements

Precision	Weights in memory
As published	2.6 GB
16-bit	0.7 GB
8-bit	0.3 GB
4-bit	0.2 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Compare deberta-v2-large-japanese-char-wwm

Questions About deberta-v2-large-japanese-char-wwm

How much GPU memory does deberta-v2-large-japanese-char-wwm need?

About 0.8 GB at 16-bit and 0.2 GB at 4-bit: the weights (330M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run deberta-v2-large-japanese-char-wwm on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use deberta-v2-large-japanese-char-wwm commercially?

Yes. deberta-v2-large-japanese-char-wwm is released under Creative Commons Attribution-ShareAlike 4.0. CC BY-SA 4.0 permits sharing and adapting, including commercially, with credit to the creator, and requires adaptations to be released under the same license.

What is deberta-v2-large-japanese-char-wwm's context length?

512 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Fill mask

bert-large-uncased

BERT community

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is uncased: it does not make a difference between english and English. Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by the Hugging Face team. BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from…

Open weights apache-2.0 336M parameters 512 tokens transformers

View model

Model · Fill mask

roberta-large

Facebook AI community

Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is case-sensitive: it makes a difference between english and English. Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team. RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels…

Open weights mit 355M parameters 514 tokens transformers

View model

Model · Fill mask

xlm-roberta-base

Facebook AI community

XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al. and first released in this repository. Disclaimer: The team releasing XLM-RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team. XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. RoBERTa is a transformers model pretrained on a large corpus in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in…

Open weights mit 279M parameters 514 tokens transformers

View model

Model · Fill mask

ModernBERT-large

Answer.AI

ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as: - Rotary Positional Embeddings (RoPE) for long-context support. - Local-Global Alternating Attention for efficiency on long inputs. - Unpadding and Flash Attention for efficient inference. ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it…

Open weights apache-2.0 396M parameters 8,192 tokens transformers

View model

Model · Fill mask

bert-base-multilingual-cased

BERT community

Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is case sensitive: it makes a difference between english and English. Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by the Hugging Face team. BERT is a transformers model pretrained on a large corpus of multilingual data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to…

Open weights apache-2.0 179M parameters 512 tokens transformers

View model

Model · Fill mask

bert-base-multilingual-uncased

BERT community

Pretrained model on the top 102 languages with the largest Wikipedia using a masked language modeling (MLM) objective. It was introduced in this paper and first released in this repository. This model is uncased: it does not make a difference between english and English. Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by the Hugging Face team. BERT is a transformers model pretrained on a large corpus of multilingual data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process…

Open weights apache-2.0 168M parameters 512 tokens transformers

View model