MobileBERT is a thin version of BERTLARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. This checkpoint is the original MobileBert Optimized Uncased English: checkpoint.
Model Card
By Google, published under apache-2.0, revision 1f90a6c24c78.
MobileBERT is a thin version of BERTLARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. This checkpoint is the original MobileBert Optimized Uncased English: checkpoint.
Read Google's full model card
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.
This checkpoint is the original MobileBert Optimized Uncased English: uncased_L-24_H-128_B-512_A-4_F-4_OPT checkpoint.
How to use MobileBERT in transformers
from transformers import pipeline
fill_mask = pipeline(
"fill-mask",
model="google/mobilebert-uncased",
tokenizer="google/mobilebert-uncased"
)
print(
fill_mask(f"HuggingFace is creating a {fill_mask.tokenizer.mask_token} that the community uses to solve NLP tasks.")
)
Configuration
- Architecture
- MobileBertForPreTraining
- Context length (tokens)
- 512
- Layers
- 24
- Hidden size
- 512
- Feed-forward size
- 512
- Attention heads
- 4
- Vocabulary size
- 30,522
- Model type
- mobilebert
Identity and Version
- Repository
- google/mobilebert-uncased
- Publisher
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 1f90a6c24c7879273a291d34a849033eba2dbc0f
- First published
- 2022-03-02
- Last updated
- 2021-04-19
Files and Weights
8 files, 473.4 MB in total. The weights are 3 files totalling 472.7 MB in bin, h5, ot.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 146.9 MB | e3468adf28d7 |
| rust_model.ot | Weights | 162.3 MB | e9674eaf1b96 |
| tf_model.h5 | Weights | 163.5 MB | 854d095950fa |
| config.json | Configuration | 847 B | — |
| README.md | Documentation | 918 B | — |
| .gitattributes | Repository | 447 B | — |
| tokenizer.json | Tokenizer | 466.1 KB | — |
| vocab.txt | Tokenizer | 231.5 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 472.7 MB
Released by Google through Kaggle. Read the license.
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 472.7 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About mobilebert-uncased
Can I use mobilebert-uncased commercially?
Yes. mobilebert-uncased is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is mobilebert-uncased's context length?
512 tokens, from the maximum position embeddings in its published configuration.