ELECTRA is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute.
SAVRN's Notes on electra-base-discriminator
No parameter count, no memory line and no task label appear on this page. Google published a discriminator from its ELECTRA pre-training method, taught to tell real input tokens from fakes planted by a second network, so what you get is an encoder to fine-tune, not a model to serve. For hardware, work from twelve layers, a 768 hidden size and 1.76 GB on disk in four formats; with no cheapest-setup figure to quote, load it on a card you already own and measure.
Under Apache 2.0 a fine-tuned derivative can go into a commercial product; the license, copyright notices and any NOTICE file go wherever the weights go, significant changes get stated, and the express patent grant from contributors is worth having on a model you retrain. Inputs must fit the 512-token context, and the evaluation is yours to run, since this page lists none.
Model Card
By Google, published under apache-2.0, revision 1ae76a97c7e8.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
ELECTRA is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the SQuAD 2.0 dataset.
For a detailed description and experimental results, please refer to our paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.
This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. GLUE), QA tasks (e.g., SQuAD), and sequence tagging tasks (e.g., text chunking).
How to use the discriminator in transformers
Configuration
- Architecture
- ElectraForPreTraining
- Context length (tokens)
- 512
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 30,522
- Model type
- electra
Identity and Version
- Repository
- google/electra-base-discriminator
- Publisher
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 1ae76a97c7e84a4e640876a07453fccd636f0667
- First published
- 2022-03-02
- Last updated
- 2024-02-29
Files and Weights
10 files, 1.8 GB in total. The weights are 4 files totalling 1.8 GB in bin, h5, msgpack, ot.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| flax_model.msgpack | Weights | 437.9 MB | c4971c644428 |
| pytorch_model.bin | Weights | 440.3 MB | ff381b988006 |
| rust_model.ot | Weights | 440.3 MB | 72b22ccb8e2b |
| tf_model.h5 | Weights | 438.2 MB | bbd846dba75a |
| config.json | Configuration | 666 B | — |
| README.md | Documentation | 2.2 KB | — |
| .gitattributes | Repository | 390 B | — |
| tokenizer.json | Tokenizer | 466.1 KB | — |
| tokenizer_config.json | Tokenizer | 48 B | — |
| vocab.txt | Tokenizer | 231.5 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 1.8 GB
Released by Google through Kaggle. Read the license.
Built From
- Described by arXiv:1406.2661
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 1.8 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Built on This Model
- Quantized fromms-marco-electra-base
- Derived fromms-marco-electra-base
- Quantized fromqnli-electra-base
- Derived fromqnli-electra-base
- Derived fromelectra-base-discriminator-nvidia-aegis-v2-augmented
Questions About electra-base-discriminator
Can I use electra-base-discriminator commercially?
Yes. electra-base-discriminator is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is electra-base-discriminator's context length?
512 tokens, from the maximum position embeddings in its published configuration.