Longformer is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096.
Model Card
By Ai2, published under apache-2.0, revision 301e6a42cb0d.
Longformer is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations. Please refer to the examples in modelinglongformer.py and the paper for more details on how to set global attention. If you use Longformer in your research, please cite Longformer: The Long-Document Transformer. Longformer is an open-source project developed by the Allen…
Read Ai2's full model card
Longformer is a transformer model for long documents.
longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096.
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in modeling_longformer.py and the paper for more details on how to set global attention.
Citing
If you use Longformer in your research, please cite Longformer: The Long-Document Transformer.
@article{Beltagy2020Longformer,
title={Longformer: The Long-Document Transformer},
author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
journal={arXiv:2004.05150},
year={2020},
}
Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
Configuration
- Context length (tokens)
- 4,098
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 50,265
- Model type
- longformer
Identity and Version
- Repository
- allenai/longformer-base-4096
- Publisher
- Ai2
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 301e6a42cb0d9976a6d6a26a079fef81c18aa895
- First published
- 2022-03-02
- Last updated
- 2023-04-05
Files and Weights
9 files, 2.1 GB in total. The weights are 3 files totalling 2.1 GB in bin, h5, ot.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 597.3 MB | 06c56757f051 |
| rust_model.ot | Weights | 751.7 MB | db996b83d7a4 |
| tf_model.h5 | Weights | 764.6 MB | 885545a75c74 |
| config.json | Configuration | 694 B | — |
| README.md | Documentation | 1.3 KB | — |
| .gitattributes | Repository | 345 B | — |
| merges.txt | Tokenizer | 456.3 KB | — |
| tokenizer.json | Tokenizer | 1.4 MB | — |
| vocab.json | Tokenizer | 898.8 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 2.1 GB
Released by Ai2 through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2004.05150
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 2.1 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About longformer-base-4096
Can I use longformer-base-4096 commercially?
Yes. longformer-base-4096 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is longformer-base-4096's context length?
4,098 tokens, from the maximum position embeddings in its published configuration.