longformer-base-4096 is an open-weight model from Ai2, released under Apache License 2.0. It has 4,098-token context. Its published files total 2.1 GB. It draws 1M downloads a month.
Longformer is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096.
SAVRN's Notes on longformer-base-4096
Longformer's base model: RoBERTa retrained for masked-language-modeling on long documents, handling up to 4,098 tokens. The repository is 2.1 GB holding three framework copies — a PyTorch bin, a TensorFlow h5 and a Rust ot file — so download the one your stack reads. Twelve layers, a 768 hidden size, a 50,265-word vocabulary. Apache 2.0 and 1M downloads a month; the paper is arXiv 2004.05150.
In 2022 the pitch was global-plus-local attention that scaled linearly with sequence length; in 2026 the honest question is whether you need the architecture at all, since mainstream models now carry far longer native context than 4,098 tokens. This checkpoint earns its keep as a fine-tuning base for long-document work in a legacy stack, not as the front end of a new pipeline. With no parameter count published there is no precision table — 2.1 GB as published is the sizing number.
Model Card
By Ai2, published under apache-2.0, revision 301e6a42cb0d.
Longformer is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations. Please refer to the examples in modelinglongformer.py and the paper for more details on how to set global attention. If you use Longformer in your research, please cite Longformer: The Long-Document Transformer. Longformer is an open-source project developed by the Allen…
Read Ai2's full model card
Longformer is a transformer model for long documents.
longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096.
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in modeling_longformer.py and the paper for more details on how to set global attention.
Citing
If you use Longformer in your research, please cite Longformer: The Long-Document Transformer.
@article{Beltagy2020Longformer,
title={Longformer: The Long-Document Transformer},
author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
journal={arXiv:2004.05150},
year={2020},
}
Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
Configuration
- Context length (tokens)
- 4,098
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 50,265
- Model type
- longformer
Identity and Version
- Repository
- allenai/longformer-base-4096
- Publisher
- Ai2
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- 301e6a42cb0d9976a6d6a26a079fef81c18aa895
- First published
- 2022-03-02
- Last updated
- 2023-04-05
Files and Weights
9 files, 2.1 GB in total. The weights are 3 files totalling 2.1 GB in bin, h5, ot.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 597.3 MB | 06c56757f051 |
| rust_model.ot | Weights | 751.7 MB | db996b83d7a4 |
| tf_model.h5 | Weights | 764.6 MB | 885545a75c74 |
| config.json | Configuration | 694 B | — |
| README.md | Documentation | 1.3 KB | — |
| .gitattributes | Repository | 345 B | — |
| merges.txt | Tokenizer | 456.3 KB | — |
| tokenizer.json | Tokenizer | 1.4 MB | — |
| vocab.json | Tokenizer | 898.8 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 2.1 GB
Released by Ai2 through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2004.05150
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 2.1 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About longformer-base-4096
Can I use longformer-base-4096 commercially?
Yes. longformer-base-4096 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is longformer-base-4096's context length?
4,098 tokens, from the maximum position embeddings in its published configuration.