# longformer-base-4096 by Ai2: Open-Weight Model Source: https://savrn.com/models/longformer-base-4096 Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## SAVRN's Notes on longformer-base-4096 Longformer's base model: RoBERTa retrained for masked-language-modeling on long documents, handling up to 4,098 tokens. The repository is 2.1 GB holding three framework copies — a PyTorch bin, a TensorFlow h5 and a Rust ot file — so download the one your stack reads. Twelve layers, a 768 hidden size, a 50,265-word vocabulary. Apache 2.0 and 1M downloads a month; the paper is arXiv 2004.05150. In 2022 the pitch was global-plus-local attention that scaled linearly with sequence length; in 2026 the honest question is whether you need the architecture at all, since mainstream models now carry far longer native context than 4,098 tokens. This checkpoint earns its keep as a fine-tuning base for long-document work in a legacy stack, not as the front end of a new pipeline. With no parameter count published there is no precision table — 2.1 GB as published is the sizing number. SAVRN Research ## Model Card By Ai2, published under apache-2.0, revision 301e6a42cb0d. Longformer is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations. Please refer to the examples in modelinglongformer.py and the paper for more details on how to set global attention. If you use Longformer in your research, please cite Longformer: The Long-Document Transformer. Longformer is an open-source project developed by the Allen… Read Ai2's full model card [Longformer](https://arxiv.org/abs/2004.05150) is a transformer model for long documents. longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations. Please refer to the examples in modeling_longformer.py and the paper for more details on how to set global attention. ### Citing If you use Longformer in your research, please cite [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150). ``` @article{Beltagy2020Longformer, title={Longformer: The Long-Document Transformer}, author={Iz Beltagy and Matthew E. Peters and Arman Cohan}, journal={arXiv:2004.05150}, year={2020}, } ``` Longformer is an open-source project developed by [the Allen Institute for Artificial Intelligence (AI2)](http://www.allenai.org). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering. ## Configuration Context length (tokens) 4,098 Layers 12 Hidden size 768 Feed-forward size 3,072 Attention heads 12 Vocabulary size 50,265 Model type longformer ## Identity and Version Repository allenai/longformer-base-4096 Publisher Ai2 Task Not stated by the source Modality Other Library transformers Parameters Not stated by the source Languages en Revision 301e6a42cb0d9976a6d6a26a079fef81c18aa895 First published 2022-03-02 Last updated 2023-04-05 ## Files and Weights 9 files, 2.1 GB in total. The weights are 3 files totalling 2.1 GB in bin, h5, ot. Weights3 files · 2.1 GB Configuration1 file · 694 B Tokenizer3 files · 2.7 MB Documentation1 file · 1.3 KB Repository1 file · 345 B Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | pytorch_model.bin | Weights | 597.3 MB | 06c56757f051 | | rust_model.ot | Weights | 751.7 MB | db996b83d7a4 | | tf_model.h5 | Weights | 764.6 MB | 885545a75c74 | | config.json | Configuration | 694 B | — | | README.md | Documentation | 1.3 KB | — | | .gitattributes | Repository | 345 B | — | | merges.txt | Tokenizer | 456.3 KB | — | | tokenizer.json | Tokenizer | 1.4 MB | — | | vocab.json | Tokenizer | 898.8 KB | — | ## License and Download License apache-2.0 Access Open weights, no gate Download size 2.1 GB [Download from Ai2](https://huggingface.co/allenai/longformer-base-4096) Released by Ai2 through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0). ## Built From - Described by arXiv:2004.05150 ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 2.1 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Questions About longformer-base-4096 ### Can I use longformer-base-4096 commercially? Yes. longformer-base-4096 is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ### What is longformer-base-4096's context length? 4,098 tokens, from the maximum position embeddings in its published configuration. ## Ai2 [All models and datasets](https://savrn.com/model-publishers/allenai) ## Versions - [301e6a42cb0d](https://savrn.com/models/longformer-base-4096/versions/301e6a42cb0d) · current 2026-09-18 ## Explore More - [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-19. - [Ai2 website](https://allenai.org/) - [Hugging Face record](https://huggingface.co/allenai/longformer-base-4096) - [How the hub is built](https://savrn.com/model-hub/methodology)