TrOCR model fine-tuned on the IAM dataset. It was introduced in the paper TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Li et al. and first released in this repository. Disclaimer: The team releasing TrOCR did not write a model card for this model so this model card has been written by the Hugging Face team. The TrOCR model is an encoder-decoder model, consisting of an image Transformer as encoder, and a text Transformer as decoder. The image encoder was initialized from the weights of BEiT, while the text decoder was initialized from the weights of RoBERTa. Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which…
Open weights
mit
333M parameters
transformers
Nougat model trained on PDF-to-markdown. It was introduced in the paper Nougat: Neural Optical Understanding for Academic Documents by Blecher et al. and first released in this repository. Disclaimer: The team releasing Nougat did not write a model card for this model so this model card has been written by the Hugging Face team. Note: this model corresponds to the "0.1.0-base" version of the original repository. Nougat is a Donut model trained to transcribe scientific PDFs into an easy-to-use markdown format. The model consists of a Swin Transformer as vision encoder, and an mBART model as text decoder. The model is trained to autoregressively predict the markdown given only the pixels of…
Open weights
cc-by-nc-4.0
349M parameters
transformers
This model is the pretrained version of Pix2Struct, use this model for fine-tuning purposes only. Pix2Struct is an image encoder - text decoder model that is trained on image-text pairs for various tasks, including image captionning and visual question answering. The full list of available models can be found on the Table 1 of the paper: The abstract of the model states that: forms. Perhaps due to this diversity, previous work has typically relied on domainspecific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks…
Open weights
apache-2.0
282M parameters
transformers
An HTR model for historical Swedish developed by the Swedish National Archives in collaboration with the Stockholm City Archives, the Finnish National Archives and Jämtlands Fornskriftsällskap. The model is trained on Swedish handwriting from the period 1600-1900. The model is trained on Swedish running-text handwriting dating from the start of the 17th century to the end of the 19th century. Like most current HTR models it operates on a text-line level, so its intended use is within an HTR pipeline that segments the text into text lines, which are transcribed by the model. The model can be used without fine-tuning on all handwriting but performs best on the type of handwriting it was…
Open weights
apache-2.0
385M parameters
htrflow
captioning pretrained on COCO dataset - base architecture (with ViT large backbone). Authors from the paper write in the abstract: Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the…
Open weights
bsd-3-clause
470M parameters
512 tokens
transformers
GIT (short for GenerativeImage2Text) model, base-sized version. It was introduced in the paper GIT: A Generative Image-to-text Transformer for Vision and Language by Wang et al. and first released in this repository. Disclaimer: The team releasing GIT did not write a model card for this model so this model card has been written by the Hugging Face team. GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a lot of (image, text) pairs. The goal for the model is simply to predict the next text token, giving the image tokens and previous text tokens. The model has full access to (i.e. a bidirectional attention mask…
Open weights
mit
177M parameters
1,024 tokens
transformers
