Optical character recognition for Japanese text, with the main focus being Japanese manga. It uses Vision Encoder Decoder framework. Manga OCR can be used as a general purpose printed Japanese OCR, but its main goal was to provide a high quality text recognition, robust against various scenarios specific to manga: - both vertical and horizontal text - text with furigana - text overlaid on images - wide variety of fonts and font styles - low quality images Code is available here.
Open-weight model · Image to text
blip-image-captioning-base
by Salesforce AI Research Salesforce/blip-image-captioning-base
captioning pretrained on COCO dataset - base architecture (with ViT base backbone). Authors from the paper write in the abstract: Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks.
Parameters—
Context512
Weights2.0 GB
Licensebsd-3-clause
AccessOpen weights
Monthly Downloads1.7M
Model Card
By Salesforce AI Research, published under bsd-3-clause, revision 82a37760796d.
captioning pretrained on COCO dataset - base architecture (with ViT base backbone). Authors from the paper write in the abstract: Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the…
Read Salesforce AI Research's full model card
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Model card for image captioning pretrained on COCO dataset - base architecture (with ViT base backbone).
| Pull figure from BLIP official repo |
TL;DR
Authors from the paper write in the abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.
Usage
You can use this model for conditional and un-conditional image captioning
Using the Pytorch model
Running the model on CPU
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
Running the model on GPU
In full precision
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base").to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
In half precision (float16)
Click to expand
import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base", torch_dtype=torch.float16).to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
Ethical Considerations
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.
BibTex and citation info
@misc{https://doi.org/10.48550/arxiv.2201.12086,
doi = {10.48550/ARXIV.2201.12086},
url = {https://arxiv.org/abs/2201.12086},
author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Configuration
- Architecture
- BlipForConditionalGeneration
- Context length (tokens)
- 512
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 30,524
- Stored precision
- float32
- Model type
- blip
Identity and Version
- Repository
- Salesforce/blip-image-captioning-base
- Publisher
- Salesforce AI Research
- Task
- Image to text
- Modality
- Image and text
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- tf
- Revision
- 82a37760796d32b1411fe092ab5d4e227313294b
- First published
- 2022-12-12
- Last updated
- 2025-02-03
Files and Weights
10 files, 2.0 GB in total. The weights are 2 files totalling 2.0 GB in bin, h5.
Weights2 files · 2.0 GB
Configuration3 files · 5.0 KB
Tokenizer3 files · 943.4 KB
Documentation1 file · 6.4 KB
Repository1 file · 1.5 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 989.8 MB | d6638651a552 |
| tf_model.h5 | Weights | 990.3 MB | d0aaa4c0e003 |
| config.json | Configuration | 4.6 KB | — |
| preprocessor_config.json | Configuration | 287 B | — |
| special_tokens_map.json | Configuration | 125 B | — |
| README.md | Documentation | 6.4 KB | — |
| .gitattributes | Repository | 1.5 KB | — |
| tokenizer.json | Tokenizer | 711.4 KB | — |
| tokenizer_config.json | Tokenizer | 506 B | — |
| vocab.txt | Tokenizer | 231.5 KB | — |
License and Download
- License
- bsd-3-clause
- Access
- Open weights, no gate
- Download size
- 2.0 GB
Download from Salesforce AI Research
Released by Salesforce AI Research through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2201.12086
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 2.0 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About blip-image-captioning-base
Can I use blip-image-captioning-base commercially?
Yes. blip-image-captioning-base is released under BSD 3-Clause License. The BSD 3-Clause License is permissive. It permits commercial use and redistribution with the copyright notice, and forbids using the authors' names to endorse derived products without permission.
What is blip-image-captioning-base's context length?
512 tokens, from the maximum position embeddings in its published configuration.
Similar Models
PP-OCRv5serverdet is one of the PP-OCRv5det series, the latest generation of text detection models developed by the PaddleOCR team. Designed for high-performance applications, it supports the detection of text in diverse scenarios—including handwriting, vertical, rotated, and curved text—across multiple languages such as Simplified Chinese, Traditional Chinese, English, and Japanese. Key features include robust handling of complex layouts, varying text sizes, and challenging backgrounds, making it suitable for practical applications like document analysis, license plate recognition, and scene text detection. The key accuracy metrics are as follow: Please refer to the following commands to…
The main purpose of text image correction is to carry out geometric transformation on the image to correct the document distortion, inclination, perspective deformation and other problems in the image, so that the subsequent text recognition can be more accurate. Please refer to the following commands to install PaddlePaddle using pip: For details about PaddlePaddle installation, please refer to the PaddlePaddle official website. Install the latest version of the PaddleOCR inference package from PyPI: You can quickly experience the functionality with a single command: You can also integrate the model inference of the TextImageUnwarping module into your project. Before running the following…
enPP-OCRv5mobilerec is one of the PP-OCRv5rec that are the latest generation text line recognition models developed by PaddleOCR team. It aims to efficiently and accurately support the recognition of English. The key accuracy metrics are as follow: Note: If any character (including punctuation) in a line was incorrect, the entire line was marked as wrong. This ensures higher accuracy in practical applications. Please refer to the following commands to install PaddlePaddle using pip: For details about PaddlePaddle installation, please refer to the PaddlePaddle official website. Install the latest version of the PaddleOCR inference package from PyPI: You can quickly experience the…
The Document Image Orientation Classification Module is primarily designed to distinguish the orientation of document images and correct them through post-processing. During processes such as document scanning or ID photo capturing, the device might be rotated to achieve clearer images, resulting in images with various orientations. Standard OCR pipelines may not handle these images effectively. By leveraging image classification techniques, the orientation of documents or IDs containing text regions can be pre-determined and adjusted, thereby improving the accuracy of OCR processing. The key accuracy metrics are as follow: Please refer to the following commands to install PaddlePaddle…
TrOCR model fine-tuned on the IAM dataset. It was introduced in the paper TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Li et al. and first released in this repository. The TrOCR model is an encoder-decoder model, consisting of an image Transformer as encoder, and a text Transformer as decoder. The image encoder was initialized from the weights of DeiT, while the text decoder was initialized from the weights of UniLM. Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds absolute position embeddings before feeding the sequence to the layers of the Transformer encoder. Next, the…