Search public pages, research tools, and SAVRN solutions.
Open-weight model · Zero shot image classification
by OpenAI openai/clip-vit-base-patch32
Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found here. The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks.
Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found here. The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within. January 2021 The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked self-attention…
Excerpt from the card by OpenAI.
12 files, 1.8 GB in total. The weights are 3 files totalling 1.8 GB in bin, h5, msgpack.
| File | Type | Size | SHA-256 |
|---|---|---|---|
| flax_model.msgpack | Weights | 605.1 MB | 2a2994d89beb |
| pytorch_model.bin | Weights | 605.2 MB | a63082132ba4 |
| tf_model.h5 | Weights | 605.6 MB | 0d7e64ea2c49 |
| config.json | Configuration | 4.2 KB | — |
| preprocessor_config.json | Configuration | 316 B | — |
| special_tokens_map.json | Configuration | 389 B | — |
| README.md | Documentation | 7.9 KB | — |
| .gitattributes | Repository | 690 B | — |
| merges.txt | Tokenizer | 524.7 KB | — |
| tokenizer.json | Tokenizer | 2.2 MB | — |
| tokenizer_config.json | Tokenizer | 592 B | — |
| vocab.json | Tokenizer | 862.3 KB | — |
Released by OpenAI through its official repository on Hugging Face.
| Precision | Weights in memory |
|---|---|
| As published | 1.8 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
77 tokens, from the maximum position embeddings in its published configuration.
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set: The following hyperparameters were used during training: - trainingprecision: float32 - Transformers 4.21.3 - TensorFlow 2.8.2 - Tokenizers 0.12.1
Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found here. The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within. January 2021 The base model uses a ViT-B/16 Transformer architecture as an image encoder and uses a masked…
A series of CLIP ConvNeXt-Base (w/ wide embed dim) models trained on subsets LAION-5B (https://laion.ai/blog/laion-5b/) using OpenCLIP (https://github.com/mlfoundations/openclip). Explore an alternative to ViT and ResNet (w/ AttentionPooling) CLIP models that scales well with model size and image resolution First known ConvNeXt CLIP models trained at scale in the range of CLIP ViT-B/16 and RN50x4 models First released model weights exploring increase of augmentation + regularization for image tower via adding (greater scale range of RRC, random erasing, stochastic depth) The models utilize the timm ConvNeXt-Base model (convnextbase) as the image tower, and the same text tower as the RN50x4…
DermLIP is a vision-language model for dermatology, trained on the Derm1M dataset—the largest dermatological image-text corpus to date. python import openclip from PIL import Image import torch model,, preprocess = openclip.createmodelandtransforms( 'hf-hub:redlessone/DermLIPViT-B-16' model.eval() tokenizer = openclip.gettokenizer('hf-hub:redlessone/DermLIPViT-B-16') image = preprocess(Image.open("yourskinimage.png")).unsqueeze(0) PADCLASSNAMES = [ "nevus", "basal cell carcinoma", "actinic keratosis", "seborrheic keratosis", "squamous cell carcinoma", "melanoma" template = lambda c: f'This is a skin image of {c}' text = tokenizer([template(c) for c in PADCLASSNAMES]) with torch.nograd()…
DermFM-Zero is a dermatology vision–language foundation model. It pairs a PanDerm ViT-L/16 vision encoder with native-resolution input (NaViT patch-and-pack) and a PubMedBERT-256 text encoder pretrained on a Derm1M knowledge tree (KEP), aligned on 517,455 public dermatology image–text pairs with multi-aspect knowledge contrastive learning (MAKE) and knowledge distillation. It performs zero-shot diagnosis and cross-modal retrieval, serves as a backbone for multimodal fine-tuning and VQA, and yields interpretable concepts through sparse autoencoders. Across extensive benchmarks and three multinational reader studies, DermFM-Zero achieves state-of-the-art zero-shot performance while improving…
A CLIP ViT-B/32 model trained with the LAION-2B English subset of LAION-5B (https://laion.ai/blog/laion-5b/) using OpenCLIP (https://github.com/mlfoundations/openclip). Model training done by Romain Beaumont on the stability.ai cluster. As per the original OpenAI CLIP model card, this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such model. The OpenAI CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.…