Learning Transferable Visual Models From Natural Language Supervision

## Learning Transferable Visual Models From Natural Language Supervision Alec Radford * 1 Jong Wook Kim * 1 Chris Hallacy 1 Aditya Ramesh 1 Gabriel Goh 1 Sandhini Agarwal 1 Girish Sastry 1 Amanda Askell 1 Pamela Mishkin 1 Jack Clark 1 Gretchen Krueger 1 Ilya Sutskever 1 Abstract State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of super-vision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision.

We demonstrate that the simple pre-training task of predicting which caption goes with which im-age is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmark-ing on over 30 different existing computer vi-sion datasets, spanning tasks such as OCR, ac-tion recognition in videos, geo-localization, and many types of fine-grained object classification.

The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset spe-cific training. For instance, we match the ac-curacy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at com/OpenAI/CLIP. 1. Introduction and Motivating Work Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et, 2018; Howard & Ruder, 2018; Rad-ford et

, 2018; Devlin et, 2018; Raffel et, 2019). > *Equal contribution 1OpenAI, San Francisco, CA 94110, USA. Correspondence to: <{alec, com >. Task-agnostic objectives such as autoregressive and masked language modeling have scaled across many orders of mag-nitude in compute, model capacity, and data, steadily im-proving capabilities. The development of “text-to-text” as a standardized input-output interface (McCann et, 2018; Radford et, 2019; Raffel et, 2019) has enabled task-agnostic architectures to zero-shot transfer to downstream datasets removing the need for specialized output heads or dataset specific customization. Flagship systems like GPT-3 (Brown et

, 2020) are now competitive across many tasks with bespoke models while requiring little to no dataset specific training data. These results suggest that the aggregate supervision acces-sible to modern pre-training methods within web-scale col-lections of text surpasses that of high-quality crowd-labeled NLP datasets. However, in other fields such as computer vision it is still standard practice to pre-train models on crowd-labeled datasets such as ImageNet (Deng et, 2009). Could scalable pre-training methods which learn directly from web text result in a similar breakthrough in computer vision? Prior work is encouraging. Over 20 years ago Mori et al.

(1999) explored improving content based image retrieval by training a model to pre-dict the nouns and adjectives in text documents paired with images. Quattoni et al. (2007) demonstrated it was possi-ble to learn more data efficient image representations via manifold learning in the weight space of classifiers trained to predict words in captions associated with images. Sri-vastava & Salakhutdinov (2012) explored deep represen-tation learning by training multimodal Deep Boltzmann Machines on top of low-level image and text tag features. Joulin et al. (2016) modernized this line of work and demon-strated that CNNs trained to predict words in image cap-tions learn useful image representations.

They converted the title, description, and hashtag metadata of images in the YFCC100M dataset (Thomee et, 2016) into a bag-of-words multi-label classification task and showed that pre-training AlexNet (Krizhevsky et, 2012) to predict these labels learned representations which preformed similarly to ImageNet-based pre-training on transfer tasks. Li et al. (2017) then extended this approach to predicting phrase n-grams in addition to individual words and demonstrated the ability of their system to zero-shot transfer to other image > arXiv:2103.00020v1 CV] 26 Feb 2021 Learning Transferable Visual Models From Natural Language Supervision 2I1·T 2 I1·T 3 … > I2·T 1I2·T 3… > I3·T 1I3·T 2… > ⋮⋮⋮ > I1·T 1 > I2·T 2 > I3·T 3 > (1)

Contrastive pre-training > Image Encoder > Text Encoder > Pepper the aussie pup > Pepper the aussie pup > Pepper the aussie pup > Pepper the aussie pup > T1T2T3… > I1 > I2 > I3 > ⋮ > (2) Create dataset classifier from label text > plane > car > dog > ⋮ > bird > Aphoto of a {object}. > ⋮ > Text Encoder > T1T2T3TN > … > (3) Use for zero-shot prediction > Image Encoder I1I1·T 2I1·T NI1·T 1 > … > … > Aphoto of a dog. > TN > IN·T 1IN·T 2IN·T 3 > I1·T N > I2·T N > I3·T N > ⋮ > …IN > … > ⋮⋱ > IN·T N > I1·T 3 Figure 1.

Summary of our approach. While standard image models jointly train an image feature extractor and a linear classifier to predict some label, CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples. At test time the learned text encoder synthesizes a zero-shot linear classifier by embedding the names or descriptions of the target dataset’s classes. classification datasets by scoring target classes based on their dictionary of learned visual n-grams and predicting the one with the highest score. Adopting more recent architec-tures and pre-training approaches, VirTex (Desai & Johnson, 2020), ICMLM (Bulent Sariyildiz et

, 2020), and Con-VIRT (Zhang et, 2020) have recently demonstrated the potential of transformer-based language modeling, masked language modeling, and contrastive objectives to learn im-age representations from text. While exciting as proofs of concept, using natural language supervision for image representation learning is still rare. This is likely because demonstrated performance on com-mon benchmarks is much lower than alternative approaches. For example, Li et al. (2017) reach only 11.5% accuracy on ImageNet in a zero-shot setting. This is well below the 88.4% accuracy of the current state of the art (Xie et, 2020).

It is even below the 50% accuracy of classic com-puter vision approaches (Deng et, 2012). Instead, more narrowly scoped but well-targeted uses of weak supervision have improved performance. Mahajan et al. (2018) showed that predicting ImageNet-related hashtags on Instagram im-ages is an effective pre-training task. When fine-tuned to ImageNet these pre-trained models increased accuracy by over 5% and improved the overall state of the art at the time. Kolesnikov et al. (2019) and Dosovitskiy et al. (2020) have also demonstrated large gains on a broader set of transfer benchmarks by pre-training models to predict the classes of the noisily labeled JFT-300M dataset.

This line of work represents the current pragmatic middle ground between learning from a limited amount of super-vised “gold-labels” and learning from practically unlimited amounts of raw text. However, it is not without compro-mises. Both works carefully design, and in the process limit, their supervision to 1000 and 18291 classes respectively. Natural language is able to express, and therefore supervise, a much wider set of visual concepts through its general-ity. Both approaches also use static softmax classifiers to perform prediction and lack a mechanism for dynamic out-puts. This severely curtails their flexibility and limits their “zero-shot”

capabilities. A crucial difference between these weakly supervised mod-els and recent explorations of learning image representations directly from natural language is scale. While Mahajan et al. (2018) and Kolesnikov et al. (2019) trained their models for accelerator years on millions to billions of images, VirTex, ICMLM, and ConVIRT trained for accelerator days on one to two hundred thousand images. In this work, we close this gap and study the behaviors of image classifiers trained with natural language supervision at large scale. Enabled by the large amounts of publicly available data of this form on the internet, we create a new dataset of 400 million (im-age, text) pairs and demonstrate that a simplified version of ConVIRT trained from scratch, which we call CLIP, for Con-trastive Language-Image Pre-training, is an efficient method of learning from natural language supervision.

We study the scalability of CLIP by training a series of eight models spanning almost 2 orders of magnitude of compute and ob-serve that transfer performance is a smoothly predictable function of compute (Hestness et, 2017; Kaplan et, 2020). We find that CLIP, similar to the GPT family, learns to perform a wide set of tasks during pre-training including OCR, geo-localization, action recognition, and many others. We measure this by benchmarking the zero-shot transfer performance of CLIP on over 30 existing datasets and find Learning Transferable Visual Models From Natural Language Supervision 32M 33M 67M 134M 268M 400M # of images processed 0510 15 20 25 30 35 40 Zero-Shot ImageNet

Accuracy > 3X efficiency 4X efficiency Bag of Words Contrastive (CLIP) Bag of Words Prediction Transformer Language Model Figure 2. CLIP is much more efficient at zero-shot transfer than our image caption baseline. Although highly expressive, we found that transformer-based language models are relatively weak at zero-shot ImageNet classification. Here, we see that it learns 3x slower than a baseline which predicts a bag-of-words (BoW) encoding of the text (Joulin et, 2016). Swapping the prediction objective for the contrastive objective of CLIP further improves efficiency another 4x. it can be competitive with prior task-specific supervised models.

We also confirm these findings with linear-probe representation learning analysis and show that CLIP out-performs the best publicly available ImageNet model while also being more computationally efficient. We additionally find that zero-shot CLIP models are much more robust than equivalent accuracy supervised ImageNet models which suggests that zero-shot evaluation of task-agnostic models is much more representative of a model’s capability. These re-sults have significant policy and ethical implications, which we consider in Section 7. 2. Approach 2.1. Natural Language Supervision At the core of our approach is the idea of learning percep-tion from supervision contained in natural language.

As discussed in the introduction, this is not at all a new idea, however terminology used to describe work in this space is varied, even seemingly contradictory, and stated motiva-tions are diverse. Zhang et al. (2020), Gomez et al. (2017), Joulin et al. (2016), and Desai & Johnson (2020) all intro-duce methods which learn visual representations from text paired with images but describe their approaches as unsuper-vised, self-supervised, weakly supervised, and supervised respectively. We emphasize that what is common across this line of work is not any of the details of the particular methods used but the appreciation of natural language as a training signal.

All these approaches are learning from natural language super-vision. Although early work wrestled with the complexity of natural language when using topic model and n-gram representations, improvements in deep contextual represen-tation learning suggest we now have the tools to effectively leverage this abundant source of supervision (McCann et, 2017). Learning from natural language has several potential strengths over other training methods. It’s much easier to scale natural language supervision compared to standard crowd-sourced labeling for image classification since it does not require annotations to be in a classic “machine learning compatible format” such as the canonical 1-of-N majority vote “gold label”.

Instead, methods which work on natural language can learn passively from the supervision contained in the vast amount of text on the internet. Learning from natural language also has an important advantage over most unsupervised or self-supervised learning approaches in that it doesn’t “just” learn a representation but also connects that representation to language which enables flexible zero-shot transfer. In the following subsections, we detail the specific approach we settled on. 2.2. Creating a Sufficiently Large Dataset Existing work has mainly used three datasets, MS-COCO (Lin et, 2014), Visual Genome (Krishna et, 2017), and YFCC100M (Thomee et

, 2016). While MS-COCO and Visual Genome are high quality crowd-labeled datasets, they are small by modern standards with approximately 100,000 training photos each. By comparison, other computer vision systems are trained on up to 3.5 billion Instagram photos (Mahajan et, 2018). YFCC100M, at 100 million photos, is a possible alternative, but the metadata for each image is sparse and of varying quality. Many images use automati-cally generated filenames like 20160716 JPG as “titles” or contain “descriptions” of camera exposure settings. After filtering to keep only images with natural language titles and/or descriptions in English, the dataset shrunk by a factor of 6 to only 15 million photos.

This is approximately the same size as ImageNet. A major motivation for natural language supervision is the large quantities of data of this form available publicly on the internet. Since existing datasets do not adequately reflect this possibility, considering results only on them would un-derestimate the potential of this line of research. To address this, we constructed a new dataset of 400 million (image, text) pairs collected form a variety of publicly available sources on the Internet. To attempt to cover as broad a set of visual concepts as possible, we search for (image, text) pairs as part of the construction process whose text

includes one of a set of 500,000 queries. 1 We approximately class > 1 The base query list is all words occurring at least 100 times in the English version of Wikipedia. This is augmented with bi-grams Learning Transferable Visual Models From Natural Language Supervision 4balance the results by including up to 20,000 (image, text) pairs per query. The resulting dataset has a similar total word count as the WebText dataset used to train GPT-2. We refer to this dataset as WIT for WebImageText. 2.3. Selecting an Efficient Pre-Training Method State-of-the-art computer vision systems use very large amounts of compute.

Mahajan et al. (2018) required 19 GPU years to train their ResNeXt101-32x48d and Xie et al. (2020) required 33 TPUv3 core-years to train their Noisy Student EfficientNet-L2. When considering that both these systems were trained to predict only 1000 ImageNet classes, the task of learning an open set of visual concepts from natural language seems daunting. In the course of our ef-forts, we found training efficiency was key to successfully scaling natural language supervision and we selected our final pre-training method based on this metric. Our initial approach, similar to VirTex, jointly trained an image CNN and text transformer from scratch to predict the caption of an image.

However, we encountered difficulties efficiently scaling this method. In Figure 2 we show that a 63 million parameter transformer language model, which already uses twice the compute of its ResNet-50 image encoder, learns to recognize ImageNet classes three times slower than a much simpler baseline that predicts a bag-of-words encoding of the same text. Both these approaches share a key similarity. They try to pre-dict the exact words of the text accompanying each image. This is a difficult task due to the wide variety of descriptions, comments, and related text that co-occur with images. Re-cent work in contrastive representation learning for images has found that contrastive objectives can learn better repre-sentations than their equivalent predictive objective (Tian et

, 2019). Other work has found that although generative models of images can learn high quality image representa-tions, they require over an order of magnitude more compute than contrastive models with the same performance (Chen et, 2020a). Noting these findings, we explored training a system to solve the potentially easier proxy task of pre-dicting only which text as a whole is paired with which image and not the exact words of that text. Starting with the same bag-of-words encoding baseline, we swapped the predictive objective for a contrastive objective in Figure 2 and observed a further 4x efficiency improvement in the rate of zero-shot transfer to ImageNet.

Given a batch of N (image, text) pairs, CLIP is trained to predict which of the N × N possible (image, text) pairings across a batch actually occurred. To do this, CLIP learns a > with high pointwise mutual information as well as the names of all Wikipedia articles above a certain search volume. Finally all WordNet synsets not already in the query list are added. multi-modal embedding space by jointly training an image encoder and text encoder to maximize the cosine similar-ity of the image and text embeddings of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the N 2 − N incorrect pairings.

We opti-mize a symmetric cross entropy loss over these similarity scores. In Figure 3 we include pseudocode of the core of an implementation of CLIP. To our knowledge this batch con-struction technique and objective was first introduced in the area of deep metric learning as the multi-class N-pair loss Sohn (2016), was popularized for contrastive representation learning by Oord et al. (2018) as the InfoNCE loss, and was recently adapted for contrastive (text, image) representation learning in the domain of medical imaging by Zhang et al. (2020). Due to the large size of our pre-training dataset, over-fitting is not a major concern and the details of training CLIP are simplified compared to the implementation of Zhang et al.

(2020). We train CLIP from scratch without initializing the image encoder with ImageNet weights or the text encoder with pre-trained weights. We do not use the non-linear projection between the representation and the contrastive embedding space, a change which was introduced by Bach-man et al. (2019) and popularized by Chen et al. (2020b). We instead use only a linear projection to map from each en-coder’s representation to the multi-modal embedding space. We did not notice a difference in training efficiency between the two versions and speculate that non-linear projections may be co-adapted with details of current image only in self-supervised

representation learning methods. We also remove the text transformation function tu from Zhang et al. (2020) which samples a single sentence at uniform from the text since many of the (image, text) pairs in CLIP’s pre-training dataset are only a single sentence. We also simplify the image transformation function tv. A random square crop from resized images is the only data augmentation used during training. Finally, the temperature parameter which controls the range of the logits in the softmax, τ, is directly optimized during training as a log-parameterized multiplicative scalar to avoid turning as a hyper-parameter.

2.4. Choosing and Scaling a Model We consider two different architectures for the image en-coder. For the first, we use ResNet-50 (He et, 2016a) as the base architecture for the image encoder due to its widespread adoption and proven performance. We make sev-eral modifications to the original version using the ResNet-D improvements from He et al. (2019) and the antialiased rect-2 blur pooling from Zhang (2019). We also replace the global average pooling layer with an attention pooling mechanism. The attention pooling is implemented as a sin-gle layer of “transformer-style” multi-head QKV attention where the query is conditioned on the global average-pooled Learning Transferable Visual Models From Natural Language Supervision 5# image_encoder - ResNet or Vision Transformer # text_encoder - CBOW or Text Transformer # I[n, h, w, c] - minibatch of aligned images # T[n, l] - minibatch of aligned texts # W_i[d_i, d_e] - learned proj of image to embed #

W_t[d_t, d_e] - learned proj of text to embed # t - learned temperature parameter # extract feature representations of each modality I_f = image_encoder(I) #[n, d_i] T_f = text_encoder(T) #[n, d_t] # joint multimodal embedding [n, d_e] I_e = dot(I_f, W_i), axis=1) T_e = dot(T_f, W_t), axis=1) # scaled pairwise cosine similarities [n, n] logits = dot(I_e, T) * exp(t) # symmetric loss function labels = arange(n) loss_i = cross_entropy_loss(logits, labels, axis=0) loss_t = cross_entropy_loss(logits, labels, axis=1) loss = (loss_i + loss_t)/2 Figure 3. Numpy-like pseudocode for the core of an implementa-tion of CLIP.

representation of the image. For the second architecture, we experiment with the recently introduced Vision Transformer (ViT) (Dosovitskiy et, 2020). We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme. The text encoder is a Transformer (Vaswani et, 2017) with the architecture modifications described in Radford et al. (2019). As a base size we use a 63M-parameter 12-layer 512-wide model with 8 attention heads. The trans-former operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sen-nrich et

, 2015). For computational efficiency, the max sequence length was capped at 76. The text sequence is bracketed with [SOS] and [EOS] tokens and the activa-tions of the highest layer of the transformer at the [EOS] token are treated as the feature representation of the text which is layer normalized and then linearly projected into the multi-modal embedding space. Masked self-attention was used in the text encoder to preserve the ability to ini-tialize with a pre-trained language model or add language modeling as an auxiliary objective, though exploration of this is left as future work.

While previous computer vision research has often scaled models by increasing the width (Mahajan et, 2018) or depth (He et, 2016a) in isolation, for the ResNet image encoders we adapt the approach of Tan & Le (2019) which found that allocating additional compute across all of width, depth, and resolution outperforms only allocating it to only one dimension of the model. While Tan & Le (2019) tune the ratio of compute allocated to each dimension for their EfficientNet architecture, we use a simple baseline of allo-cating additional compute equally to increasing the width, depth, and resolution of the model.

For the text encoder, we only scale the width of the model to be proportional to the calculated increase in width of the ResNet and do not scale the depth at all, as we found CLIP’s performance to be less sensitive to the capacity of the text encoder. 2.5. Training We train a series of 5 ResNets and 3 Vision Transformers. For the ResNets we train a ResNet-50, a ResNet-101, and then 3 more which follow EfficientNet-style model scaling and use approximately 4x, 16x, and 64x the compute of a ResNet-50. They are denoted as RN50x4, RN50x16, and RN50x64 respectively.

For the Vision Transformers we train a ViT-B/32, a ViT-B/16, and a ViT-L/14. We train all models for 32 epochs. We use the Adam optimizer (Kingma & Ba, 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2017) applied to all weights that are not gains or biases, and decay the learning rate using a cosine schedule (Loshchilov & Hutter, 2016). Initial hyper-parameters were set using a combination of grid searches, random search, and manual tuning on the baseline ResNet-50 model when trained for 1 epoch. Hyper-parameters were then adapted heuristically for larger models due to compu-tational constraints.

The learnable temperature parameter τ was initialized to the equivalent of 0.07 from (Wu et, 2018) and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training in-stability. We use a very large minibatch size of 32,768. Mixed-precision (Micikevicius et, 2017) was used to ac-celerate training and save memory. To save additional mem-ory, gradient checkpointing (Griewank & Walther, 2000; Chen et, 2016), half-precision Adam statistics (Dhariwal et, 2020), and half-precision stochastically rounded text encoder weights were used. The calculation of embedding similarities was also sharded with individual GPUs comput-ing only the subset of the pairwise similarities necessary for their local batch of embeddings.

The largest ResNet model, RN50x64, took 18 days to train on 592 V100 GPUs while the largest Vision Transformer took 12 days on 256 V100 GPUs. For the ViT-L/14 we also pre-train at a higher 336 pixel resolution for one additional epoch to boost perfor-mance similar to FixRes (Touvron et, 2019). We denote this model as ViT-L/14@336px. Unless otherwise specified, all results reported in this paper as “CLIP” use this model which we found to perform best. Learning Transferable Visual Models From Natural Language Supervision 6 3. Experiments 3.1. Zero-Shot Transfer 1.1. M OTIVATION In computer vision, zero-shot learning usually refers to the study of generalizing to unseen object categories in image classification (Lampert et

, 2009). We instead use the term in a broader sense and study generalization to unseen datasets. We motivate this as a proxy for performing un-seen tasks, as aspired to in the zero-data learning paper of Larochelle et al. (2008). While much research in the field of unsupervised learning focuses on the representation learn-ing capabilities of machine learning systems, we motivate studying zero-shot transfer as a way of measuring the task-learning capabilities of machine learning systems. In this view, a dataset evaluates performance on a task on a spe-cific distribution. However, many popular computer vision datasets were created by the research community primarily as benchmarks to guide the development of generic image classification methods rather than measuring performance on a specific task.

While it is reasonable to say that the SVHN dataset measures the task of street number transcrip-tion on the distribution of Google Street View photos, it is unclear what “real” task the CIFAR-10 dataset measures. It is clear, however, what distribution CIFAR-10 is drawn from - TinyImages (Torralba et, 2008). On these kinds of datasets, zero-shot transfer is more an evaluation of CLIP’s robustness to distribution shift and domain generalization rather than task generalization. Please see Section 3.3 for analysis focused on this. To our knowledge, Visual N-Grams (Li et, 2017) first studied zero-shot transfer to existing image classification datasets in the manner described above.

It is also the only other work we are aware of that has studied zero-shot trans-fer to standard image classification datasets using a gener-ically pre-trained model and serves as the best reference point for contextualizing CLIP. Their approach learns the parameters of a dictionary of 142,806 visual n-grams (span-ning 1- to 5- grams) and optimizes these n-grams using a differential version of Jelinek-Mercer smoothing to maxi-mize the probability of all text n-grams for a given image. In order to perform zero-shot transfer, they first convert the text of each of the dataset’s class names into its n-gram representation and then compute its probability according to their model, predicting the one with the highest score.

Our focus on studying zero-shot transfer as an evaluation of task learning is inspired by work demonstrating task learn-ing in the field of NLP. To our knowledge Liu et al. (2018) first identified task learning as an “unexpected side-effect” when a language model trained to generate Wikipedia ar-ticles learned to reliably transliterate names between lan-guages. While GPT-1 (Radford et, 2018) focused on pre-training as a transfer learning method to improve supervised fine-tuning, it also included an ablation study demonstrat-ing that the performance of four heuristic zero-shot transfer methods improved steadily over the course of pre-training, without any supervised adaption.

This analysis served as the basis for GPT-2 (Radford et, 2019) which focused exclu-sively on studying the task-learning capabilities of language models via zero-shot transfer. 1.2. U SING CLIP FOR ZERO -S HOT TRANSFER CLIP is pre-trained to predict if an image and a text snippet are paired together in its dataset. To perform zero-shot clas-sification, we reuse this capability. For each dataset, we use the names of all the classes in the dataset as the set of poten-tial text pairings and predict the most probable (image, text) pair according to CLIP. In a bit more detail, we first compute the feature embedding of the image and the feature embed-ding of the set of possible texts by their respective encoders.

The cosine similarity of these embeddings is then calculated, scaled by a temperature parameter τ, and normalized into a probability distribution via a softmax. Note that this predic-tion layer is a multinomial logistic regression classifier with L2-normalized inputs, L2-normalized weights, no bias, and temperature scaling. When interpreted this way, the image encoder is the computer vision backbone which computes a feature representation for the image and the text encoder is a hypernetwork (Ha et, 2016) which generates the weights of a linear classifier based on the text specifying the visual concepts that the classes represent.

Lei Ba et al. (2015) first introduced a zero-shot image classifier of this form while the idea of generating a classifier from natural language dates back to at least Elhoseiny et al. (2013). Continuing with this interpretation, every step of CLIP pre-training can be viewed as optimizing the performance of a randomly created proxy to a computer vision dataset which contains 1 example per class and has 32,768 total classes defined via natural language descriptions. For zero-shot evaluation, we cache the zero-shot classifier once it has been computed by the text encoder and reuse it for all subsequent predictions.

This allows the cost of generating it to be amortized across all the predictions in a dataset. 1.3. I NITIAL COMPARISON TO VISUAL N-G RAMS In Table 1 we compare Visual N-Grams to CLIP. The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 million crowd-labeled training examples available for this dataset. Additionally, the top-5 accuracy of CLIP models are noticeably higher than their top-1, and this model has a 95% top-5 accuracy, matching Inception-V4 (Szegedy et

, 2016). The ability to match the performance of a strong, fully supervised baselines in a zero-shot setting suggests Learning Transferable Visual Models From Natural Language Supervision 7aYahoo ImageNet SUN Visual N-Grams 72.4 11.5 23.0 CLIP 98.4 76.2 58.5 Table 1. Comparing CLIP to prior zero-shot transfer image classi-fication results. CLIP improves performance on all three datasets by a large amount. This improvement reflects many differences in the 4 years since the development of Visual N-Grams (Li et, 2017). CLIP is a significant step towards flexible and practical zero-shot computer vision classifiers. As mentioned above, the comparison to Visual N-Grams is meant for contextu-alizing the performance of CLIP and should not be inter-preted as a direct methods comparison between CLIP and Visual N-Grams as many performance relevant differences between the two systems were not controlled for.

For in-stance, we train on a dataset that is 10x larger, use a vision model that requires nearly 100x more compute per predic-tion, likely used over 1000x their training compute, and use a transformer-based model which did not exist when Visual N-Grams was published. As a closer comparison, we trained a CLIP ResNet-50 on the same YFCC100M dataset that Visual N-Grams was trained on and found it matched their reported ImageNet performance within a V100 GPU day. This baseline was also trained from scratch instead of being initialized from pre-trained ImageNet weights as in Visual N-Grams.

CLIP also outperforms Visual N-Grams on the other 2 re-ported datasets. On aYahoo, CLIP achieves a 95% reduction in the number of errors, and on SUN, CLIP more than dou-bles the accuracy of Visual N-Grams. To conduct a more comprehensive analysis and stress test, we implement a much larger evaluation suite detailed in Appendix A. In total we expand from the 3 datasets reported in Visual N-Grams to include over 30 datasets and compare to over 50 existing computer vision systems to contextualize results. 1.4. P ROMPT ENGINEERING AND ENSEMBLING Most standard image classification datasets treat the infor-mation naming or describing classes which enables natural language based zero-shot transfer as an afterthought.

The vast majority of datasets annotate images with just a numeric id of the label and contain a file mapping these ids back to their names in English. Some datasets, such as Flowers102 and GTSRB, don’t appear to include this mapping at all in their released versions preventing zero-shot transfer en-tirely. 2 For many datasets, we observed these labels may be > 2 Alec learned much more about flower species and German traffic signs over the course of this project than he originally antic-ipated. 6.1 9.9 21.5 75.3 265.9 Model GFLOPs > 45 > 50 > 55 > 60 > 65 > 70 Average Score (%) > 4X efficiency gain 5 point improvement RN50 RN101 RN50x4 RN50x16 RN50x64 Prompt engineering and ensembling Contextless class names (Li et al.

2017) Figure 4. Prompt engineering and ensembling improve zero-shot performance. Compared to the baseline of using contextless class names, prompt engineering and ensembling boost zero-shot classification performance by almost 5 points on average across 36 datasets. This improvement is similar to the gain from using 4 times more compute with the baseline zero-shot method but is “free” when amortized over many predictions. chosen somewhat haphazardly and do not anticipate issues related to zero-shot transfer which relies on task description in order to transfer successfully. A common issue is polysemy. When the name of a class is the only information provided to CLIP’s text encoder it is unable to differentiate which word sense is meant due to the lack of context.

In some cases multiple meanings of the same word might be included as different classes in the same dataset! This happens in ImageNet which contains both construction cranes and cranes that fly. Another example is found in classes of the Oxford-IIIT Pet dataset where the word boxer is, from context, clearly referring to a breed of dog, but to a text encoder lacking context could just as likely refer to a type of athlete. Another issue we encountered is that it’s relatively rare in our pre-training dataset for the text paired with the image to be just a single word.

Usually the text is a full sentence describing the image in some way. To help bridge this distribution gap, we found that using the prompt template “A photo of a ” to be a good default that helps specify the text is about the content of the image. This often improves performance over the baseline of using only the label text. For instance, just using this prompt improves accuracy on ImageNet by 1.3%. Learning Transferable Visual Models From Natural Language Supervision 8Similar to the “prompt engineering” discussion around GPT-3 (Brown et, 2020; Gao et

, 2020), we have also observed that zero-shot performance can be significantly improved by customizing the prompt text to each task. A few, non exhaustive, examples follow. We found on several fine-grained image classification datasets that it helped to specify the category. For example on Oxford-IIIT Pets, us-ing “ A photo of a {label}, a type of pet. ”to help provide context worked well. Likewise, on Food101 specifying a type of food and on FGVC Aircraft a type of aircraft helped too. For OCR datasets, we found that putting quotes around the text or number to be recognized improved performance.

Finally, we found that on satellite image classi-fication datasets it helped to specify that the images were of this form and we use variants of “ a satellite photo of a ”. We also experimented with ensembling over multiple zero-shot classifiers as another way of improving performance. These classifiers are computed by using different context prompts such as ‘ A photo of a big {label}” and “A photo of a small {label}”. We construct the ensemble over the embedding space instead of probability space. This allows us to cache a single set of averaged text embeddings so that the compute cost of the ensemble is the same as using a single classifier when amortized over many predictions.

We’ve observed ensembling across many gen-erated zero-shot classifiers to reliably improve performance and use it for the majority of datasets. On ImageNet, we ensemble 80 different context prompts and this improves performance by an additional 3.5% over the single default prompt discussed above. When considered together, prompt engineering and ensembling improve ImageNet accuracy by almost 5%. In Figure 4 we visualize how prompt engi-neering and ensembling change the performance of a set of CLIP models compared to the contextless baseline approach of directly embedding the class name as done in Li et al. (2017). 1.5.

A NALYSIS OF ZERO -S HOT CLIP P ERFORMANCE Since task-agnostic zero-shot classifiers for computer vision have been understudied, CLIP provides a promising oppor-tunity to gain a better understanding of this type of model. In this section, we conduct a study of various properties of CLIP’s zero-shot classifiers. As a first question, we look simply at how well zero-shot classifiers perform. To con-textualize this, we compare to the performance of a simple off-the-shelf baseline: fitting a fully supervised, regularized, logistic regression classifier on the features of the canonical ResNet-50. In Figure 5 we show this comparison across 27 datasets.

Please see Appendix A for details of datasets and setup. Zero-shot CLIP outperforms this baseline slightly more of-40 30 20 10 0 10 20 30 40 Score (%) > Zero-Shot CLIP vs. Linear Probe on ResNet50 EuroSAT -37.1 KITTI Distance -34.0 PatchCamelyon -19.5 GTSRB -18.4 CLEVRCounts -18.2 DTD -16.6 Flowers102 -12.5 RESISC45 -11.9 FGVCAircraft -11.3 MNIST -10.0 Birdsnap -3.2 +0.5 PascalVOC2007 +1.1 OxfordPets +1.9 ImageNet +2.0 Caltech101 +2.8 FER2013 +3.0 STL10 +3.0 CIFAR100 +3.9 CIFAR10 +6.7 HatefulMemes +7.7 UCF101 +7.8 SUN397 +12.4 SST2 +14.5 Kinetics700 +22.5 Food101 +23.2 Country211 +28.9 StanfordCars > Figure 5. Zero-shot CLIP is competitive with a fully super-vised baseline.

Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet. ten than not and wins on 16 of the 27 datasets. Looking at individual datasets reveals some interesting behavior. On fine-grained classification tasks, we observe a wide spread in performance. On two of these datasets, Stanford Cars and Food101, zero-shot CLIP outperforms logistic regression on ResNet-50 features by over 20% while on two others, Flowers102 and FGVCAircraft, zero-shot CLIP underper-forms by over 10%. On OxfordPets and Birdsnap, per-formance is much closer.

We suspect these difference are primarily due to varying amounts of per-task supervision between WIT and ImageNet. On “general” object classifica-tion datasets such as ImageNet, CIFAR10/100, STL10, and PascalVOC2007 performance is relatively similar with a slight advantage for zero-shot CLIP in all cases. On STL10, CLIP achieves 99.3% overall which appears to be a new state of the art despite not using any training examples. Zero-shot CLIP significantly outperforms a ResNet-50 on two datasets measuring action recognition in videos. On Kinet-ics700, CLIP outperforms a ResNet-50 by 14.5%. Zero-shot CLIP also outperforms a ResNet-50’s features by 7.7% on UCF101.

We speculate this is due to natural language providing wider supervision for visual concepts involving verbs, compared to the noun-centric object supervision in ImageNet. Looking at where zero-shot CLIP notably underperforms, Learning Transferable Visual Models From Natural Language Supervision 90 1 2 4 8 16 # of labeled training examples per class 30 35 40 45 50 55 60 65 70 75 Average Score (%) Zero-Shot CLIP BiT-M (ImageNet-21K) Linear Probe CLIP SimCLRv2 ResNet50 > Figure 6. Zero-shot CLIP outperforms few-shot linear probes. > Zero-shot CLIP matches the average performance of a 4-shot linear classifier trained on the same feature space and nearly matches the best results of a 16-shot linear classifier across publicly available models.

For both BiT-M and SimCLRv2, the best performing model is highlighted. Light gray lines are other models in the eval suite. The 20 datasets with at least 16 examples per class were used in this analysis. we see that zero-shot CLIP is quite weak on several spe-cialized, complex, or abstract tasks such as satellite image classification (EuroSAT and RESISC45), lymph node tumor detection (PatchCamelyon), counting objects in synthetic scenes (CLEVRCounts), self-driving related tasks such as German traffic sign recognition (GTSRB), recognizing dis-tance to the nearest car (KITTI Distance). These results highlight the poor capability of zero-shot CLIP on more complex tasks.

By contrast, non-expert humans can robustly perform several of these tasks, such as counting, satellite image classification, and traffic sign recognition, suggesting significant room for improvement. However, we caution that it is unclear whether measuring zero-shot transfer, as opposed to few-shot transfer, is a meaningful evaluation for difficult tasks that a learner has no prior experience with, such as lymph node tumor classification for almost all hu-mans (and possibly CLIP). While comparing zero-shot performance to fully supervised models contextualizes the task-learning capabilities of CLIP, comparing to few-shot methods is a more direct compari-son, since zero-shot is its limit.

In Figure 6, we visualize how zero-shot CLIP compares to few-shot logistic regres-sion on the features of many image models including the best publicly available ImageNet models, self-supervised learning methods, and CLIP itself. While it is intuitive to expect zero-shot to underperform one-shot, we instead find that zero-shot CLIP matches the performance of 4-shot lo-gistic regression on the same feature space. This is likely due to an important difference between the zero-shot and few-shot approach. First, CLIP’s zero-shot classifier is gen-erated via natural language which allows for visual concepts to be directly specified (“communicated”). By contrast, “normal” supervised learning must infer concepts indirectly from training examples.

Context-less example-based learn-ing has the drawback that many different hypotheses can be consistent with the data, especially in the one-shot case. A single image often contains many different visual con-cepts. Although a capable learner is able to exploit visual cues and heuristics, such as assuming that the concept being demonstrated is the primary object in an image, there is no guarantee. A potential resolution of this discrepancy between zero-shot and few-shot performance is to use CLIP’s zero-shot classifier as a prior for the weights of the few-shot classifier. While adding an L2 penalty towards the generated weights is a straightforward implementation of this idea, we found that hyperparameter optimization would often select for such a large value of this regularizer that the resulting

few-shot classifier was “just” the zero-shot classifier. Research into better methods of combining the strength of zero-shot transfer with flexibility of few-shot learning is a promising direction for future work. When comparing zero-shot CLIP to few-shot logistic re-gression on the features of other models, zero-shot CLIP roughly matches the performance of the best performing 16-shot classifier in our evaluation suite, which uses the fea-tures of a BiT-M ResNet-152x2 trained on ImageNet-21K. We are certain that a BiT-L model trained on JFT-300M would perform even better but these models have not been publicly released. That a BiT-M ResNet-152x2 performs best in a 16-shot setting is somewhat surprising since, as analyzed in Section 3.2, the Noisy Student EfficientNet-L2 outperforms it in a fully supervised setting by almost 5% on average across 27 datasets.

In addition to studying the average performance of zero-shot CLIP and few-shot logistic regression, we also examine performance on individual datasets. In Figure 7, we show estimates for the number of labeled examples per class that a logistic regression classifier on the same feature space requires to match the performance of zero-shot CLIP. Since zero-shot CLIP is also a linear classifier, this estimates the effective data efficiency of zero-shot transfer in this setting. In order to avoid training thousands of linear classifiers, we estimate the effective data efficiency based on a log-linear interpolation of the performance of a 1, 2, 4, 8, 16-shot (when possible), and a fully supervised linear classifier trained on

each dataset. We find that zero-shot transfer can Learning Transferable Visual Models From Natural Language Supervision 10 0 25 50 75 100 125 150 175 200 # of labeled examples per class required to match zero-shot Flowers102 EuroSAT RESISC45 CLEVRCounts GTSRB FGVCAircraft DTD Birdsnap UCF101 KITTI Distance Caltech101 SUN397 MNIST StanfordCars HatefulMemes CIFAR100 STL10 Kinetics700 SST2 PCam ImageNet Country211 OxfordPets Food101 CIFAR10 FER2013 0.9 0.9 1.5 1.5 1.6 2.0 2.6 2.7 2.9 2.9 3.5 3.9 4.8 6.0 9.8 12.0 12.7 13.6 14.4 14.7 16.0 32 48 64 81 184 > Median: 5.4 Mean: 20.8 Figure 7.

The data efficiency of zero-shot transfer varies widely. Calculating the number of labeled examples per class a linear classifier on the same CLIP feature space requires to match the performance of the zero-shot classifier contextualizes the ef-fectiveness of zero-shot transfer. Values are estimated based on log-linear interpolation of 1, 2, 4, 8, 16-shot and fully supervised results. Performance varies widely from still underperforming a one-shot classifier on two datasets to matching an estimated 184 labeled examples per class. have widely varying efficiency per dataset from less than 1 labeled example per class to 184. Two datasets, Flowers102 and EuroSAT underperform one-shot models.

Half of the datasets require less than 5 examples per class with a median of 5.4. However, the mean estimated data efficiency is 20.8 examples per class. This is due to the 20% of datasets where supervised classifiers require many labeled examples per class in order to match performance. On ImageNet, zero-shot CLIP matches the performance of a 16-shot linear classifier trained on the same feature space. If we assume that evaluation datasets are large enough that the parameters of linear classifiers trained on them are well estimated, then, because CLIP’s zero-shot classifier is also a linear classifier, the performance of the fully supervised classifiers roughly sets an upper bound for what zero-shot transfer can achieve.

In Figure 8 we compare CLIP’s zero-shot performance with fully supervised linear classifiers across datasets. The dashed, y = x line represents an “op-timal” zero-shot classifier that matches the performance of its fully supervised equivalent. For most datasets, the per-formance of zero-shot classifiers still underperform fully su-pervised classifiers by 10% to 25%, suggesting that there is still plenty of headroom for improving CLIP’s task-learning and zero-shot transfer capabilities. There is a positive correlation of 0.82 (p-value < 10 −6)between zero-shot performance and fully supervised perfor-20 30 40 50 60 70 80 90 100 Linear Probe CLIP Performance 20 30 40 50 60 70 80 90 100 Zero-Shot CLIP Performance > r = 0.82 > VOC2007 Country211 HatefulMemes MNIST CIFAR10 SST2 DTD PCAM RESISC45 EuroSAT GTSRB CLEVRCounts FER2013 UCF101 Birdsnap OxfordPets CIFAR100 FGVCAircraft Food101 Flowers102 Stanford Cars Caltech101 SUN397 ImageNet STL10 KITTI Distance Kinetics700 Figure 8.

Zero-shot performance is correlated with linear probe performance but still mostly sub-optimal. Comparing zero-shot and linear probe performance across datasets shows a strong correlation with zero-shot performance mostly shifted 10 to 25 points lower. On only 5 datasets does zero-shot performance approach linear probe performance (≤3 point difference). mance, suggesting that CLIP is relatively consistent at con-necting underlying representation and task learning to zero-shot transfer. However, zero-shot CLIP only approaches fully supervised performance on 5 datasets: STL10, CI-FAR10, Food101, OxfordPets, and Caltech101. On all 5 datasets, both zero-shot accuracy and fully supervised accu-racy are over 90%.

This suggests that CLIP may be more effective at zero-shot transfer for tasks where its underly-ing representations are also high quality. The slope of a linear regression model predicting zero-shot performance as a function of fully supervised performance estimates that for every 1% improvement in fully supervised performance, zero-shot performance improves by 1.28%. However, the 95th-percentile confidence intervals still include values of less than 1 (0.93-1.79). Over the past few years, empirical studies of deep learning systems have documented that performance is predictable as a function of important quantities such as training compute and dataset size (Hestness et

, 2017; Kaplan et, 2020). The GPT family of models has so far demonstrated consis-tent improvements in zero-shot performance across a 1000x increase in training compute. In Figure 9, we check whether the zero-shot performance of CLIP follows a similar scaling pattern. We plot the average error rate of the 5 ResNet CLIP models across 39 evaluations on 36 different datasets and find that