Scaling Laws for Autoregressive Generative Modeling

# Scaling Laws for Autoregressive Generative Modeling Tom Henighan ∗ Jared Kaplan ∗† Mor Katz ∗ Mark Chen Christopher Hesse Jacob Jackson Heewoo Jun Tom B. Brown Prafulla Dhariwal Scott Gray Chris Hallacy Benjamin Mann Alec Radford Aditya Ramesh Nick Ryder Daniel M. Ziegler John Schulman Dario Amodei Sam McCandlish OpenAI Abstract We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image ↔text models, and mathematical prob-lem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law.

The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as S(True) + DKL (True || Model), and the empirical scaling laws suggest a prediction for both the true data distribution’s entropy and the KL divergence between the true and model distribu-tions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an 8 × 8 resolution, and we can forecast the model size needed to achieve any given reducible loss (ie DKL) in nats/image for other resolutions.

We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question “Is a picture worth a thousand ”; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

> ∗ equal contribution > † Johns Hopkins University and OpenAI Correspondence to: com Author contributions listed at end of paper. > arXiv:2010.14701v2 LG] 6 Nov 2020 Contents 1 Introduction 3 1.1 Summary of Results....................................... 5 2 Central Empirical Scaling Laws in Each Domain 6 2.1 Domain Descriptions and Training Setups........................... 62.2 Model Size Scaling and Aspect Ratios............................. 92.3 Compute Scaling and Optimal Model Sizes.......................... 92.4 Loss versus Position in the Context Depends on the Structure of the Data........... 11 3 Image and Video Modeling, the Reducible Loss, and Downstream Tasks 11 3.1 Varying the Image Resolution and Encoding..........................

11 3.2 Video Modeling and Individual Frames............................ 13 3.3 Scaling Trends for Individual Images.............................. 14 3.4 Finetuning on ImageNet at 32x32 Resolution......................... 14 4 Multimodal Models and Information Gain 16 5 Mathematical Problem Solving and Extrapolation 16 6 An Inconsistency in Compute and Datasize Scaling Laws 17 7 Related Work 19 8 Discussion 20 A More Details on Image Modeling 22 B Details of Math Experiments and Additional Results 26 1 Procedurally Generated Training Data............................. 26 2 Dataset Size Scaling...................................... 26 3 Additional Math Results.................................... 26 C Additional Multimodal Results 30 D Additional Language Results 31 E Mutual Information, Infogain, and Scaling 33

1 Approximate Derivation of Scaling Relations......................... 33 2 Estimating DKL Between Real-World Distributions...................... 33 F Hyperparameter Settings 34 2Figure 1 Smooth scaling of reducible loss across domains— We show power-law scaling laws for the reducible loss L − L∞ as a function of compute, where the irreducible loss L∞ is a fitted domain-dependent constant. Under plausible assumptions concerning the infinite data and compute limits, the irreducible loss estimates the entropy of the underlying data distribution, while the reducible loss approximates the KL diver-gence between the data and model distributions. In the case of language we use results from [BMR +20], and only show the full loss L.

1 Introduction Large scale models, datasets, and compute budgets have driven rapid progress in machine learning. Recent work [HNA +17, RRBS19, LWS +20, RDG +20, KMH +20, SK20, BMR +20] suggests that the benefits of scale are also highly predictable. When the cross-entropy loss L of a language model is bottlenecked by either the compute budget C, dataset size D, or model size N, the loss scales with each of these quantities as a simple power-law. Sample efficiency also improves with model size. These results raise a number of questions. Do they apply to all data modalities?

How do improvements on the loss translate to improvements in representation quality and performance on downstream tasks? Is there any way to determine when and why the performance of a model might be maxed out, so that further scaling will be met with diminishing returns? What explains the precision and universality of these trends, and what else can we learn from them? We will demonstrate that scaling laws apply to generative modeling across a wide variety of data modali-ties, including generative language [KMH +20, BMR +20], image [TSF +15, CRC +20], and video modeling [WTU19], multimodal modeling [TBL +19] of text-image correlations, and even mathematical problem solv-ing [SGHK19], a task requiring a degree of reasoning ability.

Moreover, we demonstrate that a single archi-tecture – the Transformer [VSP +17, LSP +18], with an autoregressive cross-entropy loss – scales smoothly in all of these domains, with only minimal changes to hyperparameters such as width, depth, or learning rate. We also observe that larger models consistently learn faster, achieving any given value of the loss in fewer steps. By studying many different model sizes N, compute budgets C, or dataset sizes D, we demonstrate that the scaling relation for the loss L(x) = L∞ + (x0 x)αx (1.1) applies to each data modality, where αx is a modality-dependent scaling exponent, and we primarily study x = N, C, and occasionally D.

We will refer to L∞ as the irreducible loss and the power-law scaling term as the reducible loss. These scaling relations often hold to high precision, even when the reducible loss is much smaller than the irreducible loss; we display trends in L(C) for the reducible loss in figure 1. Note that small deviations are visually amplified on the log-plot, but nevertheless the trends fit remarkably well. 310 6 10 4 10 2 10 0 10 2 > Compute (PF-days) 10 4 > 10 5 > 10 6 > 10 7 > 10 8 > 10 9 > 10 10 > Parameters > Optimal Model Size vs Compute > Image 32x32 Image 8x8 Video Math Image-to-Text Text-to-Image Language Trend > (C > 5 × 10 12)0.7 Figure 2 Optimal model size is consistent across domains— We display the optimal model size Nopt as a function of the training compute budget C.

Not only does Nopt (C) behave as a power-law, but the behavior is remarkably similar for all data modalities. These observations suggest the information theoretic interpretation L∞ ≈ S(True) “Irreducible Loss” (x0 x)αx ≈ DKL (True || Model) “Reducible Loss” (1.2) In other words, the irreducible loss estimates the entropy of the true data distribution, while the reducible loss is an estimate of the KL divergence between the true and model distributions. One might have guessed that as the L(x) curve bends and the loss approaches L∞, returns to increasing N, C, D are diminishing.

But the identification of the reducible loss with DKL suggests this is not necessarily the case, and further increases in scale may still provide important additional semantic information. To justify equation (1.2), we must assume that in the limit D → ∞ followed 3 by N, C → ∞, an infinitely large transformer could model the data distribution exactly. The scaling relations provide insights into the complexity of the data and clarify the value of increasing N, D, and C. By evaluating the reducible loss for a full image or video, we are actually estimating the number of bits of information that ‘remain to be understood’ by a given model.

Equivalently, the reducible loss approximates the degree to which the data could be further compressed. We find that billion-parameter models can extract all but a few nats/image concerning YFCC100M images [TSF +15] downsampled to an 8x8 resolution, so they may be nearly perfect models of this data distribution. For larger, more practically relevant images we would need far larger models to achieve this feat, but the scaling laws make it possible to forecast this precisely. These trends are closely tied to the scaling exponents αx: smaller exponents imply slower improvement with increasing scale, meaning that the data can only be compressed further with much larger models.

The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget. We find that just as in [KMH +20] this relation is very nearly a pure power-law Nopt (C) ∝ Surprisingly, the exponent β ∼ 0.7 is very similar for all domains, as shown in figure 2. This has important implications for the scaling of dataset size with model size for compute-optimal training, suggesting that D ∝ N 0.4 if we only train on each data element once. Even allowing for significant errors or deviations, this strongly suggests sub-linear scaling of dataset size with model size.

We can learn more if we focus on questions specific to each data modality. Generative image models can be finetuned for classification. We will show that ImageNet [CLH17] classification performance improves smoothly with pre-trained model size, following another power law. This trend continues even into the large-model regime where the generative loss trend “bends” and becomes dominated by the irreducible component. This strongly suggests that there are benefits to squeezing as much performance as possible out of large > 3We specify this order of limits to make it clear that regularization will not be required.

410 5 10 6 10 7 10 8 10 9 10 10 10 11 10 12 > Parameters > 1.8 2.4 3.0 3.6 4.2 > Test Loss > Language Modeling > (X/1.44 × 10 14)0.070 10 410 510 610 710 810 9 > Parameters > 2.4 × 10 0 > 2.6 × 10 0 > 2.8 × 10 0 > 3 × 10 0 > 3.2 × 10 0 > 3.4 × 10 0 > Loss > Image Modeling > 10 15 20 25 30 > Image Size (Pixels) 10 310 410 510 610 710 810 9 > Parameters > 2 × 10 0 > 3 × 10 0 > 4 × 10 0 > Loss >

Video: 16 Frames, 16x16 VQ > 1.01 + (X > 3.7 e+ 04)0.24 10 410 510 610 710 8 > Parameters > 10 0 > Loss > Math Extrapolating to Various Difficulties > Extrapolate 0.28 + (X > 1.1 e+ 04)0.16 > 11 12 13 14 15 16 17 18 19 > Difficulty 10 510 610 710 810 9 > Parameters > 10 0 > 1.25 × 10 0 > 1.5 × 10 0 > 1.75 × 10 0 > 2 × 10 0 > 2.25 × 10 0 > 2.5 × 10 0 > Loss > Multimodal Text-to-Image > Image Loss 2.00 + (X > 5.1 e+ 03)0.16 > Text Loss (X/5.57 e+ 08) 0.037 10 510 610 710 8 > Parameters > 1.2 × 10 0 > 1.4 × 10 0 > 1.6 × 10 0 > 1.8 × 10 0 > 2 × 10 0 > 2.2 × 10 0 > 2.4 × 10 0 > 2.6 × 10 0 > Loss > Multimodal Image-to-Text > Image Loss 2.00 + (X > 5.5 e+ 03)0.15 > Text Loss (X/7.03 e+ 08) 0.039 Figure 3 Scaling with model size— We show scaling laws with model size for various domains, along with fits (dashed) to equation (1.1).

Note that the largest language models [BMR +20] in the top-left figure were not trained to convergence, so deviations from the trend are not necessarily meaningful. Very small models for video and higher-resolution images are off-trend; we speculate this is due to these models attempting to attend to a context with length comparable to their non-embedding parameter count. generative image models, as significant semantic information may lie in the ‘last few bits’. The smooth trends for finetuned performance on image classification suggest a more general lesson: that the scaling laws for unsupervised learning imply that downstream performance also improves with model size and compute.

Information theory provides a useful lens for examining model performance in other contexts. A striking case is provided by multimodal models, such as those that model the joint distribution between text captions and images. Typically the entropy of the caption is much smaller than that of the image, so the ratio between the (empirical) mutual information 4 and the model’s loss on the text, which we refer to as the Infogain ≡ I(text, image) L(text) (1.3) provides an interesting metric for model performance. The mutual information shared between distributions must be smaller than the amount of information in either distribution, so this ratio must be less than

Furthermore, it appears that the Infogain increases smoothly with model size, so that the bound Infogain < 1 can suggest a target model size for maximum performance. Typically this is far beyond current capabilities. These smooth scaling results on a wide variety of datasets also demonstrate the remarkable versatility of the Transformer architecture. 1.1 Summary of Results We apply autoregressive decoder-only Transformer models to all data modalities, which include web-scraped YFCC100M images [TSF +15] of various resolutions, video data from various sources, multimodal im-age+language data, and procedurally generated math problems. We also reference prior results on language [KMH +20, BMR +20].

Across all domains we find: • The scaling laws of equation (1.1) apply consistently, including for very small values of the reducible loss. Since the L(C) trends can be extended to arbitrarily large data distributions, model sizes, and training steps, we argue that this supports the interpretation of equation (1.2). • We identify the optimal model size Nopt (C) for a given compute budget, and find that it can be accurately modeled as a pure power law [KMH +20] Nopt ∝ Cβ (1.4) > 4 By the empirical mutual information we are referring to Ex,y ∼q [log p(x,y) > p(x)p(y)] where p is the model distribution and q is the true distribution of the data.

This must be smaller than the cross-entropy loss of the model on both X and Y. 5with a power β ∼ 0.7 for all modalities, as shown in figure 2. As compute budgets grow, it’s best to devote a majority of resources towards training larger models. This strongly suggests sub-linear scaling of D ∝ N 0.4 for dataset size with model size during compute-optimal training. • For each domain, there is an optimal aspect ratio dmodel /n layer for the Transformer. Most data modalities require smaller aspect ratios e. deeper networks) as compared to language [KMH +20].

• We study an apparent inconsistency between L(D) and L(C) trends in section 6. We also find a number of results specific to certain domains, though we expect that many of the lessons are more general. For image and video modeling (see section 3): • When generative image models are finetuned for ImageNet classification, we find a power-law for classification loss vs model size (see figure 11), even beyond the model size where we approach the irreducible loss for generative modeling. We conclude that the approach to the irreducible loss does not necessarily indicate diminishing returns for representation quality or semantic

• We explore scaling trends for individual images and for percentiles of the image loss distribution (see figures 17, 10, 20, 21). We find that the loss on individual images scales with model size in the same way as the mean over all images in the data distribution. We expect similar behavior in other data modalities. • We test a variety of image resolutions (see figure 8), and find distinct scaling exponents and irre-ducible losses for each. We also test two VQVAE [vdOVK18] based models. • We examine scaling of the loss with video frame index (see figures 6 and 9).

For multimodal models (see section 4): • We explore the mutual information between captions and images (see figure 12), and the information gain defined in equation (1.3). We find a smooth scaling for both the mutual info and information gain with model size • We revisit the question “Is a picture worth a thousand ” by comparing the information-content of textual captions to the image/text mutual information. For mathematical problem solving (see section 5 and appendix B): • We explore the ability of models to extrapolate from the training distribution to increasingly more challenging problems.

We find that extrapolation performance depends predominantly on perfor-mance on the training distribution (figure 24), and is otherwise independent of model size. So while larger models perform better, model size does not provide benefits to ‘strong generalization’. • We provide a detailed breakdown of performance by math problem type (see appendix B). 2 Central Empirical Scaling Laws in Each Domain In this section we will describe our common experiments in each domain and our results establishing equation (1.1) for compute, model size, and (in a few cases) dataset size scaling. 2.1 Domain Descriptions and Training Setups In every domain we use decoder-only transformer models trained using an autoregressive cross-entropy loss.

For many models we use a sparse attention pattern [CGRS19], though we use dense attention when solving math problems. The transformers used for language and multimodal modeling have fully connected layers of size 4dmodel and attention layers of size dmodel, in the notation of [KMH +20, BMR +20]. For math, image, and video modeling we scale the FC layers to dmodel and the attention layers to dmodel /4. We use an aspect ratio dmodel /n layer ≈ 10 for math, images, and videos as we find that this is approximately optimal, meaning that these domains prefer much deeper models as compared to language [KMH +20], where the optimal aspect ratio ∼ 100.

Thus our math, image, and video models are essentially identical, differing only in context length. For math alone we used a weight decay [LH17] of 0.05. We provide more detailed hyperparameter settings in appendix F. 6Domain L(N) (model size) L(C) (compute) Nopt (C) Language (N > 1.47 ×10 14)−0.070 (C > 3.47 ×10 8)−0.048 (C > 3.3×10 −13)0.73 Image 8x8 3.12 + (N > 8.0×10 1)−0.24 3.13 + (C > 1.8×10 −8)−0.19 (C > 5.3×10 −14)0.64 Image 16x16 2.64 + (N > 2.8×10 2)−0.22 2.64 + (C > 1.6×10 −8)−0.16 (C > 4.8×10 −12)0.75 Image 32x32 2.20 + (N > 6.3×10 1)−0.13 2.21 + (C > 3.6×10 −9)−0.1 (C > 1.6×10 −13)0.65 Image VQ 16x16

3.99 + (N > 2.7×10 4)−0.13 4.09 + (C > 6.1×10 −7)−0.11 (C > 6.2×10 −14)0.64 Image VQ 32x32 3.07 + (N > 1.9×10 4)−0.14 3.17 + (C > 2.6×10 −6)−0.12 (C > 9.4×10 −13)0.7 Text-to-Im (Text) (N > 5.6×10 8)−0.037 (combined text/image loss) Text-to-Im (Image) 2.0 + (N > 5.1×10 3)−0.16 1.93 + (C > 1.5×10 −6)−0.15 (C > 9.4×10 −13)0.7 Im-to-Text (Text) (N > 7.0×10 8)−0.039 (combined text/image loss) Im-to-Text (Image) 2.0 + (N > 5.5×10 3)−0.15 1.97 + (C > 1.5×10 −6)−0.16 (C > 3.3×10 −12)0.72 Video VQ 16x16x16 1.01 + (N > 3.7×10 4)−0.24 0.95 + (C > 2.2×10 −5)−0.14 (C > 1.13 ×10 −12)0.71 Math (Extrapolate) 0.28 + (N > 1.1×10 4)−0.16 0.14 + (C > 1.4×10 −5)−0.17 (C > 2.3×10 −12)0.69 Table 1 Summary of scaling laws— In this table we summarize the model size and compute scaling fits to equation (1.1) along with Nopt (C), with the loss in nats/token, and compute measured in petaflop-days.

In most cases the irreducible losses match quite well between model size and compute scaling laws. The math compute scaling law may be affected by the use of weight decay, which typically hurts performance early in training and improves performance late in training. The compute scaling results and data for language are from [BMR +20], while Nopt (C) comes from [KMH +20]. Unfortunately, even with data from the largest language models we cannot yet obtain a meaningful estimate for the entropy of natural language. 1.1 Language We show results from GPT-3 [BMR +20] for comparison, including the performance of much larger models than we train in other domains.

In figure 2 we use the optimal model size trend from [KMH +20]. In appendix D we show some experiments on the scaling of arithmetic and factual question answering abilities, and make some additional qualitative observations about the progression of language understanding with scale. 1.2 Images We study a dataset of approximately 10 8 web images [TSF +15] scaled to pixel resolutions R × R = 8x8, 16x16, and 32x32 represented in raster order using RGB colors, each in the range [0, 255], giving a total of 3R2 tokens per image. We also study the same images at 64x64 resolution but VQ [vdOVK18] encoded with either a 16x16 or 32x32 VQ encoding pattern, for a total of either 256 or 1024 tokens per image.

To reduce compute, we use sparse attention patterns [CGRS19], alternating between locally-banded attention and fixed-stride attention in sequential layers, where both the local context length and fixed-stride length are given by the side-length in tokens of the square images. 1.3 Video We study a dataset of approximately 7×10 5 videos totaling about 100 hours scraped from the web, where each frame is scaled to a pixel resolution of 64x64. Each individual frame is encoded with the same 16x16 VQVAE [vdOVK18] used for images, resulting in 256 tokens per frame. We train on sequences of 16 sequential frames, resulting in a total of 4096 tokens per video.

As with images, we reduce compute by using a sparse attention pattern [CGRS19] alternating between locally-banded and fixed-stride attention, where both the local context length and fixed-stride length are given by the side length in tokens of the square frames. 7Input Resolution (pixels) Output Resolution (VQ Codes) Codebook Size 64x64 16x16 4096 64x64 32x32 1024 Table 2 Details of VQVAEs used to encode images and frames of video. 1.4 VQ Encoding The VQVAE models mentioned in 1.2 and 1.3 were trained on frames of the web-scraped videos described in 1.3, using the VQ-VAE architecture [vdOVK18] with modifications described in [DJP +20], including dead code revival.

More details can be found in table 2. 1.5 Multimodal Text and Images Multimodal models are trained to autoregressively predict both image tokens and language tokens in se-ries. We simply concatenate together the token lists for BPE encoding of text (using the tokenization of [BMR +20]) and the [0, 255] colorscale of each of the RGB pixels in the images, and let the model learn the necessary embedding matrix. We separately study models for text-to-image and image-to-text mappings, as we found poor performance for bidirectional models in preliminary experiments. For both image-to-text and text-to-image models we compute the mean pixel and mean text token loss, and then weight them to form the total loss L = 9 Limage + Ltext, as we found this weighting produced good results in a scan.

We use 32x32 images together with a 128-token captions (padded or trimmed as needed), for a total context length of 3200 tokens per image/caption pair. For the multimodal dataset we used a wide variety of image/text pairs curated through web search. 1.6 Mathematical Problem Solving Mathematical problem solving would seem to be a rather different domain from generative language, image, video, and multimodal modeling. To solve math problems, a model needs to learn to execute an algorithm to arrive at a deterministic answer. In contrast, the other distributions we have studied are typically genuinely probabilistic, and at least at an intuitive level, seem to require something a bit different from the simple algo-rithms that perform arithmetic or solve equations.

We have included math problems to probe the generality of scaling laws and transformer performance. We train and test models using the math problem generator [SGHK19], which generates a variety of prob-lems in algebra, arithmetic, calculus, comparisons, numbers (integer properties), measurement, polynomials, and probability. When studying model and compute-budget scaling we procedurally generate the training problems in an online setting. We sample the default mixture of easy, medium, and hard problems, without a progressive curriculum. When studying dataset size scaling we use static training data sampled from the same distribution. As discussed further in appendix B, the data distribution has some unusual features, as easier problems will naturally appear more often than more difficult problems.

A few problem types require interpreting both numbers and strings as sequences of individual characters, so for simplicity we model all questions and responses at the character (byte) level. The model receives the problems as plain text, and we fill a transformer’s 512 -token context window with concatenated problems, using a mask so that only the tokens corresponding to answers contribute to the loss. The problem generator 5 [SGHK19] can be provided with an ‘entropy’ s. The training distribution samples from s ∈ [3, 10], while interpolate-testing corresponds to s = 8, and the extrapolate test involves s = 12,along with some other extensions to increase compositionality.

In the online setting, we cannot be sure the interpolate tests are deduplicated from the training data, but the extrapolate test must be. To supplement the test data and further study extrapolation, we generated new test sets with s ∈ [1, 19], with larger s posing a greater challenge to the model, as s > 10 is literally out of the training distribution, and requires extrapolation. We found consistently poor performance on the two extrapolation generators probability__swr_p_level_set_more_samples and probability__swr_p_sequence_more_samples from [SGHK19], with larger models overfitting against them and achieving worse loss (but higher accuracy) > 5The generator settings vary somewhat among problem types, with some depending on more parameters.

810 1 10 0 10 1 > Aspect Ratio > 2.4 × 10 0 > 2.45 × 10 0 > 2.5 × 10 0 > 2.55 × 10 0 > 2.6 × 10 0 > Loss > 32x32 Images: Best Aspect Ratio > 10 6 > 10 7 > 10 8 > Parameters 10 010 110 2 > Aspect Ratio > 0.2 0.3 0.4 0.5 0.6 > Loss > Math: Extrap. (Solid), Interp. (Dashed) > 10 8 > 6 × 10 7 > 2 × 10 8 > Parameters 10 010 110 2 > Aspect Ratio > 2.12 × 10 0 > 2.14 × 10 0 > 2.16 × 10 0 > 2.18 × 10 0 > 2.2 × 10 0 > 2.22 × 10 0 > 2.24 × 10 0 > Loss > Text-to-Image, 27M Params Figure 4 Optimal aspect ratio— We show trained performance as a

function of the aspect ratio, defined as width / depth, or more precisely ≡ dmodel /n layer. The optimal aspect ratio for language [KMH +20] was about 10x larger. than some smaller models. So we have not included their contribution in figures 1 and 5, as the poor loss on these modules would dominate the trends. We provide more details and many additional results on math in appendix B, including results per module, dataset size 6 scaling, and further analysis of performance vs difficulty level. There we also show trends for the training loss, which do not adhere as well to a power-law form, perhaps because of the implicit curriculum in the frequency distribution of easy and hard problems.

2.2 Model Size Scaling and Aspect Ratios Arguably the simplest scaling relation compares the loss achieved by models of various sizes N once they are trained to convergence with a dataset large enough to obviate overfitting. Throughout this paper we report N as the number of non-embedding parameters in a transformer model, motivated by prior results on language [KMH +20]. Results for the scaling of L(N) are depicted in figure 3, along with fits to equation (1.1). We define L(N) using the loss at convergence (practically, this means as close to convergence as is feasible), but the largest models we study will not have fully converged.

Thus caution is warranted when interpreting L(N) trends according to equation (1.2) and identifying the irreducible loss as an entropy, and the reducible loss as a KL divergence. Nevertheless, the reducible losses typically fit very well to a pure power-law trend. As an aside, we often find intriguingly good power-law plus constant trends when recording the loss after training all models for a fixed number of training steps. We have found that for any given data modality, transformer models typically have an ideal aspect ratio dmodel /n layer that maximizes performance while holding model size N fixed.

In figure 4 we display con-verged performance as a function of aspect ratio for a few model sizes in several domains. We see that image and math models perform optimally with an aspect ratio ≈ 5, which suggests that on these domains we should aim for deeper and thinner models, with at least a 10x smaller aspect ratio compared to optimized language models. The difference may be even greater due variations in mattn and mmlp settings. Finally, note that image and video models with roughly 10 4 parameters under-perform the trends, with worse performance evident for higher resolution images.

The video models must attend to a 4096 -token context, while 32x32 images have a 3072 -token context, so we speculate that tiny models under-perform because they have difficulty attending to contexts comparable in length to their non-embedding parameter count. 2.3 Compute Scaling and Optimal Model Sizes Instead of focusing on converged performance, one can study the loss L achieved with a finite training compute budget C when training with a large enough dataset to avoid overfitting. We define C theoretically rather than empirically, and approximate 7 it as C ≡ 6N E where N is the non-embedding parameter count (model size) and E = SB is the total

number of tokens processed during training (with S the number of parameter updates and B the batch size in tokens). The results for L(C) from a variety of model sizes are depicted in figure 5, along with the pareto-frontier of optimal loss for a given compute budget, and a power-law plus constant fit forced to lie below this frontier. > 6 The math models in figure 4 used mmlp = 4, m attn = 1 like language models, unlike the math models used to study model and compute scaling, as these aspect ratio tests were performed earlier.

> 7 The factor of 6 includes a factor of 2 for add-multiply and a 3 to include forward and backward passes. 910 8 10 6 10 4 10 2 10 0 > Compute (PF-days) > 6 × 10 2 > 7 × 10 2 > Loss > Compute Scaling for 8x8 Images > 601.54 + (X > 1.9 e+ 03)0.19 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 610 410 210 0 > Compute (PF-days) > 2 × 10 0 > 3 × 10 0 > Loss > Text-to-Image Loss vs Compute > 1.93 + (X > 1.5 e06)0.15 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 610 410 210 0 > Compute (PF-days) > 2 × 10 0 > 3 × 10 0 > 4 × 10 0 > 6 × 10 0 > Loss > Video Compute Scaling > 0.95 + (X > 2.2 e05)0.14 > 10 4 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 610 410 210 0 > Compute (PF-days) > 10 0 > Loss > Proc.

Gen. Extrapolate Loss vs Compute > 0.15 + (X > 1.4 e05)0.18 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 610 410 210 0 > Compute (PF-days) > 2 × 10 0 > 3 × 10 0 > Loss > Image-to-Text Loss vs Compute > 1.97 + (X > 1.5 e06)0.16 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 610 410 210 010 210 4 > Compute (PF-days) > 1.5 23456 > Loss Language Modeling > L= 2.57 C0.048 > 10 5 > 10 6 > 10 7 > 10 8 > 10 9 > 10 10 > 10 11 > Parameters Figure 5 Scaling laws with compute— Scaling laws with compute (total estimated floating point opera-tions) for various domains, along with power-law plus constant fits (dashed).

This is identical to figure 1, except that we do not subtract the fitted constant irreducible loss. Note that very small models underperform compared to the trends when they model images or videos with very large contexts. Note also that the largest language models [BMR +20] were not trained to convergence. The compute trends are most relevant for differentiating between the irreducible loss and reducible losses, since they avoid the issue of training to convergence, which makes the interpretation of L(N) difficult. We display the reducible loss trends for L(C) in figure 1, and emphasize that these appear to be pure power-laws, even when the reducible loss is much smaller than the irreducible loss.

We can use the L(C) trends to estimate the model size Nopt that optimizes the loss when training is con-strained by a fixed compute 8 budget C. For this purpose we select points on the convex hull of the loss versus compute frontier; these can be seen as blue points in figure 5. The results for all domains together appear in figure 2, while each domain is shown separately with individual fits in figure 16. In all cases we find that Nopt (C) ∝ Cβ can be fit with a pure power-law, with all exponents fairly close to β ∼ 0.7.

This suggests that one should spend most of a growing training compute budget by training much larger generative models. When estimating Nopt (C), one might worry about errors due to a sub-optimal usage of data. Specifically, if the batch size is too large early in training, then some compute may effectively be wasted. This can be studied by identifying the critical batch size [MBB17, MKAT18] above which there are diminishing returns to further data parallelism. In prior work [KMH +20] this was taken into account by measuring the critical batch size and using relations derived in [MKAT18] to adjust compute estimates.

We have not made this adjustment here, as it would require a number of additional experiments in order to measureme the critical batch size in each domain. For large model sizes and compute budgets these effects should be small, because most or all of training involves batches smaller than the critical batch size (which grows quickly during training [MKAT18]), but this issue may be worth revisiting in the future. The total number of tokens processed during all of training is E = C 6N ≥ D, where D is the dataset size, with equality representing training for only a single epoch.

This means that D ∝ C1−β ∝ N 1−ββ. We clearly have β > 0.6 for all data modalities and by a comfortable margin, suggesting that dataset size should not grow faster than D ∝ N 2/3 during compute-optimal training, with a more reasonable median estimate of D ∝ N 0.4. This unambiguously sub-linear scaling across all data modalities runs somewhat counter to conventional wisdom. As a word of caution, we have yet to train models in a regime where compute optimal training actually implies D  N numerically. We discuss this further in section 6.

> 8 For a fixed amount of training compute, we can train smaller models at the expense of worse performance. Hence, when accounting for both inference and training compute, the optimal model size may be somewhat smaller than de-scribed here. See [KMH +20] for a discussion this tradeoff. 10 0 20 010 20 30 > 32x32 > 010 0510 15 > 16x16 > 050246 > 8x8 > 2.1 2.2 2.3 2.4 2.5 2.6 2.7 > 2.4 2.5 2.6 2.7 2.8 2.9 > 2.9 3.0 3.1 3.2 3.3 > Image Losses per Pixel for 400M Model 10 12 × 10 03 × 10 0 > 4 × 10 06 × 10 0 > Frame Index > 10 0 > 9 × 10 1 > 1.1 × 10 0 > 1.2 × 10 0 > 1.3 × 10 0 > Mean Loss per Frame > Video: Loss per Frame > 10 7 > 10 8 > Parameters Figure 6 Position-dependent loss for images and video— We show trends for the loss as a function of position in the context for image and video models.

On the left we have the mean loss over the three colors for images of various resolutions. The top-left pixel actually has significantly higher loss, off the color scale, which was set to make the pattern clear for the image as a whole. On the right we see the mean loss per frame for video models, as a function of the frame index. The oscillatory behavior per frame is due to the video encoding. 2.4 Loss versus Position in the Context Depends on the Structure of the Data Some trends in the loss are highly dependent on the structure of the data.

A clear example of this is the loss as a function of the position in the context, ie the loss per token for language models, loss per frame for video models, or the loss per pixel in visual domains. We provide two examples in figure 6. Note that for images the very first pixel typically has a large loss, outside the color range shown; we chose not to extend the color range as it would have obscured the patterns in the remainder of the image. Language [KMH +20] and videos (per frame) show a power-law plus constant trend as a function of context position, as their data is naturally sequential.

However, these trends do not apply at all to image modeling, where the loss is largest for the first pixels and near the center of the image. Thus power-law correlations in the context depend in an essential way on the nature of the data, and are not universal. In contrast, the form of the compute and model size scaling laws appears to be largely independent of the data distribution. 3 Image and Video Modeling, the Reducible Loss, and Downstream Tasks Image data can be presented at a wide variety of resolutions, or it may be compressed, for example with VQ codes [vdOVK18].

These settings provide a way to modify the complexity of the data distribution, creating a useful arena for the study of neural scaling laws. Furthermore, we can finetune generative image models for classification to explore the quality of their learned features. We will use these tools to explore the nature of the reducible and irreducible loss. In particular, at very low resolution (8x8) we can follow the power-law trend in the reducible loss all the way to a few nats/image, which can be achieved by models approaching a billion parameters. This gives us some reason for optimism when extrapolating

similar trends on larger images beyond the realm that we can currently explore. It also strongly suggests that the power-law plus constant form of equation (1.1) will remain an excellent approximation. Furthermore, we will show that improvement in fine-tuned classification performance continues smoothly even as the generative loss approaches the irreducible loss. This result strongly suggests that representation quality continues to improve smoothly even when the generative loss trend appears to taper off. 3.1 Varying the Image Resolution and Encoding We trained Transformers on the YFCC100m dataset after scaling images down to 8x8, 16x16, and 32x32 pixel resolutions, along with 64x64 images encoded with VQ codes [vdOVK18] with 16x16 and 32x32 VQ code patterns.

We display the trends for the reducible loss per image as a function of the compute budget in figure 8 (see figure 18 in the appendix for trends for the full loss). We include these figures to emphasize that the reducible loss for an optimally-allocated compute budget follows a power-law trend, even when the reducible loss becomes very small. Note that the smallest models underperform as compared to the trends at resolutions greater than 8x8. We see this both for the compute trends in figure 8 as well as in model-size trends in figure 7.

We speculate that this is due to difficulty utilizing the positional encodings. For example, our smallest models have only 10k 11 10 4 10 5 10 6 10 7 10 8 10 9 Parameters > 2.4 × 10 0 > 2.6 × 10 0 > 2.8 × 10 0 > 3 × 10 0 > 3.2 × 10 0 > 3.4 × 10 0 > Loss Image Modeling > 10 15 20 25 30 > Image Size (Pixels) 10 410 510 610 710 8 Parameters > 4 × 10 0 > 5 × 10 0 > Loss 64x64 Image Modeling with VQ Codes VQ 16x16 3.99 + (X > 2.7 e+ 04) 0.13 VQ 32x32 3.07 + (X > 1.9 e+ 04) 0.14 10 4 10 5 10 6 10 7 10 8 Parameters > 10 1 > 10 2 > 10 3 > Reducible Loss (nats/image) Pixel-Level Image Modeling (X/3.91 e + 11) 0.236 (X/1.42 e + 16) 0.210 (X/4.61 e + 27) 0.137 > 10 1 > 2 × 10 1 > 3 × 10 1 > Image Size (Pixels) 10 410 510 610 710 8 Parameters > 10 2 > 10 3 > Reducible Loss per Image VQ Code Reducible Loss 16x16 VQ (X/2.29 e + 22) 0.133 32x32 VQ (X/1.70 e + 25) 0.144 Figure 7 Comparison of image resolutions (model size scaling)— Top: We display scaling laws with model size for various image resolutions, and also with various VQ encodings, along with power-law plus constant fits (dashed) to equation (1.1).

The fits for pixel-level image modeling are shown in table 3. Note that the tiniest (10k non-embedding parameter) pixel models underperform at higher resolutions; we suspect they have difficulty recognizing relative positions in larger images. These deficiencies are even more clearly visible in the compute trends. Bottom: We show the reducible losses, which estimate the KL divergence between the true probability distribution over images and the distribution predicted by our models. We show the result as a function of model size and image resolution or encoding, along with pure power-law trends. 10 8 10 6 10 4 10 2 10 0 Compute (PF-days) > 10 1 > 10 2 > 25 50 75 125 150 > Reducible Loss per Image 8x8 Pixel Images > (X > 1.9 e+ 03)0.19 > 10 5 > 10 6 > 10 7 > 10 8

> Parameters 10 710 510 310 110 1 Compute (PF-days) > 10 2 > 200 300 400 500 600 > Reducible Loss per Image 16x16 Pixel Images > (X > 1.7 e+ 10)0.16 10 5 > 10 6 > 10 7 > 10 8 > 10 9 > Parameters 10 710 510 310 110 1 Compute (PF-days) > 10 3 > 800 1600 2400 3200 4000 > Reducible Loss per Image 32x32 Pixel Images > (X > 2.7 e+ 26)0.10 10 5 > 10 6 > 10 7 > 10 8 > 10 9 > Parameters 10 810 610 410 210 0 Compute (PF-days) > 10 2 > 60 120 180 240 300 360 > Reducible Loss per Image 16x16 VQ Encoded Images > (X > 6.1 e07)0.11 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 810 610 410 210 0 Compute (PF-days) > 10 3 > 300 600 900 1200 1500 1800 > Reducible Loss per Image 32x32 VQ Encoded Images > (X > 2.6 e06)0.12 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters Figure 8 Comparison of image resolutions (compute scaling)— We display scaling of the reducible loss with compute for pixel-level image modeling at various resolutions (first line), and for various VQ encodings of 64x64 images (second line).

We show the test loss, but we did not observe any train/test gap for these models. A few models diverged late in training. 12 Resolution Reducible Loss per Image (nats) Irreducible Loss per Image (nats) 8x8 (C > 1.9×10 3)−0.19 602 16x16 (C > 1.7×10 10)−0.16 2026 32x32 (C > 2.7×10 26)−0.1 6806 64x64 (16x16 VQ) (C > 4.7×10 15)−0.11 1047 64x64 (32x32 VQ) (C > 3.1×10 19)−0.12 3246 Table 3 Per-image loss trends— Fits for the reducible and irreducible loss as a function of compute for various image resolutions, shown per-image rather than per-token as in table 1.

Here compute C is measured in PF-days, so the denominators estimate the amount of compute needed to achieve a reducible loss of 1 nat/image. The irreducible losses estimate the entropy of the YFCC100M data distribution [TSF +15]. non-embedding parameters, while 32x32 images include 3072 tokens in their context, each with a distinct positional embedding. To understand the significance of the reducible loss trends in table 3, recall that the cross entropy loss between the true distribution P and the model distribution Q is Ex∼P [log 1 Q(x)] = DKL (P || Q) + S(P) (3.1) The KL divergence vanishes when P = Q, and is otherwise strictly non-negative.

Thus we can identify the irreducible loss with S(P), the constant entropy of the true distribution. Then the reducible loss estimates the KL divergence between the true distribution and the distribution predicted by the model. This interpretation can only make sense if in the limit of infinite data and compute, we expect the transformer to perfectly model the data distribution. We have focused on L(C) trends because the asymptotic limits of the model size trend L(N) could be misleading if the models have not all been trained fully to convergence. The power-law trends in DKL can be extrapolated down to the level of just a few nats per image.

Models powerful enough to reach this level of performance model the distribution of images with near-perfect fidelity. In fact we see that models with ∼ 1B parameters nearly achieve this feat for 8x8 ‘images’. However, we see that for larger images we would need enormous quantities of compute to perfectly model the true image distribution. The consistency of the trends among distinct image resolutions in figure 7 and the strikingly small reducible loss for the 8x8 case suggests that if we could run much larger models, we would continue to see smooth improvements at higher resolution.

It seems that compute requirements for a near-perfect model of the data distribution grow as a steep power-law or even an exponential in the image resolution. Of course we do not expect to need a perfect model of the probability distribution of real-world images for practical tasks. 3.2 Video Modeling and Individual Frames For the case of video modeling, it is natural to extend the overall trends to the study of specific frames. We display several frame-dependent results in figure 9. On the left we show loss as a function of model size, omitting the first frame, which has a much larger loss and should be considered an image modeling problem.

In the center we show compute scaling of the reducible loss on the final frame. On the right in the same figure we show the reducible loss for the final (16th) frame, which is of particular interest when generating a continuation of an existing video. Much like the trends for image modeling, we see that the reducible loss is very well approximated by a power-law, making it possible to forecast that we would need a model size around ∼ 10 13 parameters and compute of around 10 4 PF-days to achieve a loss of just a few nats/frame on the final frame of this type of video.

13 10 5 10 6 10 7 10 8 10 9 > Parameters > 10 0 > 1.2 × 10 0 > 1.4 × 10 0 > 1.6 × 10 0 > 1.8 × 10 0 > Loss > Video Loss by Frame Index > 10 1 > 2 × 10 0 > 3 × 10 0 > 4 × 10 0 > 6 × 10 0 > Frame Index 10 910 710 510 310 110 1 > Compute (PF-days) > 10 2 > 10 3 > 500 1500 2000 2500 > Reducible Loss / Frame Final Video Frame Reducible Loss > (C > 1.8 × 10 5)0.19 > 10 4 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 510 610 710 810 9 > Parameters > 10 2 > nats/frame > Final Frame Reducible Loss Trend > Reducible Loss of Final Frame (X/3.85 e+ 13) 0.277 Figure 9 Per-frame video performance trends — On the left we show scaling trends for specific frames in 16-frame videos.

In the center we show the reducible loss as a function of compute for the final frame of the video. On the right we show the reducible loss and its pure power-law trend with model size for the final frame in a video. 10 4 10 5 10 6 10 7 10 8 > Parameters 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Loss > Loss Trends in Percentiles of the Image Distribution > 20 40 60 80 Percentile Figure 10 Performance trends for image dataset percentiles— We selected one thousand images from the 32x32 image test set, and evaluated the loss of all models on each image.

In this figure we plot the trends in the 1, 5, 20, 50, 80, 95, 99 percentiles of the loss distribution over these images, along with power-law plus constant fits (dashed). We also observe similar trends for randomly chosen individual images (figure 17). 3.3 Scaling Trends for Individual Images We have observed very consistent scaling trends on a variety of data modalities. This raises a question – does the loss achieved by different sized models on specific, individual data examples scale in the same way? Or are the distribution-level trends an aggregate of many different trends on individual examples?

To answer these questions, we evaluated the loss of all the pixel-level 32x32 image models on a thousand randomly chosen images from the test set. When plotting the loss as a function of model size for individual, randomly chosen examples, in essentially all cases we observe a smooth, power-law plus constant trend. To convey this information, for each model size we evaluate the 1,5,20,50,80,95, and 99 percentile of the loss among a thousand images in the distribution, for each model size. We then plot the trends in these percentile losses in figure 10. We see very similar trends among all percentiles of the loss distribution, and all are well-described by equation (1.1).

We show model size trends for eight randomly chosen individual test images in figure 17. We also display the most and least improved 10 images from a sample of one thousand test images in figure 20. Finally, we visualize the trends in a different way, by generating conditional samples at each model size, in figure 21. We would expect that these findings also apply to other data modalities. On a quick inspection, we found the same patterns for randomly chosen text sequences and language models of different sizes. 3.4 Finetuning on ImageNet at 32x32 Resolution By finetuning generative models for image classification we gain another handle on the scaling of perfor-mance with model size.

We use the scaled-down 32x32 resolution ImageNet [CLH17] and finetune the 32x32 resolution pixel-level generative image models. To turn these models into classifiers, we remove their final embedding matrix and use the mean-pooled (over all pixels) activations of the transformer’s final layer as the input to a new single-layer classifier. During 14 10 5 10 6 10 7 10 8 10 9 > Parameters > 2 × 10 0 > 3 × 10 0 > 4 × 10 0 > Classification Test Loss ImageNet (32x32) Classification > Pretrained (X/1.72 e+ 10) 0.105 > From Scratch 10 510 610 710 810 9 > Parameters > 4 × 10 1 > 6 × 10 1 > Classification Top 1 Error Rate ImageNet (32x32) Classification > Pretrained (X/2.09 e+ 03) 0.089 > From Scratch 10 210 310 410 5 > Step > 1.5 3.0 4.5 6.0 7.5 > Classification Test Loss Pretrained (Solid) vs Scratch (Dashed) > 10 4 > 10 5 > 10 6 > 10 7 > 10 8 > Parameters 10 210 310 410 5 > Step > 10 0 > 0.45 0.60 0.75 0.90 1.05 > Test Top 1 Error Rate Pretrained (Solid) vs Scratch (Dashed) > 10 5 > 10 6 > 10 7 > 10 8 > Parameters Figure 11 Trends in image classification performance— Top: We show model size scaling results for 32x32 pixel ImageNet [CLH17] classification.

We compare models trained from scratch on ImageNet clas-sification (ie with no pre-training) to finetuned generative models. Though the generative loss trend bends as it approaches the irreducible loss (figure 7), the pretrained models exhibit a straight power-law trend in clas-sification performance vs model size, which also continues far beyond the point where the models that were trained from scratch exhibit overfitting. Bottom: Larger pre-trained models fine-tune significantly faster, and to significantly better performance,