Scaling Laws for Neural Language Models

# Scaling Laws for Neural Language Models Jared Kaplan ∗ Johns Hopkins University, OpenAI edu Sam McCandlish ∗ OpenAI com Tom Henighan OpenAI com Tom B. Brown OpenAI com Benjamin Chess OpenAI com Rewon Child OpenAI com Scott Gray OpenAI com Alec Radford OpenAI com Jeffrey Wu OpenAI com Dario Amodei OpenAI com ## Abstract We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.

Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence. > ∗ Equal contribution. Contributions: Jared Kaplan and Sam McCandlish led the research. Tom Henighan contributed the LSTM ex-periments.

Tom Brown, Rewon Child, and Scott Gray, and Alec Radford developed the optimized Transformer implementation. Jeff Wu, Benjamin Chess, and Alec Radford developed the text datasets. Dario Amodei provided guidance throughout the project. > arXiv:2001.08361v1 LG] 23 Jan 2020 ## Contents 1 Introduction 22 Background and Methods 63 Empirical Results and Basic Power Laws 74 Charting the Infinite Data Limit and Overfitting 10 5 Scaling Laws with Model Size and Training Time 12 6 Optimal Allocation of the Compute Budget 14 7 Related Work 18 8 Discussion 18 Appendices 20 A Summary of Power Laws 20 B Empirical Model of Compute-Efficient Frontier 20 C Caveats 22 D Supplemental Figures 23 ## 1 Introduction Language provides a natural domain for the study of artificial intelligence, as the vast majority of reason-ing tasks can be efficiently expressed and evaluated in language, and the world’s text provides a wealth of data for unsupervised learning via generative modeling.

Deep learning has recently seen rapid progress in lan-guage modeling, with state of the art models [RNSS18, DCLT18, YDY +19, LOG +19, RSR +19] approaching human-level performance on many specific tasks [WPN +19], including the composition of coherent multi-paragraph prompted text samples [RWC +19]. One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this training process. In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP +17, LSP +18].

The high ceiling and low floor for performance on language tasks allows us to study trends over more than seven orders of magnitude in scale. Throughout we will observe precise power-law scalings for performance as a function of training time, con-text length, dataset size, model size, and compute budget. 1.1 Summary Our key findings for Transformer language models are are as follows: > 2Here we display predicted compute when using a sufficiently small batch size. See Figure 13 for comparison to the purely empirical data. 2Dataset Size > tokens > Parameters > non-embedding > Compute > PF-days, non-embedding > Test Loss Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute 2 used for training.

For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two. Performance depends strongly on scale, weakly on model shape: Model performance depends most strongly on scale, which consists of three factors: the number of model parameters N (excluding embed-dings), the size of the dataset D, and the amount of compute C used for training. Within reasonable limits, performance depends very weakly on other architectural hyperparameters such as depth vs. width. (Section 3) Smooth power laws: Performance has a power-law relationship with each of the three scale factors N, D, C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1).

We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3) Universality of overfitting: Performance improves predictably as long as we scale up N and D in tandem, but enters a regime of diminishing returns if either N or D is held fixed while the other increases. The performance penalty depends predictably on the ratio N 0.74 /D, meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty.

(Section 4) Universality of training: Training curves follow predictable power-laws whose parameters are roughly independent of the model size. By extrapolating the early part of a training curve, we can roughly predict the loss that would be achieved if we trained for much longer. (Section 5) Transfer improves with test performance: When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss – in other words, transfer to a different distribution incurs a constant penalty but otherwise improves roughly in line with performance on the training set.

(Section 2.2) Sample efficiency: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4). Convergence is inefficient: When working within a fixed compute budget C but without any other restric-tions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C0.27 with training compute.

(Section 6) Optimal batch size: The ideal batch size for training these models is roughly a power of the loss only, and continues to be determinable by measuring the gradient noise scale [MKAT18]; it is roughly 1-2 million tokens at convergence for the largest models we can train. (Section 5.1) Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. We expect that larger language models will perform better and be more sample efficient than current models. 3Larger models require fewer samples to reach the same performance > 10 > 8 > 6 > 4 The optimal model size grows smoothly with the loss target and compute budget > Line color indicates > number of parameters > 10 710 910 11 > Tokens Processed Compute (PF-days) > 10 -9 10 -6 10 -3 10 0 > Test Loss > Compute-e ffi cient > training stops far > short of convergence > 10 310 910 6 > 10 3Params > 10 9Params > 10 > 8 > 6 > 4 Figure 2 We show a series of language model training runs, with models ranging in size from 10 3 to 10 9 parameters (excluding embeddings).

100x Batch Size <10x Serial Steps >1,000,000x Model Size > Data requirements > grow relatively slowly > Optimal model size > increases very quickly > Minimum serial steps > increases negligibly Figure 3 As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps. We illustrate this for a billion-fold increase in compute. For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required.

1.2 Summary of Scaling Laws The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters N, the dataset size D,or the optimally allocated compute budget Cmin (see Figure 1): 1. For models with a limited number of parameters, trained to convergence on sufficiently large datasets: L(N) = (Nc/N)αN; αN ∼ 0.076, Nc ∼ 8.8 × 10 13 (non-embedding parameters) (1.1) 2. For large models trained with a limited dataset with early stopping: L(D) = (Dc/D)αD; αD ∼ 0.095, Dc ∼ 5.4 × 10 13 (tokens) (1.2) 3.

When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal 3 use of compute): L(Cmin) = (Cmin c /C min)αmin > C; αmin > C ∼ 0.050, Cmin c ∼ 3.1 × 10 8 (PF-days) (1.3) > 3 We also observe an empirical power-law trend with the training compute C (Figure 1) while training at fixed batch size, but it is the trend with Cmin that should be used to make predictions. They are related by equation (5.5). 410 7 10 8 10 9 10 10 > Tokens in Dataset 2.5 3.0 3.5 4.0 4.5 Loss > Loss vs Model and Dataset Size > Params > 708M 302M 85M 3M 25M 393.2K 10 410 5 > Estimated Smin > 2.4 2.8 3.2 3.6 4.0 4.4 Loss > Loss vs Model Size and Training Steps > 10 6 > 10 7 > 10 8 > Parameters (non-embed) Figure 4 Left: The early-stopped test loss L(N, D) varies predictably with the dataset size D and model size N according to Equation (1.5).

Right: After an initial transient period, learning curves for all model sizes N can be fit with Equation (1.6), which is parameterized in terms of Smin, the number of steps when training at large batch size (details in Section 5.1). These relations hold across eight orders of magnitude in Cmin, six orders of magnitude in N, and over two orders of magnitude in D. They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [RWC +19]. The power laws αN, α D, α min > C specify the degree of performance improvement expected as we scale up N, D, or Cmin; for example, doubling the number of parameters yields a loss that is smaller by a factor 2−αN = 0.95.

The precise numerical values of Nc, C min c, and Dc depend on the vocabulary size and tokenization and hence do not have a fundamental meaning. The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ([MKAT18]), also roughly obeys a power law in L: Bcrit (L) = B∗ L1/α B, B∗ ∼ 2 · 10 8 tokens, αB ∼ 0.21 (1.4) Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to D ∝ N αNαD ∼ N 0.74. In fact, we find that there is a single equation combining (1.1) and (1.2) that governs the simultaneous dependence on N and D and governs the degree of overfitting: L(N, D) = [(Nc N) αNαD + Dc D]αD (1.5) with fits pictured on the left in figure 4.

We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks. When training a given model for a finite number of parameter update steps S in the infinite data limit, after an initial transient period, the learning curves can be accurately fit by (see the right of figure 4) L(N, S) = (Nc N)αN + (Sc Smin (S))αS (1.6) where Sc ≈ 2.1 × 10 3 and αS ≈ 0.76, and Smin (S) is the minimum possible number of optimization steps (parameter updates) estimated using Equation (5.4). When training within a fixed compute budget C, but with no other constraints, Equation (1.6) leads to the prediction that the optimal model size N, optimal batch size B, optimal number of steps S, and dataset size D should grow as N ∝ Cαmin > C/α N, B ∝ Cαmin > C/α B, S ∝ Cαmin > C/α S, D = B · S (1.7) with αmin > C = 1 / (1 /α S + 1 /α B + 1 /α N) (1.8) which closely matches the empirically optimal results N ∝ C0.73 min, B ∝ C0.24 min, and S ∝ C0.03 min.

As the computational budget C increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size (see Figure 3). This also implies that as models grow larger, they become increasingly sample efficient. In practice, researchers typically train smaller models for longer than would 5be maximally compute-efficient because of hardware constraints. Optimal performance depends on total compute as a power law (see Equation (1.3)). We provide some basic theoretical motivation for Equation (1.5), an analysis of learning curve fits and their implications for training time, and a breakdown of our results per token.

We also make some brief compar-isons to LSTMs and recurrent Transformers [DGV +18]. 1.3 Notation We use the following notation: • L – the cross entropy loss in nats. Typically it will be averaged over the tokens in a context, but in some cases we report the loss for specific tokens within the context. • N – the number of model parameters, excluding all vocabulary and positional embeddings • C ≈ 6N BS – an estimate of the total non-embedding training compute, where B is the batch size, and S is the number of training steps (ie parameter updates).

We quote numerical values in PF-days, where one PF-day = 10 15 × 24 × 3600 = 8.64 × 10 19 floating point operations. • D – the dataset size in tokens • Bcrit – the critical batch size [MKAT18], defined and discussed in Section 5.1. Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency. • Cmin – an estimate of the minimum amount of non-embedding compute to reach a given value of the loss. This is the training compute that would be used if the model were trained at a batch size much less

than the critical batch size. • Smin – an estimate of the minimal number of training steps needed to reach a given value of the loss. This is also the number of training steps that would be used if the model were trained at a batch size much greater than the critical batch size. • αX – power-law exponents for the scaling of the loss as L(X) ∝ 1/X αX where X can be any of N, D, C, S, B, C min. ## 2 Background and Methods We train language models on WebText2, an extended version of the WebText [RWC +19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size nvocab = 50257.

We optimize the autoregres-sive log-likelihood e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric. We record the loss on the WebText2 test distribution and on a selection of other text distributions. We primarily train decoder-only [LSP +18, RNSS18] Transformer [VSP +17] models, though we also train LSTM models and Universal Transformers [DGV +18] for comparison. 2.1 Parameter and Compute Scaling of Transformers We parameterize the Transformer architecture using hyperparameters nlayer (number of layers), dmodel (di-mension of the residual stream), dff (dimension of the intermediate feed-forward layer), dattn (dimension of the attention output), and nheads (number of attention heads per layer).

We include nctx tokens in the input context, with nctx = 1024 except where otherwise noted. We use N to denote the model size, which we define as the number of non-embedding parameters N ≈ 2dmodel nlayer (2 dattn + dff)= 12 nlayer d2model with the standard dattn = dff /4 = dmodel (2.1) where we have excluded biases and other sub-leading terms. Our models also have nvocab dmodel parameters in an embedding matrix, and use nctx dmodel parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N; we will see that this produces significantly cleaner scaling laws.

Evaluating a forward pass of the Transformer involves roughly Cforward ≈ 2N + 2 nlayer nctx dmodel (2.2) add-multiply operations, where the factor of two comes from the multiply-accumulate operation used in matrix multiplication. A more detailed per-operation parameter and compute count is included in Table 1. 6Operation Parameters FLOPs per Token Embed (nvocab + nctx) dmodel 4dmodel Attention: QKV nlayer dmodel 3dattn 2nlayer dmodel 3dattn Attention: Mask — 2nlayer nctx dattn Attention: Project nlayer dattn dmodel 2nlayer dattn dembd Feedforward nlayer 2dmodel dff 2nlayer 2dmodel dff De-embed — 2dmodel nvocab Total (Non-Embedding) N = 2 dmodel nlayer (2 dattn + dff) Cforward = 2 N + 2 nlayer nctx dattn Table 1 Parameter counts and compute (forward pass) estimates for a Transformer model.

Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted. For contexts and models with dmodel > n ctx /12, the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where dmodel  nctx /12,we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C ≈ 6N floating point operators per training token. 2.2 Training Procedures Unless otherwise noted, we train models with the Adam optimizer [KB14] for a fixed 2.5 × 10 5 steps with a batch size of 512 sequences of 1024 tokens.

Due to memory constraints, our largest models (more than 1B parameters) were trained with Adafactor [SS18]. We experimented with a variety of learning rates and schedules, as discussed in Appendix 6. We found that results at convergence were largely independent of learning rate schedule. Unless otherwise noted, all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero. 2.3 Datasets We train our models on an extended version of the WebText dataset described in [RWC +19]. The original WebText dataset was a web scrape of outbound links from Reddit through December 2017 which received at least 3 karma.

In the second version, WebText2, we added outbound Reddit links from the period of January to October 2018, also with a minimum of 3 karma. The karma threshold served as a heuristic for whether people found the link interesting or useful. The text of the new links was extracted with the Newspaper3k python library. In total, the dataset consists of 20.3M documents containing 96 GB of text and 1.62 × 10 10 words (as defined by wc). We then apply the reversible tokenizer described in [RWC +19], which yields 2.29 × 10 10 tokens. We reserve 6.6 × 10 8 of these tokens for use as a test set, and we

also test on similarly-prepared samples of Books Corpus [ZKZ +15], Common Crawl [Fou], English Wikipedia, and a collection of publicly-available Internet Books. ## 3 Empirical Results and Basic Power Laws To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding parameters) • Dataset size (ranging from 22 million to 23 billion tokens) • Shape (including depth, width, attention heads, and feed-forward dimension) • Context length (1024 for most runs, though we also experiment with shorter contexts) • Batch size (219 for most runs, but we also vary it to measure the critical batch size) 7Feed-Forward Ratio (d ff / d model) > 50M Parameters Aspect Ratio (d model / n layer)Attention Head Dimension (d model / n head) > 25M Parameters > 10% > 8% > 6% > 4% > 2% > 0% > Loss Increase > A wide range of architectures > achieve similar performance 22% additional compute > compensates for 1% loss increase Figure 5 Performance depends very mildly on model shape when the total number of non-embedding parameters N is held fixed.

The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to L(N) as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (nlayer, d model) = (6, 4288) reaches a loss within 3% of the (48, 1600) model used in [RWC +19]. 10 6 10 7 10 8 10 9 > Parameters (with embedding) 234567Test Loss > 0 Layer 1 Layer 2 Layers 3 Layers 6 Layers > >6 Layers 10 310 410 510 610 710 810 9 > Parameters (non-embedding) 234567Test Loss > 1 Layer 2 Layers 3 Layers 6 Layers > >6 Layers Figure 6 Left: When we include embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters.

Right: When we exclude embedding parameters, the performance of models with different depths converge to a single trend. Only models with fewer than 2 layers or with extreme depth-to-width ratios deviate significantly from the trend. In this section we will display data along with empirically-motivated fits, deferring theoretical analysis to later sections. 3.1 Approximate Transformer Shape and Hyperparameter Independence Transformer performance depends very weakly on the shape parameters nlayer, n heads, and dff when we hold the total non-embedding parameter count N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter.

This was simplest for the case of nheads. When varying nlayer,we simultaneously varied dmodel while keeping N ≈ 12 nlayer d2model fixed. Similarly, to vary dff at fixed model size we also simultaneously varied the dmodel parameter, as required by the parameter counts in Table 1. Independence of nlayers would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5. 3.2 Performance with Non-Embedding Parameter Count N In Figure 6 we display the performance of a wide variety of models, ranging from small models with shape (nlayer, d model) = (2, 128) through billion-parameter models, ranging in shape from (6, 4288) through (207, 768).

Here we have trained to near convergence on the full WebText2 dataset and observe no over-fitting (except possibly for the very largest models). As shown in Figure 1, we find a steady trend with non-embedding parameter count N, which can be fit to the first term of Equation (1.5), so that L(N) ≈ (Nc N)αN (3.1) 8LSTM plateaus after <100 tokens > Transformer improves through the whole context > 2M > 200M > 3M > 300M > 5 > 4 > 3 > 2 > 6 > Token Index in Context > 10 310 210 1 > Transformers asymptotically outperform LSTMs > due to improved use of long contexts > 3.6 > 4.2 > 3.0 > 2.4 > 4.8 > 5.4 > 10 510 810 610 710 9 > Parameters (non-embedding) > Transformers > LSTMs > 1 Layer > 2 Layers > 4 Layers > Test Loss Per-token > Test Loss > Parameters: > 400K > 400K Figure 7 To observe these trends it is crucial to study performance as a function of N; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6).

This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG +19]. Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasets is also a power-law in N with nearly identical power, as shown in Figure 8. 2.1 Comparing to LSTMs and Universal Transformers In Figure 7 we compare LSTM and Transformer performance as a function of non-embedding parameter count N. The LSTMs were trained with the same dataset and context length. We see from these figures that the LSTMs perform as well as Transformers for tokens appearing early in the context, but cannot match the Transformer performance for later tokens.

We present power-law relationships between performance and context position Appendix 5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns. We also compare the performance of standard Transformers to recurrent Transformers [DGV +18] in Figure 17 in the appendix. These models re-use parameters, and so perform slightly better as a function of N, at the cost of additional compute per-parameter. 2.2 Generalization Among Data Distributions We have also tested our models on a set of additional text data distributions. The test loss on these datasets as a function of model size is shown in Figure 8; in all cases the models were trained only on the WebText2 dataset.

We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2. We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence. We also observe no dependence on model depth (see Appendix 8). 3.3 Performance with Dataset Size and Compute We display empirical trends for the test loss as a function of dataset size D (in tokens) and training compute C in Figure 1. For the trend with D we trained a model

with (nlayer, n embd) = (36, 1280) on fixed subsets of the WebText2 dataset. We stopped training once the test loss ceased to decrease. We see that the resulting test losses can be fit with simple power-law L(D) ≈ (Dc D)αD (3.2) in the dataset size. The data and fit appear in Figure 1. The total amount of non-embedding compute used during training can be estimated as C = 6 N BS, where B is the batch size, S is the number of parameter updates, and the factor of 6 accounts for the forward and backward passes.

Thus for a given value of C we can scan over all models with various N to find the model 910 4 10 5 10 6 10 7 10 8 10 9 > Parameters (non-embedding) 34567Test Loss > WebText2 (Test) Internet Books Books Wikipedia Common Crawl 2.5 3.0 3.5 4.0 4.5 5.0 Test Loss on Training Distribution 2.5 3.0 3.5 4.0 4.5 5.0 Loss on Other Distribution Books during training Wikipedia during training Books at convergence Wikipedia at convergence Figure 8 Left: Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution.

Right: Gener-alization performance depends only on training distribution performance, and not on the phase of training. We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains. with the best performance on step S = C > 6BS. Note that in these results the batch size B remains fixed for all models, which means that these empirical results are not truly optimal. We will account for this in later sections using an adjusted Cmin to produce cleaner trends. The result appears as the heavy black line on the left-hand plot in Figure 1.

It can be fit with L(C) ≈ (Cc C)αC (3.3) The figure also includes images of individual learning curves to clarify when individual models are optimal. We will study the optimal allocation of compute more closely later on. The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix. ## 4 Charting the Infinite Data Limit and Overfitting In Section 3 we found a number of basic scaling laws for language modeling performance. Here we will study the performance of a model of size N trained on a dataset with D tokens while varying N and D simultaneously.

We will empirically demonstrate that the optimally trained test loss accords with the scaling law of Equation (1.5). This provides guidance on how much data we would need to train models of increasing size while keeping overfitting under control. 4.1 Proposed L(N, D) Equation We have chosen the parameterization (1.5) (repeated here for convenience): L(N, D) = [(Nc N) αNαD + Dc D]αD (4.1) using three principles: 1. Changes in vocabulary size or tokenization are expected to rescale the loss by an overall factor. The parameterization of L(N, D) (and all models of the loss) must naturally allow for such a rescaling.

2. Fixing D and sending N → ∞, the overall loss should approach L(D). Conversely, fixing N and sending D → ∞ the loss must approach 3. L(N, D) should be analytic at D = ∞, so that it has a series expansion in 1/D with integer powers. Theoretical support for this principle is significantly weaker than for the first two. Our choice of L(N, D) satisfies the first requirement because we can rescale Nc, D c with changes in the vocabulary. This also implies that the values of Nc, D c have no fundamental meaning.

10 10 6 10 7 10 8 10 9 > Params (non-embed) 2.5 3.0 3.5 4.0 4.5 Test Loss > Data Size Bottleneck > Data Size > 21M 43M 86M 172M 344M 688M 1.4B 22.0B 10 410 310 210 1 > NN/D/D > 0.0 0.1 0.2 0.3 0.4 0.5 > L/L(D =) 1 > Overfitting > Data Size > 21M 43M 86M 172M 344M 688M 1.4B 22.0B Figure 9 The early-stopped test loss L(N, D) depends predictably on the dataset size D and model size N according to Equation (1.5). Left: For large D, performance is a straight power law in N.

For a smaller fixed D, performance stops improving as N increases and the model begins to overfit. (The reverse is also true, see Figure 4.) Right: The extent of overfitting depends predominantly on the ratio N αNαD /D, as predicted in equation (4.3). The line is our fit to that equation. Since we stop training early when the test loss ceases to improve and optimize all models in the same way, we expect that larger models should always perform better than smaller models. But with fixed finite D, we also do not expect any model to be capable of approaching the best possible loss (ie the entropy of text).

Similarly, a model with fixed size will be capacity-limited. These considerations motivate our second principle. Note that knowledge of L(N) at infinite D and L(D) at infinite N fully determines all the parameters in L(N, The third principle is more speculative. There is a simple and general reason one might expect overfitting to scale ∝ 1/D at very large D. Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as 1/D. This expectation should hold for any smooth loss function, since we expect to be able to expand the loss about the D → ∞ limit.

However, this argument assumes that 1/D corrections dominate over other sources of variance, such as the finite batch size and other limits on the efficacy of optimization. Without empirical confirmation, we would not be very confident of its applicability. Our third principle explains the asymmetry between the roles of N and D in Equation (1.5). Very similar symmetric expressions 4 are possible, but they would not have a 1/D expansion with integer powers, and would require the introduction of an additional parameter. In any case, we will see that our equation for L(N, D) fits the data well, which is the most important justifi-cation for our L(N, D) ansatz.

4.2 Results We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing. The results are displayed in Figure 9, including a fit to the four parameters αN, α D, N c, D c in Equation (1.5): Parameter αN αD Nc Dc Value 0.076 0.103 6.4 × 10 13 1.8 × 10 13 Table 2 Fits to L(N, D) We obtain an excellent fit, with the exception of the runs where the dataset has been reduced by a factor of 1024, to about 2 × 10 7 tokens.

With such a small dataset, an epoch consists of only 40 parameter updates. Perhaps such a tiny dataset represents a different regime for language modeling, as overfitting happens very early in training (see Figure 16). Also note that the parameters differ very slightly from those obtained in Section 3, as here we are fitting the full L(N, D) rather than just L(N, ∞) or L(∞, To chart the borderlands of the infinite data limit, we can directly study the extent of overfitting. For all but the largest models, we see no sign of overfitting when training with the full 22B token WebText2 dataset, so we can take it as representative of D = ∞.

Thus we can compare finite D to the infinite data limit by > 4 For example, one might have used L(N, D) = [(Nc > N)αN + (Dc > D)αD]β, but this does not have a 1/D expansion. 11 10 1 3 × 10 0 > 4 × 10 0 > 6 × 10 0 > WebText2 Train Loss 10 3 > 10 4 > 10 5 > 10 6 > Critical Batch Size (Tokens) Critical Batch Size vs. Performance > Empirical Bcrit,N=3M > Empirical Bcrit,N=85M > Bcrit = 2.1 × 10 8tokens L4.8 > Noise Scale Measurement Figure 10 The critical batch size Bcrit follows a power law in the loss as performance increase, and does not depend directly on the model size.

We find that the critical batch size approximately doubles for every 13% decrease in loss. Bcrit is measured empirically from the data shown in Figure 18, but it is also roughly predicted by the gradient noise scale, as in [MKAT18]. defining δL (N, D) ≡ L(N, D) L(N, ∞) − 1 (4.2) and studying it as a function of N, D. In fact, we see empirically that δL depends only a specific combination of N and D, as shown in Figure 16. This follows from the scaling law of Equation (1.5), which implies δL ≈ (1 + (NNc) αNαD Dc D)αD − 1 (4.3) Note that at large D this formula also has a series expansion in powers of

We estimate that the variation in the loss with different random seeds is roughly 0.02, which means that to avoid overfitting when training to within that threshold of convergence we require D & (5 × 10 3) N 0.74 (4.4) With this relation, models smaller than 10 9 parameters can be trained with minimal overfitting on the 22B token WebText2 dataset, but our largest models will encounter some mild overfitting. More generally, this relation shows that dataset size may grow sub-linearly in model size while avoiding overfitting. Note however that this does not typically represent maximally compute-efficient training.

We should also emphasize that we have not optimized regularization (eg the dropout probability) while varying dataset and model size. ## 5 Scaling Laws with Model Size and Training Time In this section we will demonstrate that a simple scaling law provides a good description for the loss as a function of model size N and training time. First we will explain how to use the results of [MKAT18] to define a universal training step Smin, which accounts for the fact that most of our models have not been trained at an optimal batch size.

Then we will demonstrate that we can fit the model size and training time dependence of the loss using Equation (1.6). Later we will use these results to predict the optimal allocation of training compute between model size and training time, and then confirm that prediction. 5.1 Adjustment for Training at Bcrit (L) A simple empirical theory for the batch size dependence of training was developed in [MKAT18] (see also [SLA +18, ZLN +19]). It was argued that there is a critical batch size Bcrit for training; for B up to Bcrit the batch size can be increased with very minimal degradation in compute-efficiency, whereas for B > B crit increases in B result in diminishing returns.

It was also argued that the gradient noise scale provides a simple 12 prediction for Bcrit, and that neither depends directly on model size except through the value of the loss that has been attained. These results can be used to predict how training time and compute will vary with the batch size. To utilize both training time and compute as effectively as possible, it is best to train with a batch size B ≈ Bcrit. Training at B  Bcrit minimizes the number of training steps, while B  Bcrit minimizes the use of compute.

More specifically, it was demonstrated that for a wide variety of neural network tasks, the number of training steps S and the number of data examples processed E = BS satisfy the simple relation (SSmin − 1) (EEmin − 1) = 1 (5.1) when training to any fixed value of the loss L. Here Smin is the minimum number of steps necessary to reach L, while Emin is the minimum number of data examples that must be processed. We demonstrate the relation (5.1) for Transformers in Figure 18 in the appendix. This relation defines the critical batch size Bcrit (L) ≡ Emin Smin (5.2) which is a

function of the target value of the loss. Training at the critical batch size makes a roughly optimal time/compute tradeoff, requiring 2Smin training steps and processing E = 2 Emin data examples. In Figure 10 we have plotted the critical batch size and gradient noise scale 5 as a function of training loss for two different models. We see that Bcrit (L) is independent of model size, and only depends on the loss L. So the predictions of [MKAT18] continue to hold for Transformer language models. The critical batch size can be fit with a power-law in the loss Bcrit (L) ≈ B∗ L1/α B (5.3) where B∗ ≈ 2 × 10 8 and αB ≈

We have chosen this parameterization for Bcrit (L) because as the loss approaches its minimum value Lmin,the gradient noise scale is expected to diverge, and we expect Bcrit to track this noise scale. We do not know Lmin, as we see no sign that our models are approaching it, but Lmin > 0 since the entropy of natural language is non-zero. Since apparently Lmin is much smaller than the values of L we have achieved, we used a parameterization where Bcrit diverges as L → We will use Bcrit (L) to estimate the relation between the number of training steps S while training at batch size B = 2 19 tokens and the number of training steps while training at B  Bcrit.

This is simply Smin (S) ≡ S 1 + Bcrit (L)/B (minimum steps, at B  Bcrit) (5.4) for any given target value L for the loss. This also defines a critical value of the compute needed to train to L with a model of size N if we were to train at B  Bcrit (L). This is Cmin (C) ≡ C 1 + B/B crit (L) (minimum compute, at B  Bcrit) (5.5) where C = 6 N BS estimates the (non-embedding) compute used at batch size B. 5.2 Results for L(N, S min) and Performance with Model Size and Compute Now we will use Smin defined in Equation (5.4) to obtain a simple and universal fit for the dependence of the loss on model size and training time in the infinite data limit.

We will fit the stable, Adam-optimized training runs using Equation (1.6), repeated here for convenience: L(N, S min) = (Nc N)αN + (Sc Smin)αS (5.6) for the loss. We include all training steps after the warmup period of the learning rate schedule, and find a fit to the data with the parameters: > 5Although the critical batch size roughly matches the gradient noise scale, we are using a direct measurements of > Bcrit from Figures 18 and 10 for all our later analyses. 13 10 4 10 6 10 8 > Parameters (non-embedding) 2345678Test Loss > Performance vs Compute Budget > 10 5 > 10 4 > 10 3 > 10 2 > 10 1 > 10 0 > PF-dayss 10 610 710 810 9 > Parameters (non-embedding) 2.4 3.0 3.6 4.2 4.8 5.4 Test Loss > Performance vs Steps > 10 4 > 10 5 > Steps Figure 11 When we hold either total compute or number of training steps fixed, performance follows L(N, S) from Equation (5.6).

Each value of compute budget has an associated optimal model size that maximizes performance. Mediocre fits at small S are unsurprising, as the power-law equation for the learning curves breaks down very early in training. Parameter αN αS Nc Sc Value 0.077 0.76 6.5 × 10 13 2.1 × 10 3 Table 3 Fits to L(N, S) With these parameters, we obtain the learning curve fits in Figure 4. Though the fits are imperfect, we believe they are quite compelling given the simplicity of Equation (5.6). The data and fits can be visualized in a different and more interesting way, as shown in Figure 11.

There we study the test loss as a function of model size while fixing either the total non-embedding compute C used in training, or the number of steps S. For the fits we use Equation (5.5) and (5.4) along with the parameters above and Equation (5.6). The power-law dependence of the loss on Smin reflects the interplay of optimizer dynamics and the loss landscape. Since the fits are best late in training, when the loss may be approximately quadratic, the power-law should provide information about the spectrum of the Hessian of the loss. Its universality suggests that the Hessian eigenvalue density is roughly independent of model size.

5.3 Lower Bound on Early Stopping Step The results for L(N, S min) can be used to derive a lower-bound (and rough estimate) of the step at which early stopping should occur when training is data limited. It is motivated by the idea that finite and infinite D learning curves for a given model will be very similar until we reach Smin ≈ Sstop. Thus overfitting should be proportional to the correction from simply ending training at Sstop. This will underestimate Sstop, because in reality the test loss will decrease more slowly when we have a finite D, and therefore we will require more training steps to reach the optimal test loss at finite D.

This line of reasoning leads to the inequality Sstop (N, D) & Sc [L(N, D) − L(N, ∞)] 1/α S (5.7) where L(N, ∞) is the converged loss, evaluated with infinite available data. This inequality and its com-parison to the empirical data is displayed in Figure 16 in the appendix. In that figure, the values of Sstop and L(N, D) are empirical (though Sstop is adjusted to mimic training at B  Bcrit), while L(N, ∞) is computed from the fit to L(N, D) evaluated at D = ∞. ## 6 Optimal Allocation of the Compute Budget We displayed the empirical trend of performance as a function of the computation used during training in the top-right of Figure 1.

However, this result involved training at a fixed batch size B, whereas we know 14 Models between 0.6x and 2.2x the > optimal size can be trained with a > 20% larger compute budget > Smaller models require > more steps to train, while > larger models require fewer > Our framework does not > capture early training dynamics Figure 12 Left: Given a fixed compute budget, a particular model size is optimal, though somewhat larger or smaller models can be trained with minimal additional compute. Right: Models larger than the compute-efficient size require fewer steps to train, allowing for potentially faster training if sufficient additional paral-lelism is possible.

Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve, after initial transient effects. 10 8 10 6 10 4 10 2 10 0 > Compute (PF-days), non-embedding 234567Test Loss > L=(Cmin /2.3 10 8)0.050 > L= (C/2.0 10 7)0.057 Figure 13 When adjusting performance to simulate training far below the critical batch size, we find a somewhat altered power law for L(Cmin) when compared with the fully empirical results. The conspicuous lump at 10 −5 PF-days marks the transition from 1-layer to 2-layer networks; we exclude 1-layer networks in the power-law fits.

It is the L(Cmin) trend that we expect to provide a reliable extrapolation for larger compute. that in fact we could train more efficiently 6 by training at the batch size Bcrit discussed in Section 5.1. Large and small values of the loss could have been achieved with fewer samples or fewer steps, respectively, and correcting for this inefficiency by standardizing to the critical batch size results in cleaner and more predictable trends. In this section we will adjust for this oversight. More importantly, we will use the results of Section 5 to determine the optimal allocation of compute between model size N and the quantity of data processed during training, namely 2Bcrit Smin.

We will determine this allocation both empirically and theoretically, by using the equation for L(N, S min), and we will demonstrate that these methods agree. 6.1 Optimal Performance and Allocations Let us first study the loss as a function of the optimally allocated compute from Equation (5.5). The result is plotted in Figure 13, along with a power-law fit. We see that as compared to the compute plot of Figure 1, the new fit with Cmin is somewhat improved. Given L(Cmin), it is natural to ask for the optimal model size N (Cmin) that provides the minimal loss with a given quantity of training compute.

The optimal model size is shown in Figure 14. We observe that N (Cmin) > 6 One might ask why we did not simply train at Bcrit in the first place. The reason is that it depends not only on the model but also on the target value of the loss we wish to achieve, and so is a moving target. 15 10 7 10 5 10 3 10 1 > Compute (PF-days), non-embedding 10 3 > 10 5 > 10 7 > Parameters (non-embedding) > N= (1.3 10 9)C0.73 min > N= (1.6 10 9)C0.88 10 710 510 310 1 > Compute (PF-days), excluding embeddings 05000 10000 15000 Steps > Smin (adjusted) > Smin = (5.4 10 3)C0.03 min > S(fixed-batch) Figure 14 Left: Each value of the compute budget Cmin has an associated optimal model size N.

Optimal model size grows very rapidly with Cmin, increasing by 5x for each 10x increase in compute. The number of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x. Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes. can be fit very well with a power-law N (Cmin) ∝ (Cmin)0.73. (6.1) In Figure 12, we show the effect of training models of sub-optimal sizes (see Appendix

4). By definition Cmin ≡ 6N B crit S, and so we can use N (Cmin) to extract further results. In particular, since prior fits show B ∝ L−4.8 and L ∝ C−0.05 min, we can conclude that Bcrit ∝ C0.24 min. This leads us to conclude that the optimal number of steps will only grow very slowly with compute, as Smin ∝ (Cmin)0.03, (6.2) matching the empirical results in Figure 14. In fact the measured exponent is sufficiently small that our results may even be consistent with an exponent of zero. Thus we conclude that as we scale up language modeling with an optimal allocation of computation, we should predominantly increase the model size N, while simultaneously scaling up the batch size via B ∝ Bcrit with negligible increase in the number of serial steps.

Since compute-efficient training uses relatively few optimization steps, additional work on speeding up early training dynamics may be warranted. 6.2 Predictions from L(N, S min) The results for L(Cmin) and the allocations can be predicted from the L(N, S min) equation obtained in Section 5. Given our equation for L(N, S min), we can substitute Smin = Cmin > 6N B and then find the minimum of the loss as a function of N, while fixing the training compute. We carry out this procedure in detail in Appendix B, where we also provide some additional predictions.

For the loss as a function of training compute, we predict that L(Cmin) = (Cmin > c Cmin)αmin > C (6.3) where αmin > C ≡ 11/α S + 1 /α B + 1 /α N ≈ 0.054 (6.4) in excellent agreement with the exponent of Figure 13. We also predict that N (Cmin) ∝ (Cmin)αmin > C/α N ≈ (Cmin)0.71 (6.5) which also matches the scaling of Figure 14 to within a few percent. Our scaling laws provide a predictive framework for the performance of language modeling. 16 The intersection point is sensitive to > the precise power-law parameters Figure 15 Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(Cmin) and L(D)

due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits. 6.3 Contradictions and a Conjecture We observe no signs of deviation from straight power-law trends at large values of compute, data, or model size. Our trends must eventually level off, though, since natural language has non-zero entropy. Indeed, the trends for compute-efficient training described in this section already contain an apparent contra-diction. At scales several orders of magnitude above those documented here, the performance predicted by the L(Cmin) scaling law decreases below what should be possible given the slow growth in training data with compute.

This implies that our scaling laws must break down before this point, but we conjecture that the intersection point has a deeper meaning: it provides an estimate of the point at which Transformer language models reach maximal performance. Since the amount of data used by compute-efficient training grows slowly with the compute budget, the performance predicted by L(Cmin) eventually hits a lower bound set by the L(D) power law (see Figure 15). Let us work this out in more detail. To keep overfitting under control, the results of Section 4 imply that we should scale the dataset size as D ∝ N 0.74 ∝ C0.54 min (6.6) where we have used the compute-efficient N (Cmin) from Figure 14.

Let us compare this to the data requirements of compute-efficient training. If we train at the critical batch size e. C = 2 Cmin) and never re-use data during training, we find that data usage grows with compute as D(Cmin) = 2Cmin 6N (Cmin) ≈ (4 × 10 10 tokens) (Cmin /PF -Day) 0.26 (6.7) This is the maximum rate at which the dataset size can productively grow with compute, since it means that we are only training for a single epoch. But it grows the dataset much more slowly than in Equation (6.6). It appears to imply that compute-efficient training will eventually run into a problem with overfitting, even if the training process never re-uses any data!

According to Figure 1, we expect that when we are bottlenecked by the dataset size (ie by overfitting), the loss should scale as L(D) ∝ D−0.095. This implies that the loss would scale with compute as L(D(Cmin)) ∝ C−0.03 min once we are data-limited. Once again, we have a contradiction, as this will eventually intersect with our prediction for L(Cmin) from Figure 13, where we found a scaling L(Cmin) ∝ C−0.050 The intersection point of L(D(Cmin)) and L(Cmin) occurs at C∗ ∼ 10 4 PF -Days N ∗ ∼ 10 12 parameters, D∗ ∼ 10 12 tokens, L∗ ∼ 1.7 nats/token (6.8) though the numerical values are highly uncertain, varying by an order or magnitude in either direction de-pending on the precise values of the exponents from the power-law fits.

The most obvious interpretation is that our scaling laws break down at or before we reach this point, which is still many orders of magnitude away in both compute and model size. 17 One might also conjecture that this intersection point has a deeper meaning. If we