Evaluating Large Language Models Trained on Code

# Evaluating Large Language Models Trained on Code Mark Chen * 1 Jerry Tworek * 1 Heewoo Jun * 1 Qiming Yuan * 1 Henrique Ponde de Oliveira Pinto * 1 Jared Kaplan * 2 Harri Edwards 1 Yuri Burda 1 Nicholas Joseph 2 Greg Brockman 1 Alex Ray 1 Raul Puri 1 Gretchen Krueger 1 Michael Petrov 1 Heidy Khlaaf 3 Girish Sastry 1 Pamela Mishkin 1 Brooke Chan 1 Scott Gray 1 Nick Ryder 1 Mikhail Pavlov 1 Alethea Power 1 Lukasz Kaiser 1 Mohammad Bavarian 1 Clemens Winter 1 Philippe Tillet 1 Felipe Petroski Such 1 Dave Cummings 1 Matthias Plappert 1 Fotios Chantzis 1 Elizabeth Barnes 1 Ariel Herbert-Voss 1 William Hebgen Guss 1 Alex Nichol 1 Alex Paino 1 Nikolas Tezak 1 Jie Tang 1 Igor Babuschkin 1 Suchir Balaji 1 Shantanu Jain 1 William Saunders 1 Christopher Hesse 1 Andrew N.

Carr 1 Jan Leike 1 Josh Achiam 1 Vedant Misra 1 Evan Morikawa 1 Alec Radford 1 Matthew Knight 1 Miles Brundage 1 Mira Murati 1 Katie Mayer 1 Peter Welinder 1 Bob McGrew 1 Dario Amodei 2 Sam McCandlish 2 Ilya Sutskever 1 Wojciech Zaremba 1 ## Abstract We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evalua-tion set we release to measure functional correct-ness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%.

Fur-thermore, we find that repeated sampling from the model is a surprisingly effective strategy for pro-ducing working solutions to difficult prompts. Us-ing this method, we solve 70.2% of our problems with 100 samples per problem. Careful investiga-tion of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to vari-ables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and eco-nomics. > * Equal contribution > 1 OpenAI, San Francisco, California, USA. > 2 Anthropic AI, San Francisco, California, USA.

Work per-formed while at OpenAI. > 3 Zipline, South San Francisco, California, USA. Work per-formed while at OpenAI. Correspondence to: Mark Chen com >,Jerry Tworek com >, Heewoo Jun com >, Qiming Yuan com >. ## 1. Introduction Scalable sequence prediction models (Graves, 2014; Vaswani et, 2017; Child et, 2019) have become a general-purpose method for generation and representation learning in many domains, including natural language pro-cessing (Mikolov et, 2013; Sutskever et, 2014; Dai & Le, 2015; Peters et, 2018; Radford et, 2018; Devlin et, 2018), computer vision (Van Oord et

, 2016; Menick & Kalchbrenner, 2018; Chen et, 2020; Bao et, 2021), audio and speech processing (Oord et, 2016; 2018; Dhari-wal et, 2020; Baevski et, 2020), biology (Alley et, 2019; Rives et, 2021), and even across multiple modali-ties (Das et, 2017; Lu et, 2019; Ramesh et, 2021; Zellers et, 2021). More recently, language models have also fueled progress towards the longstanding challenge of program synthesis (Simon, 1963; Manna & Waldinger, 1971), spurred by the presence of code in large datasets (Husain et, 2019; Gao et

, 2020) and the resulting pro-gramming capabilities of language models trained on these datasets (Wang & Komatsuzaki, 2021). Popular language modeling objectives like masked language modeling (Devlin et, 2018) and span prediction (Raffel et, 2020) have also been adapted to train their programming counterparts CodeBERT (Feng et, 2020) and PyMT5 (Clement et, 2020). Similarly, our early investigation of GPT-3 (Brown et, 2020) revealed that it could generate simple programs from Python docstrings. While rudimentary, this capability was exciting because GPT-3 was not explicitly trained for code generation. Given the considerable success of large lan-guage models in other modalities and the abundance of publicly available code, we hypothesized that a specialized GPT model, called Codex, could excel at a variety of coding tasks.

This paper describes several early Codex models, whose descendants power GitHub Copilot and the Codex models in the OpenAI API. > arXiv:2107.03374v2 LG] 14 Jul 2021 Evaluating Large Language Models Trained on Code > Figure 1. Pass rates of our models on the HumanEval dataset as a function of model size. When a single sample is generated for each problem, GPT-12B solves no problems, but Codex (fine-tuned on code) solves 28.8% of the problems, and Codex-S (further fine-tuned on correctly implemented standalone functions) solves 37.7% of the problems. From here, further gains can be realized by generating 100 samples per problem and selecting the sample with the highest mean log-probability (44.5% solved) or by selecting the sample that passes the unit tests (77.5% solved).

All samples are generated with temperature 0.8. In this work, we focus on the task of generating stan-dalone Python functions from docstrings, and evaluate the correctness of code samples automatically through unit tests. This is in contrast to natural language generation, where samples are typically evaluated by heuristics or by human evaluators. To accurately benchmark our model, we create a dataset of 164 original programming problems with unit tests. These problems assess language compre-hension, algorithms, and simple mathematics, with some comparable to simple software interview questions. We release this data along with an evaluation framework at

com/openai/human-eval. To solve a problem in our test set, we generate multiple samples from the models, and check if any of them pass the unit tests. With just a single sample, a 12B parameter Codex solves 28.8% of these problems, and a 300M parameter Codex solves 13.2% of these problems. In contrast, the 6B parameter GPT-J (Wang & Komatsuzaki, 2021) achieves 11.4% on the same dataset, while all GPT models achieve near 0%. To improve our model’s performance at the task of function synthesis from docstrings, we fine-tune Codex on standalone, correctly implemented functions. The resulting model, Codex-S, solves 37.7% of problems with a single sample.

Figure 2 showcases problems of varying difficulty in our dataset, along with correct model generated solutions. Real-world programming tasks often involve iterations of approaches and bug fixes, which is approximated by gener-ating many samples from our models and selecting one that passes all unit tests. Within 100 samples, Codex-S is able to generate at least one correct function for 77.5% of the prob-lems. This result suggests that accurate code samples can be selected via heuristic ranking instead of fully evaluating each sample, the latter of which may not be possible or prac-tical in deployment. Indeed, we find that the sample with highest mean log-probability passes unit tests for 44.5% of the problems.

We conclude by discussing the limitations and potential broader impacts of these Codex models and of increasingly powerful code generating models more generally. ## 2. Evaluation Framework In this section, we discuss the details of our evaluation framework. We begin by defining the pass@ k metric, and explain its advantages over standard match-based metrics. Next, we describe the dataset of hand-written problems, called “HumanEval,” which we created in order to bench-mark our models. Finally, we discuss the sandbox environ-ment we used to safely execute model-generated code. 2.1. Functional Correctness Generative models for code are predominantly benchmarked by matching samples against a reference solution, where the match can be exact or fuzzy (as in BLEU score).

How-ever, recent work has surfaced deficiencies in match-based metrics for code. For instance, Ren et al. (2020) finds that BLEU has problems capturing semantic features specific to code, and suggests several semantic modifications to the score. More fundamentally, match-based metrics are unable to ac-count for the large and complex space of programs function-ally equivalent to a reference solution. As a consequence, recent works in unsupervised code translation (Lachaux et, 2020) and pseudocode-to-code translation (Kulal et, 2019) have turned to functional correctness instead, where a sample is considered correct if it passes a set of unit tests.

We argue that this metric should be applied to docstring-conditional code generation as well. Perhaps the most convincing reason to evaluate functional correctness is that it is used by human developers to judge code. A framework known as test-driven development dic-tates that software requirements be converted into test cases before any implementation begins, and success is defined by a program that passes these tests. While few organiza-tions employ full test-driven development, integration of new code is usually dependent on creating and passing unit tests. Kulal et al. (2019) evaluate functional correctness using the pass@ k metric, where k code samples are generated per problem, a problem is considered solved if any sample Evaluating Large Language Models Trained on Code Figure 2.

Three example problems from the HumanEval dataset, where the probabilities that a single sample from Codex-12B passes unit tests are 0.9, 0.17, and 0.005. The prompt provided to the model is shown with a white background, and a successful model-generated completion is shown in a yellow background. Though not a guarantee for problem novelty, all problems were hand-written and not programmatically copied from existing sources. Random problems and samples can be found in Appendix B. passes the unit tests, and the total fraction of problems solved is reported. However, computing pass@ k in this way can have high variance.

Instead, to evaluate pass@ k,we generate n ≥ k samples per task (in this paper, we use n = 200 and k ≤ 100), count the number of correct samples c ≤ n which pass unit tests, and calculate the unbiased estimator pass@ k:= E > Problems [1 − (n−ck)(nk)] (1) Calculating this estimator directly results in very large num-bers and numerical instability. In Figure 3, we include a numerically stable numpy implementation that simplifies the expression and evaluates the product term-by-term. One may be tempted to estimate pass@ k with 1−(1 − ˆp)k where ˆp is the empirical estimate of pass@1, but we show that it is biased in Appendix A.

def pass_at_k(n, c, k): """:param n: total number of samples:param c: number of correct samples:param k: k in pass@$k$ """ if n - c < k: return 1.0 return 1.0 - prod(1.0 - k arange(n - c + 1, n + 1)) Figure 3. A numerically stable script for calculating an unbiased estimate of pass@ k. Later, we provide evidence that BLEU score may not be a reliable indicator of functional correctness by showing that functionally inequivalent programs generated by our model (which are guaranteed to disagree with the reference solution on some input) often have higher BLEU scores than functionally equivalent ones.

Evaluating Large Language Models Trained on Code 2.2. HumanEval: Hand-Written Evaluation Set We evaluate functional correctness on a set of 164 hand-written programming problems, which we call the Hu-manEval dataset. Each problem includes a function sig-nature, docstring, body, and several unit tests, with an av-erage of 7.7 tests per problem. It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources. For example, there are more than ten public repositories containing solutions to Codeforces problems, which make up part of the recently proposed APPS dataset (Hendrycks et

, 2021). Programming tasks in the HumanEval dataset assess lan-guage comprehension, reasoning, algorithms, and simple mathematics. We release the HumanEval dataset so that others can evaluate functional correctness and measure the problem-solving capabilities of their models. The dataset can be found at com/openai/human-eval. 2.3. Sandbox for Executing Generated Programs Since publicly available programs have unknown intent and generated programs are often incorrect, executing these programs poses a security risk. Indeed, GitHub is known to contain malicious programs that alter or change their environments (Rokon et, 2020). Therefore, we developed a sandbox environment to safely run untrusted programs against unit tests.

Our goals were to prevent these programs from modifying, gaining persistence on, accessing sensitive resources on, or exfiltrating data from a host or network. Since OpenAI’s training infrastructure is built on Kubernetes and cloud services, we designed our sandbox to address the limitations of these environments while remaining idiomatic with their patterns of use. We selected the gVisor container runtime (Lacasse, 2018) as the main host protection component. Since container runtimes like Docker can share host resources with contain-ers, a malicious container could potentially compromise a host. gVisor protects the host by emulating its resources to introduce a security boundary between the host and its con-tainers.

Network-adjacent hosts and services are protected by eBPF-based firewall rules that prevent inbound and out-bound connections except for those required for experiment control. ## 3. Code Fine-Tuning We fine-tune GPT models containing up to 12B parameters on code to produce Codex. In contrast with GPT, Codex displays non-trivial performance on the HumanEval dataset. In fact, Codex is able to solve the majority of the problems in HumanEval if we generate and evaluate 100 samples per problem, and pick one that passes unit tests. When limited to a budget of one evaluation per problem, producing multiple samples with Codex and choosing the one with the highest mean log-probability provides significant gains.

3.1. Data Collection Our training dataset was collected in May 2020 from 54 mil-lion public software repositories hosted on GitHub, contain-ing 179 GB of unique Python files under 1 MB. We filtered out files which were likely auto-generated, had average line length greater than 100, had maximum line length greater than 1000, or contained a small percentage of alphanumeric characters. After filtering, our final dataset totaled 159 GB. 3.2. Methods Since Codex is evaluated on natural language prompts, we hypothesized that it would be beneficial to fine-tune from the GPT-3 (Brown et, 2020) model family, which already contains strong natural language representations.

Surpris-ingly, we did not observe improvements when starting from a pre-trained language model, possibly because the fine-tuning dataset is so large. Nevertheless, models fine-tuned from GPT converge more quickly, so we apply this strategy for all subsequent experiments. We train Codex using the same learning rate as the corre-sponding GPT model, with a 175 step linear warmup and cosine learning rate decay. We train for a total of 100 billion tokens, using the Adam optimizer with β1 = 0.9, β2 = 0.95,  = 10 −8, and a weight decay coefficient of In order to maximally leverage text representations from GPT, we base our code lexer on the GPT-3 text tokenizer.

Since the distribution of words in GitHub code differs from that of natural text, this tokenizer is not very effective for representing code. The largest source of inefficiency arises from encoding whitespace, so we add an additional set of tokens for representing whitespace runs of different lengths. This allows us to represent code using approximately 30% fewer tokens. To compute pass@ k, we assemble each HumanEval prob-lem into a prompt consisting of a header, a signature, and a docstring, which is illustrated in Figure 2. We sample tokens from Codex until we encounter one of the following stop sequences: ‘ \nclass ’, ‘ \ndef ’, ‘ \n# ’, ‘ \nif ’, or ‘\nprint ’, since the model will continue generating addi-tional functions or statements otherwise.

We use nucleus sampling (Holtzman et, 2020) with top p = 0.95 for all sampling evaluation in this work. 3.3. Results In Figure 4, we plot test loss on a held-out validation set against Codex model size. We find that just as language Evaluating Large Language Models Trained on Code > Figure 4. Model cross-entropy test loss measured on a held-out split of our Python GitHub code corpus. The smooth power law scaling of performance with model size observed in GPT-3 appears to hold even after code fine-tuning. model test loss follows a power law in model size (Kaplan et

, 2020), test loss after code fine-tuning follows a similar power law with functional form (N > 5.92 ×10 7)−0.13 where N is the number of non-embedding parameters in the model. When evaluating pass@ k, it is important to optimize sam-pling temperature for the particular value of k. In Figure 5, we plot pass@ k against the number of samples k and the sampling temperature. We find that higher temperatures are optimal for larger k, because the resulting set of samples has higher diversity, and the metric rewards only whether the model generates any correct solution.

In particular, for a 679M parameter model, the optimal tem-perature for pass@1 is T ∗ = 0.2 and the optimal tempera-ture for pass@100 is T ∗ = 0.8. With these temperatures, we find that pass@1 and pass@100 scale smoothly as a function of model size (Figure 6). Pass@ k can also be interpreted as the result of evaluating the best out of k samples, where the best sample is picked by an oracle with prior knowledge of the unit tests. From a practical perspective, we are also interested in the set-ting where we must select a single sample from k samples without having access to an oracle.

For instance, when the model is used as an autocomplete tool where a user provides a prompt, we do not have unit tests, but would like to return only a single completion to the user for evaluation so as to not overwhelm them. Inspired by similar work in language modeling, we find that choosing the sample with the highest mean token log probability outperforms evaluating a random sample, while choosing the sample based on sum log probability can per-form slightly worse than picking randomly. Figure 7 demon-strates the benefits of applying these heuristics to samples (at temperature 0.8) from Codex-12B.

> Figure 5. In the top panel, we plot pass@ kagainst the number of samples (k) for various temperature settings. Higher temperatures are better when the number of samples is large, likely due to the increased sample diversity. In the bottom panel, we plot the best temperature setting for each k, obtained by taking the upper hull of the top panel. > Figure 6. Using the optimal temperatures 0.2 and 0.8 for pass@1 and pass@100, we plot these two metrics as a function of model size. Performance appears to scale smoothly as a sigmoid in log-parameters.

Evaluating Large Language Models Trained on Code > Figure 7. Model performance in the setting where we can generate multiple samples, but only evaluate one. We can do better than ran-domly selecting a sample by choosing the solution with the highest mean log-probability (red) or with the highest back-translation score (orange) described in Sec. 5. The blue line represents the theoretical best performance obtained using an oracle with prior knowledge of the unit tests. Finally, we compute BLEU scores for all Codex-12B Hu-manEval samples (at temperature 0.8) against their reference solutions. For each problem, when we plot the distributions of BLEU scores for correct and incorrect solutions, we notice significant overlap (Figure 8).

Since an incorrect solution is guaranteed to be functionally inequivalent to the reference solution, we conclude that improvements in BLEU score may not indicate improved rates of functional correctness in practice. 3.4. Comparative Analysis of Related Models and Systems Two recent works similar in spirit to Codex are GPT-Neo (Black et, 2021) and GPT-J (Wang & Komatsuzaki, 2021), which are trained on The Pile (Gao et, 2020), a dataset containing text from a variety of sources as well as 8% GitHub code. The broader research community has found that these models outperform existing GPT systems in qualitative programming evaluations (Woolf, 2021).

We confirm these findings using the HumanEval dataset, showing that GPT-Neo achieves 6.4% pass@1 and 21.3% pass@100, while GPT models of comparable sizes achieve near 0% on both metrics. We see a remarkable progression in capabilities, with GPT-Neo-2.7B roughly equivalent to Codex-85M (30 × fewer parameters). Similarly, GPT-J-6B achieves 11.6% pass@1 and 27.7% pass@100, which is roughly equivalent to Codex-300M (20 × fewer parameters). Pass rates are obtained by taking the best result from eval- > Figure 8. BLEU score probability densities for correct (blue) and wrong (green) solutions from Codex-12B for 4 random tasks from HumanEval.

Note that the distributions are not cleanly separable, suggesting that optimizing for BLEU score is not equivalent to optimizing for functional correctness. uating at temperatures 0.2, 0.4, and 0.8 for GPT-Neo, and from temperatures 0.2 and 0.8 for GPT-J. Detailed results across multiple model sizes can be found in Table 1. Finally, we benchmark Codex against the largest free model from Tabnine, a leading code autocomplete system, which achieves 2.6% pass@1 (at T = 0.4) and 7.6% pass@100 (at T = 0.8). This is roughly equivalent to Codex-12M, one of the smallest models in our suite.

3.5. Results on the APPS Dataset Recently, Hendrycks et al. (2021) introduced the APPS dataset to measure the coding challenge competence of lan-guage models. The APPS dataset consists of 5000 training and 5000 test examples of coding problems, each with a set of unit tests and, for the training data, a set of correct solu-tions. Most of the APPS tests problems are not formulated as single-function synthesis tasks, but rather as full-program synthesis, reading input from stdin and printing output to stdout, in contrast to the main Codex training data. In the paper that introduces APPS, the authors benchmark a few language models and

report two metrics: the percentage of problems where the model finds a correct solution (called the “strict accuracy”) and the percentage of unit tests passed, even if the solution is incorrect. The latter measure is re-ported only so as to reduce variance of the measurements, because the results on the first metric were so low. We avoid this metric and only focus on “strict accuracy”, and - as in Evaluating Large Language Models Trained on Code > Table 1. Codex, GPT-Neo, & TabNine evaluations for HumanEval. We find that GPT-J pass@1 is between Codex-85M and Codex-300M performance.

> PASS @kk= 1 k= 10 k= 100 > GPT-N EO 125M 0.75% 1.88% 2.97% GPT-N EO 1.3B 4.79% 7.47% 16.30% GPT-N EO 2.7B 6.41% 11.27% 21.37% GPT-J 6B 11.62% 15.74% 27.74% TAB NINE 2.58% 4.35% 7.59% CODEX -12M 2.00% 3.62% 8.58% CODEX -25M 3.21% 7.1% 12.89% CODEX -42M 5.06% 8.8% 15.55% CODEX -85M 8.22% 12.81% 22.4% CODEX -300M 13.17% 20.37% 36.27% CODEX -679M 16.22% 25.7% 40.95% CODEX -2.5B 21.36% 35.42% 59.5% CODEX -12B 28.81% 46.81% 72.31% the previous sections - we report pass@ k numbers for vari-ous k (Table 2). There are 2 additional factors, well-known from coding competitions, that we take into account: • In coding competitions and in the APPS datasets, tasks are provided with 3 input/output examples included in the task description.

We utilize this by sampling 1000 solutions from the model and filtering out only those that pass these 3 unit tests (if such solutions exist). We then calculate pass rates in this filtered set, and call it filtered pass@ k. Results without filtering are presented as raw pass@ • It is often the case both in coding competitions and in the results from Codex that a correct solution is found, but it is not algorithmically efficient enough to be con-sidered passing. While this is not acceptable in the competitions, we also report the number of solutions that Codex produces that do not fail on any unit test, but that do time-out on some of them.

We use a timeout of 3 seconds in our evaluation. To compensate for the fact the Codex is not fine-tuned on APPS, we append a single input/output example from the task description to the docstring as a formatting hint. We de-note this setting as “1-shot” in Table 2, and find that Codex-12B evaluated 1-shot achieves comparable performance to a GPT-Neo model fine-tuned on APPS. Consistent with our earlier findings, there are large benefits from generating and evaluating as many as 1000 samples per task, though for more difficult problems, solutions are often not efficient enough to pass the time limits.

Finally, evaluating the first sample which passes the 3 public unit tests for each problem yields higher performance than raw pass@100 samples. ## 4. Supervised Fine-Tuning In addition to standalone functions, Python code found on GitHub contains class implementations, configuration files, scripts, and even files used to store data. This code is seem-ingly unrelated to synthesizing functions from docstrings, and we hypothesize that the distribution mismatch reduces HumanEval performance. In order to adapt Codex to the distribution of the task of in-terest, we construct a set of training problems from correctly implemented standalone functions, and use them for addi-tional supervised fine-tuning.

We describe two approaches for collecting these examples: from competitive program-ming websites and from repositories with continuous inte-gration. We call the supervised fine-tuned models Codex-S, and show that they produce consistent gains across model size. 4.1. Problems from Competitive Programming Programming contest and interview preparation websites use hidden unit tests to automatically judge the func-tional correctness of submissions. These problems are self-contained, come with well-written problem statements, and generally have excellent test coverage. Additionally, these problems test algorithmic reasoning over a broad range of core skills and difficulties. We collected problem statements, function signatures, and solutions from several popular programming contest and interview preparation websites.

We then assembled these into programming tasks similar to HumanEval, using the problem description as the docstring. Since complete test suites are often hidden, we created unit tests from examples found in the problem statements, or extracted additional test cases through submitting incorrect solutions. In total, we curated 10,000 problems in this way. 4.2. Problems from Continuous Integration Next, we curated programming problems from open source projects. Taking advantage of setprofile, we were able to trace and collect inputs and outputs for all functions called during integration tests. This data could then be used to create unit tests for the functions.

Projects that employ continuous integration (CI) are ideal candidates for tracing. We follow the commands in the CI configuration files, which contain build and test commands, to set up the virtual environments, install dependencies, and run integration tests. We considered GitHub repos using travis and tox as their CI frameworks, as they are two of the most popular CI tools. We additionally used publicly available source code from pip packages found in the python package index (PyPI). Evaluating Large Language Models Trained on Code > Table 2. Finetuned GPT-Neo numbers from the APPS paper referenced above.

For Codex-12B, the number of passing programs that timeout on some test is in the bracket. We used temperature 0.6 for sampling to cover all kin pass@ k, so raw pass@1 results could be improved with lower temperature. INTRODUCTORY INTERVIEW COMPETITION > GPT-N EO 2.7B RAW PASS @1 3.90% 0.57% 0.00% GPT-N EO 2.7B RAW PASS @5 5.50% 0.80% 0.00% 1-SHOT CODEX RAW PASS @1 4.14% (4.33%) 0.14% (0.30%) 0.02% (0.03%) 1-SHOT CODEX RAW PASS @5 9.65% (10.05%) 0.51% (1.02%) 0.09% (0.16%) 1-SHOT CODEX RAW PASS @100 20.20% (21.57%) 2.04% (3.99%) 1.05% (1.73%) 1-SHOT CODEX RAW PASS @1000 25.02% (27.77%) 3.70% (7.94%) 3.23% (5.85%) 1-SHOT CODEX FILTERED PASS @1 22.78% (25.10%) 2.64% (5.78%) 3.04% (5.25%) 1-SHOT CODEX FILTERED PASS @5 24.52% (27.15%) 3.23% (7.13%) 3.08% (5.53%) Because these projects contained untrusted code, it was im-portant to run integration tests in the sandboxed environment described above.

While there are millions of potential functions to curate problems from, we only collected about 40,000 because not all functions accept inputs and return outputs. Even when they do, most objects captured at runtime cannot be pickled and restored outside the sandbox unless the project was installed. Since our tracing methodology produced inputs and outputs for all invoked functions, even builtin and library calls im-ported by the project were turned into problems. For this reason, functions from tracing tended to be the building blocks of command-line utilities. To excel at these tasks, the model does not need to know advanced algorithms and data structures.

Rather, it needs to be able to follow in-structions to implement the functionality specified in the docstring. Thus, tracing complements the puzzle nature of coding competition problems and broadens the distribution of tasks. 4.3. Filtering Problems In the previous sections, we presented two methods we used to automatically create training problems. However, it is unclear how to control for quality. Some prompts underspecify the function that is implemented, in which case a perfectly valid solution may be wrongly penalized by the unit test. Some problems are stateful, and subsequent executions can result in different outcomes.

To address these issues, we use Codex-12B to generate 100 samples per curated problem. If no samples pass the unit tests, we consider the task to be either ambiguous or too difficult, and filter it out. We reran this verification several times to remove stateful or non-deterministic problems. 4.4. Methods We fine-tune Codex on these training problems to produce a set of “supervised fine-tuned” models, which we call Codex-S. To produce examples from training problems, we assem-ble the problems into the format shown in Figure 2. If there are prompts of varying length in a batch, we left-pad shorter prompts to the length of the longest prompt, so that the first tokens in the reference solutions line up in context.

We train to minimize negative log-likelihood of the reference solution, and mask out loss for any tokens in the prompt. We train using a learning rate 1/10 as large as used for fine-tuning Codex, but adhere to the same learning rate schedule, and train until validation loss plateaus (less than 10B tokens). 4.5. Results As with Codex, we first compute the optimal temperature for evaluating pass@ k for 1 ≤ k ≤ 100. We find that Codex-S prefers slightly higher temperatures for all k > 1, which possibly reflects the fact that Codex-S captures a narrower distribution than Codex.

We use T ∗ = 0 for computing pass@1 and T ∗ = 1 for computing pass@100. Next, we compare Codex-S against Codex on pass@1 and pass@100. Codex-S outperforms the corresponding Codex by an average margin of 6.5 percentage points on pass@1 and by a larger average margin of 15.1 percentage points on pass@100 across model size. We also plot the performance of different sample selection heuristics for Codex-S-12B against the same heuristics for Codex-12B. When ranking between 1 and 100 samples by mean log probability, the average benefit over random ranking is 11.6 percentage points, which is over 2 percentage points higher than the corresponding benefit for Codex.

Evaluating Large Language Models Trained on Code > Figure 9. Optimal sampling temperatures as a function of the num-ber of samples generated for both Codex and Codex-S. Codex-S generally requires a higher temperature for any particular value of > k, possibly to compensate for the fact that it models a narrower distribution. > Figure 10. Comparing Codex-S against Codex on the metrics pro-posed in Section 3. Codex-S is one or two orders of magnitude more parameter efficient on pass@1 and pass@100, and log-prob sample ranking with Codex-S yields similar benefits over random sampling that Codex does.

## 5. Docstring Generation Generating code from docstrings is possible with Codex because code typically follows after a docstring, but it is not easy to induce Codex to generate docstrings from code. Nev-ertheless, we are motivated to produce a docstring writing model for safety reasons, as such a model can be used to de-scribe the intent behind generated code. Using the training problems described in the previous section, we can eas-ily create a training dataset for code-conditional docstring generation. Specifically, for each training problem, we assemble a train-ing example by concatenating the function signature, the reference solution, and then the docstring.

Just as we train Codex-S by minimizing negative log-likelihood of the ref-erence solution, we train the docstring generating models Codex-D by minimizing negative log-likelihood of the doc-string. When we benchmark our code generation models, we mea-sure pass@ k on the HumanEval dataset, where correctness is defined by passing a set of unit tests. However, there is no similar way to evaluate docstring samples automatically. Therefore, we grade sample docstrings by hand, considering a docstring correct if it uniquely and accurately specifies the code body. Due to the time consuming nature of this process, we only grade 10 samples per problem, for a total of 1640 problems, from Codex-D-12B at temperature 0.8.

Codex-D often generates incorrect unit tests along with a docstring, but we ignore these during grading. However, we do not consider the docstring correct when the model simply copies the code body into the docstring. The most common failure modes we observe are when the docstring model leaves out an important detail (such as “an answer must be to two decimal places”) or when it over-conditions on the function name and invents a problem unrelated to the function body. As shown in Table 3, pass rates for Codex-D are lower but comparable to the corresponding pass rates for Codex-S at the same temperature.

We do not have a strong hypothesis for which direction should yield higher pass rates. While generating docstrings may be more forgiving because natu-ral language syntax is less strict than code syntax, docstrings in our dataset may be lower quality because developers tend to devote less time to writing docstrings. Indeed, our model produces docstrings like “I just found this function online” and “This test is not correctly written and it’s not my ” Finally, with a docstring model, we have yet another way to choose a single sample from a set of k samples.

In-stead of picking the sample with the best mean log proba-bility as investigated in the previous two sections, we can choose the sample that maximizes the back-translation ob-Evaluating Large Language Models Trained on Code > Table 3. Pass rates for our docstring generating model Codex-D, which is evaluated by hand-grading 10 samples per task due to the lack of a ground-truth automatic evaluation. We find similar but lower pass-rates compared to Codex-S. MODEL PASS @1 PASS @10 CODEX -S-12B 32.2% 59.5% CODEX -D-12B 20.3% 46.5% jective P (ground truth docstring |generated sample) where P is evaluated using Codex-D.

Unfortunately, in Figure 7, we show that ranking samples via back-translation under-performs mean log-probability ranking, though it outper-forms random ranking. This heuristic also appears to overfit quickly. ## 6. Limitations While Codex is able to sample correct solutions for the majority of HumanEval problems, we find that it has a number of limitations. First, Codex is not sample efficient to train. Our training dataset comprises a significant fraction of publicly available Python code on GitHub, totaling hundreds of millions of lines of code. Even seasoned developers do not encounter anywhere near this amount of code over their careers.

In-deed, a strong student who completes an introductory com-puter science course is expected to be able to solve a larger fraction of problems than Codex-12B. Next, we explore prompts on which Codex is likely to fail or display counter-intuitive behavior. While evaluating code generation is well-studied (Xu et, 2021; Helmuth & Spec-tor, 2015; Pantridge et, 2017), many existing metrics measure performance in tightly specified, constrained prob-lem instances, string manipulation in FlashFill (Gul-wani, 2011)). Therefore, we developed a set of qualitative metrics for measuring the capabilities of code generating models while controlling for the complexity and abstrac-tion level of the specifications (Appendix D).

Applying this framework, we find that Codex can recommend syntacti-cally incorrect or undefined code, and can invoke functions, variables, and attributes that are undefined or outside the scope of the codebase. Moreover, Codex struggles to parse through increasingly long and higher-level or system-level specifications. To concretely illustrate model performance degradation as docstring length increases, we create a dataset of synthetic problems assembled from 13 basic building blocks, each of which modifies an input string in a deterministic way. Ex-ample building blocks are “convert the string to lowercase” or “remove every third character from the string” (the full list is described in Appendix C).

We find that as the number of chained building blocks in the docstring increases, model performance decreases exponentially. This behavior is un-characteristic of a human programmer, who should be able to correctly implement a program for a chain of arbitrary length if they can do so for a chain of length two. > Figure 11. Pass rates of Codex-12B samples against the number of chained components in the synthetically generated docstring. With each additional component, pass rate drops by roughly a factor of 2-3. Further, just as text-conditional generative models in other modalities (Ramesh et

, 2021) have difficulty with bind-ing attributes to objects, Codex can make mistakes binding operations to variables, especially when the number of oper-ations and variables in the docstring is large. For instance, in the following prompt, Codex-12B does not decrement the variable w and also fails to return the product of all numbers. > def do_work(x, y, z, w): > """ Add 3to y, then subtract 4from both xand w. Return the product of the four numbers. """ > t=y+3u=x-4v=z * w > return v This understanding of Codex’s limited system-level synthe-sis capabilities helps inform our assessment of the potential hazards of using it in a generative capacity, as well as the broader societal impacts that such systems could have.

## 7. Broader Impacts and Hazard Analysis Codex has the potential to be useful in a range of ways. For example, it could help onboard users to new codebases, reduce context switching for experienced coders, enable non-programmers to write specifications and have Codex draft implementations, and aid in education and exploration. However, Codex also raises significant safety challenges, does not always produce code that is aligned with user intent, Evaluating Large Language Models Trained on Code and has the potential to be misused. To better understand some of the hazards of using Codex in a generative capacity, we conducted a hazard analysis focused on identifying risk factors (Leveson, 2019) with the potential to cause harm.

1 We outline some of our key findings across several risk areas below. While some of our findings about the potential societal impacts of code generation systems were informed by work towards responsible deployment of the production-oriented Codex models (which descended from the research-oriented Codex models described in this paper), this section is not intended to provide a full account of any particular product’s safety features. Unless otherwise specified, we anchor our analysis in the specific properties of the models described in this paper. We share this analysis in the belief that some of it generalizes to the broader class of code generation systems, and to encourage a norm of performing detailed impact analysis as part of major machine learning research projects.

Note that by focusing largely on risks in this section, we do not mean to imply that we expect the impact of this class of technologies to be net-negative; rather, risks merit particular attention here because they may be subtle or require deliber-ate effort to address, whereas we expect the benefits to be more obvious and “automatic” from the perspective of most users and affected stakeholders. 7.1. Over-reliance One of the key risks associated with using code generation models in practice is over-reliance on generated outputs. Due to the limitations described above as well as alignment issues described below, Codex may suggest solutions that superficially appear correct but do not actually perform the task the user intended.

This could particularly affect novice programmers, and could have significant safety implications depending on the context. We discuss a related issue in Appendix G, namely that code generation models can sug-gest insecure code. For these reasons, human oversight and vigilance is required for safe use of code generation systems like Codex. We note several immediate ways to improve safety in the subsection on risk mitigation below, though over-reliance in particular is one that we believe merits further inquiry in industry and academia. While it is conceptually straight- > 1We sought to include harms spanning geographic and temporal scales.

We also considered not only the severity and probability, but also the distribution of harms. However, we note that the analysis described here is only one milestone in what we hope will be a larger cross-sectoral and cross-organizational effort to steer code generation in a societally beneficial direction. As we describe our findings, we note various specific uncertainties and areas for future work in different sections. > Figure 12. When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of. This persists when the prompt also includes instructions to write correct code.

This gap increases with model size. forward to provide documentation to users reminding them about model limitations, empirical investigation is neces-sary in order to identify how to reliably ensure vigilance in practice across a range of user experience levels, UI designs, and tasks. One challenge researchers should consider is that as capabilities improve, it may become increasingly difficult to guard against “automation ” 7.2. Misalignment As with other large language models trained on a next-token prediction objective, Codex will generate code that is as sim-ilar as possible to its training distribution. One consequence of this is that such models may do things that are unhelpful for the user, despite having the capability to be more helpful (see Figure 12).

For example, if the user has some subtle mistakes in their code, Codex may “deliberately” suggest code that superficially appears good but is incorrect. This is an alignment failure - the model is not aligned with the user’s intentions. Informally, a system is misaligned if there’s some task X that we want it to do, and it is “capable” of doing X but “chooses” not to. In contrast, if a system fails to do X because it does not have the ability to do so, then this system is not misaligned; it is just incompetent.

See Appendix E for more detail, including a more precise definition of alignment. It is important to study misalignment because it is a problem that is likely to become worse, not better, as the capabili-ties of our systems increase. For example, the model size scaling trend for the example in Figure 12 indicates that misalignment would likely persist and even get worse if data, parameters, and training time were scaled up. While we expect that misaligned behaviour like this is un-likely to cause significant harm in current models, it is likely to become more dangerous and harder to eliminate as model Evaluating Large Language Models Trained on Code capabilities increase.

A highly capable but sufficiently mis-aligned model trained on user approval might produce ob-fuscated code that looks good to the user even on careful inspection, but in fact does something undesirable or even harmful. 7.3. Bias and representation Mirroring what has been found in the case of other language models trained on Internet data (Bender et, 2021; Blod-gett et, 2020; Abid et, 2021; Brown et, 2020), we found that Codex can be prompted in ways that generate racist, denigratory, and otherwise harmful outputs as code comments, meriting interventions such as those discussed in the subsection on risk mitigation below.

We also found that code generation models raise further bias and represen-tation issues beyond problematic natural language: Codex can generate code with structure that reflects stereotypes about gender, race, emotion, class, the structure of names, and other characteristics. Particularly in the context of users who might over-rely on Codex or use it without first think-ing through project design, this issue could have significant safety implications, giving further motivation to discourage over-reliance. We discuss bias and representation issues further in Appendix F. Filtration or modulation of generated outputs, documentation, and other interventions may help to mitigate these risks.

7.4. Economic and labor market impacts Code generation and associated capabilities have several possible economic and labor market impacts. While Codex at its current capability level may somewhat reduce the cost of producing software by increasing programmer produc-tivity, the size of this effect may be limited by the fact that engineers don’t spend their full day writing code (O*NET, 2021). Other important tasks include conferring with col-leagues, writing design specifications, and upgrading ex-isting software stacks. 2 We also found that Codex imports packages at different rates, which could advantage some package authors over others, particularly if programmers and engineers come to rely on Codex’s suggestions.

Over a longer time horizon, the effects of this class of technologies on software-related labor markets and on the economy more generally could be more substantial as capabilities improve. More study is needed both on the effects of code genera-tion capabilities and on appropriate responses. We discuss economic and labor market implications in more detail in Appendix H. > 2Indeed, BLS classifies computer programmers and software developers separately, where developers are more highly paid than programmers, have more tasks indirectly related to writing and interacting with code, and, in the US, are already projected to see greater demand over the next 10 years (Li et

, 2020; Bureau of Labor Statistics, 2021a;b). 7.5. Security implications Codex could have various effects on the security landscape. Because Codex can produce vulnerable or misaligned code, 3 qualified operators should review its generations before ex-ecuting or trusting them, absent appropriate precautions. Future code generation models may be able to be trained to produce more secure code than the average developer, though that is far from certain. Codex could also be misused to aid cybercrime. Although this is worthy of concern, based on our testing, we believe that at their current level of capability, Codex models do not materially lower the barrier to entry for malware devel-opment.

4 We expect that more powerful code generation models will lead to future advancements, and therefore fur-ther research into mitigations and continued study of model capabilities are necessary. The non-deterministic nature of systems like Codex could enable more advanced malware. This non-determinism makes it easier to create diverse software that accomplish the same tasks. While software diversity can sometimes aid defenders, 5 it presents unique challenges for traditional malware detection and antivirus systems that rely on finger-printing and signature-matching against previously sampled binaries. For example, a more capable code generation model could conceivably advance techniques for generating polymorphic malware.

6 We believe that application secu-rity and model deployment strategies including rate-limiting access and abuse monitoring can manage this threat in the near term; however, the efficacy of these mitigations may scale sublinearly as more capable models are developed. Similar to large language models, Codex models can learn patterns present in their training data (Carlini et, 2021). Sensitive data present in source code are liable to be pre-dicted by the model. Because Codex is trained on public repositories, we consider any sensitive data present in the training data to have already been compromised. Similarly, the public data should generally be treated as untrusted, as

previous work (Goldblum et, 2021; Schuster et, 2020) has found that attackers may be able to corrupt training data to trigger specific model behaviors at runtime. We further discuss security implications in Appendix G. > 3See Appendix G - Insecure Code for examples of Codex pro-ducing insecure code. > 4For more on characterizing Codex’s capability limitations, see the Limitations section and experiments in the security analysis in Appendix G. > 5For example, by helping to prevent certain types of memory corruption vulnerabilities. See (Davis, 2018) for more. > 6Polymorphic malware is malicious code that mutates its im-plementation while maintaining its function.

Evaluating Large Language Models Trained on Code 7.6. Environmental impacts Codex, like other large generative models, has an energy footprint from both training and inference (Schwartz et, 2019; Bender et, 2021; Patterson et, 2021). The origi-nal training of GPT-3-12B consumed hundreds of petaflop/s-days of compute, while fine-tuning it to create Codex-12B consumed a similar amount of compute. This training was performed on a platform (Azure) that purchases carbon credits and sources significant amounts of renewable energy, reducing its carbon footprint. 7 Compute consumption also has costs in the wider supply chain that can be quite con-centrated on certain regions.

8 Looking more globally and long-term, the compute demands of code generation could grow to be much larger than Codex’s training if significant inference is used to tackle challenging problems. 9 7.7. Legal implications There are