Dota 2 with Large Scale Deep Reinforcement Learning

# Dota 2 with Large Scale Deep Reinforcement Learning OpenAI, ∗ Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław “Psyho" Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, Susan Zhang March 10, 2021 Abstract On April 13th, 2019, OpenAI Five became the first AI system to defeat the world cham-pions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems.

OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approxi-mately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task. # 1 Introduction The long-term goal of artificial intelligence is to solve advanced real-world challenges. Games have served as stepping stones along this path for decades, from Backgammon (1992) to Chess (1997) to Atari (2013)[1–3].

In 2016, AlphaGo defeated the world champion at Go using deep reinforcement learning and Monte Carlo tree search[4]. In recent years, reinforcement learning (RL) models have tackled tasks as varied as robotic manipulation[5], text summarization [6], and video games such as Starcraft[7] and Minecraft[8]. Relative to previous AI milestones like Chess or Go, complex video games start to capture the complexity and continuous nature of the real world. Dota 2 is a multiplayer real-time strategy game produced by Valve Corporation in 2013, which averaged between 500,000 and 1,000,000 concurrent players between 2013 and 2019. The game is actively played by full time professionals; the prize pool for the 2019 international championship exceeded $35 million (the largest of any esports game in the world)[9, 10].

The game presents challenges for reinforcement learning due to long time horizons, partial observability, and high dimensionality of observation and action spaces. Dota 2’s > ∗Authors listed alphabetically. Please cite as OpenAI et, and use the following bibtex for citation: https: bib 1 > arXiv:1912.06680v1 LG] 13 Dec 2019 rules are also complex — the game has been actively developed for over a decade, with game logic implemented in hundreds of thousands of lines of code. The key ingredient in solving this complex environment was to scale existing reinforcement learning systems to unprecedented levels, utilizing thousands of GPUs over multiple months.

We built a distributed training system to do this which we used to train a Dota 2-playing agent called OpenAI Five. In April 2019, OpenAI Five defeated the Dota 2 world champions (Team OG 1), the first time an AI system has beaten an esport world champion 2. We also opened OpenAI Five to the Dota 2 community for competitive play; OpenAI Five won 99.4% of over 7000 games. One challenge we faced in training was that the environment and code continually changed as our project progressed. In order to train without restarting from the beginning after each change, we developed a collection of tools to resume training with minimal loss in performance which we call surgery.

Over the 10-month training process, we performed approximately one surgery per two weeks. These tools allowed us to make frequent improvements to our strongest agent within a shorter time than the typical practice of training from scratch would allow. As AI systems tackle larger and harder problems, further investigation of settings with ever-changing environments and iterative development will be critical. In section 2, we describe Dota 2 in more detail along with the challenges it presents. In section 3 we discuss the technical components of the training system, leaving most of the details to appendices cited therein.

In section 4, we summarize our long-running experiment and the path that lead to defeating the world champions. We also describe lessons we’ve learned about reinforcement learning which may generalize to other complex tasks. # 2 Dota 2 Dota 2 is played on a square map with two teams defending bases in opposite corners. Each team’s base contains a structure called an ancient; the game ends when one of these ancients is destroyed by the opposing team. Teams have five players, each controlling a hero unit with unique abilities. During the game, both teams have a constant stream of small “creep” units, uncontrolled by the players, which walk towards the enemy base attacking any opponent units or buildings.

Players gather resources such as gold from creeps, which they use to increase their hero’s power by purchasing items and improving abilities. 3 To play Dota 2, an AI system must address various challenges: • Long time horizons. Dota 2 games run at 30 frames per second for approximately 45 minutes. OpenAI Five selects an action every fourth frame, yielding approximately 20,000 steps per episode. By comparison, chess usually lasts 80 moves, Go 150 moves[11]. • Partially-observed state. Each team in the game can only see the portion of the game state near their units and buildings; the rest of the map is hidden.

Strong play requires making inferences based on incomplete data, and modeling the opponent’s behavior. > com/OGDota2/ > 2Full game replays and other supplemental can be downloaded from: com/blog/ how-to-train-your-openai-five/ > 3Further information the rules and gameplay of Dota 2 is readily accessible online; a good introductory resource is io/g/dota-2-guide/ 2• High-dimensional action and observation spaces. Dota 2 is played on a large map containing ten heroes, dozens of buildings, dozens of non-player units, and a long tail of game features such as runes, trees, and wards. OpenAI Five observes ∼ 16, 000 total values (mostly floats and

categorical values with hundreds of possibilities) each time step. We discretize the action space; on an average timestep our model chooses among 8,000 to 80,000 actions (de-pending on hero). For comparison Chess requires around one thousand values per observation (mostly 6-possibility categorical values) and Go around six thousand values (all binary)[12]. Chess has a branching factor of around 35 valid actions, and Go around 250[11]. Our system played Dota 2 with two limitations from the regular game: • Subset of 17 heroes — in the normal game players select before the game one from a pool of 117 heroes to play; we support 17 of them.

4 • No support for items which allow a player to temporarily control multiple units at the same time (Illusion Rune, Helm of the Dominator, Manta Style, and Necronomicon). We removed these to avoid the added technical complexity of enabling the agent to control multiple units. # 3 Training System 3.1 Playing Dota using AI Humans interact with the Dota 2 game using a keyboard, mouse, and computer monitor. They make decisions in real time, reason about long-term consequences of their actions, and more. We adopt the following framework to translate the vague problem of “play this complex game at a superhuman level" into a detailed objective suitable for optimization.

Although the Dota 2 engine runs at 30 frames per second, OpenAI Five only acts on every 4th frame which we call a timestep. Each timestep, OpenAI Five receives an observation from the game engine encoding all the information a human player would see such as units’ health, position, etc (see Appendix E for an in-depth discussion of the observation). OpenAI Five then returns a discrete action to the game engine, encoding a desired movement, attack, etc. Certain game mechanics were controlled by hand-scripted logic rather than the policy: the order in which heroes purchase items and abilities, control of the unique courier unit, and which items heroes keep in reserve.

While we believe the agent could ultimately perform better if these actions were not scripted, we achieved superhuman performance before doing so. Full details of our action space and scripted actions are described in Appendix F. Some properties of the environment were randomized during training, including the heroes in the game and which items the heroes purchased. Sufficiently diverse training games are necessary to ensure robustness to the wide variety of strategies and situations that arise in games against human opponents. See subsection 2 for details of the domain randomizations. We define a policy (π) as a function from the history of observations to a probability distribution over actions, which we parameterize as a recurrent neural network with approximately 159 million parameters (θ).

The neural network consists primarily of a single-layer 4096-unit LSTM [13] (see Figure 1). Given a policy, we play games by repeatedly passing the current observation as input and sampling an action from the output distribution at each timestep. > 4See Appendix P for experiments characterizing the effect of hero pool size. 3Figure 1: Simplified OpenAI Five Model Architecture: The complex multi-array observation space is processed into a single vector, which is then passed through a 4096-unit LSTM. The LSTM state is projected to obtain the policy outputs (actions and value function). Each of the five heroes on the team is controlled by a replica of this network with nearly identical inputs, each with its own hidden state.

The networks take different actions due to a part of the observation processing’s output indicating which of the five heroes is being controlled. The LSTM composes 84% of the model’s total parameter count. See Figure 17 and Figure 18 in Appendix H for a detailed breakdown of our model architecture. Separate replicas of the same policy function (with identical parameters θ) are used to control each of the five heroes on the team. Because visible information and fog of war (area that is visible to players due to proximity of friendly units) are shared across a team in Dota 2, the observations are nearly 5 identical for each hero.

Instead of using the pixels on the screen, we approximate the information available to a human player in a set of data arrays (see Appendix E for full details of the observations space). This approximation is imperfect; there are small pieces of information which humans can gain access to which we have not encoded in the observations. On the flip side, while we were careful to ensure that all the information available to the model is also available to a human, the model does get to see all the information available simultaneously every time step, whereas a human needs to actively click to see various parts of the map

and status modifiers. OpenAI Five uses this semantic observation space for two reasons: First, because our goal is to study strategic planning and high-level decision-making rather than focus on visual processing. Second, it is infeasible for us to render each frame to pixels in all training games; this would multiply the computation resources required for the project many-fold. Although these discrepancies exist, we do not believe they introduce significant bias when benchmarking against human players. To allow the five networks to choose different actions, the LSTM receives an extra input from the observation processing, indicating which of the five heroes is being controlled, detailed in Figure 17.

Because of the expansive nature of the problem and the size and expense of each experiment, it was not practical to investigate all the details of the policy and training system. Many details, even > 5We do include a very small number of derived features which depend on the hero being controlled, for example the “distance to me” feature of each unit in the game. 4some large ones, were set for historical reasons or on the basis of preliminary investigations without full ablations. 3.2 Optimizing the Policy Our goal is to find a policy which maximizes the probability of winning the game against professional human experts.

In practice, we maximize a reward function which includes additional signals such as characters dying, collecting resources, etc. We also apply several techniques to exploit the zero-sum multiplayer structure of the problem when computing the reward function — for example, we symmetrize rewards by subtracting the reward earned by the opposing team. We discuss the details of the reward function in Appendix G. We constructed the reward function once at the start of the project based on team members’ familiarity with the game. Although we made minor tweaks when game versions changed, we found that our initial choice of what to reward worked fairly well.

The presence of these additional signals was important for successful training (as discussed in Appendix G). The policy is trained using Proximal Policy Optimization (PPO)[14], a variant of advantage actor critic[15, 16]. 6 The optimization algorithm uses Generalized Advantage Estimation [17] (GAE), a standard advantage-based variance reduction technique [15] to stabilize and accelerate training. We train a network with a central, shared LSTM block, that feeds into separate fully connected layers producing policy and value function outputs. The training system is represented in Figure 2. We train our policy using collected self-play experience from playing Dota 2, similar to [18].

A central pool of optimizer GPUs receives game data and stores it asynchronously in local buffers called experience buffers. Each optimizer GPU computes gradients using minibatches sampled randomly from its experience buffer. Gradients are averaged across the pool using NCCL2[19] allreduce before being synchronously applied to the parameters. In this way the effective batch size is the batch size on each GPU (120 samples, each with 16 timesteps) multiplied by the number of GPUs (up to 1536 at the peak), for a total batch size of 2,949,120 time steps (each with five hero policy replicas).

We apply the Adam optimizer [20] using truncated backpropagation through time[21] over sam-ples of 16 timesteps. Gradients are additionally clipped per parameter to be within between ±5√v where v is the running estimate of the second moment of the (unclipped) gradient. Every 32 gradi-ent steps, the optimizers publish a new version of the parameters to a central Redis 7 storage called the controller. The controller also stores all metadata about the state of the system, for stopping and restarting training runs. “Rollout” worker machines run self-play games. They run these games at approximately 1/2 real time, because we found that we could run slightly more than twice as many games in parallel at this speed, increasing total throughput.

We describe our integration with the Dota 2 engine in Appendix K. They play the latest policy against itself for 80% of games, and play against older policies for 20% of games (for details of opponent sampling, see Appendix N). The rollout machines run the game engine but not the policy; they communicate with a separate pool of GPU machines which run forward passes in larger batches of approximately 60. These machines frequently poll the controller to gather the newest parameters. > 6Early on in the project, we considered other algorithms including other policy gradient methods, q-learning, and evolutionary strategies.

PPO was the first to show initial learning progress. > io 5Figure 2: System Overview: Our training system consists of 4 primary types of machines. Roll-outs run the Dota 2 game on CPUs. They communicate in a tight loop with Forward Pass GPUs, which sample actions from the policy given the current observation. Rollouts send their data to Optimizer GPUs, which perform gradient updates. The Optimizers publish the parameter versions to storage in the Controller, and the Forward Pass GPUs occasionally pull the latest parameter version. Machine numbers are for the Rerun experiment described in subsection 4.2; OpenAI Five’s numbers fluctuated between this scale and approximately 3x larger.

6Rollout machines send data asynchronously from games that are in progress, instead of waiting for an entire game to finish before publishing data for optimization 8; see Figure 8 in Appendix C for more discussion of how rollout data is aggregated. See Figure 5b for the benefits of keeping the rollout-optimization loop tight. Because we use GAE with λ = 0.95, the GAE rewards need to be smoothed over a number of timesteps  1/λ = 20; using 256 timesteps causes relatively little loss. The entire system runs on our custom distributed training platform called Rapid[5], running on Google Cloud Platform.

We use ops from the blocksparse library for fast GPU training[22]. For a full list of the hyperparameters used in training, see Appendix C. 3.3 Continual Transfer via Surgery As the project progressed, our code and environment gradually changed for three different reasons: 1. As we experimented and learned, we implemented changes to the training process (reward structure, observations, etc) or even to the architecture of the policy neural network. 2. Over time we expanded the set of game mechanics supported by the agent’s action and obser-vation spaces. These were not introduced gradually in an effort to build a perfect curriculum.

Rather they were added incrementally as a consequence of following the standard engineering practice of building a system by starting simple and adding complexity piece by piece over time. 3. From time to time, Valve publishes a new Dota 2 version including changes to the core game mechanics and the properties of heroes, items, maps, etc; to compare to human players our agent must play on the latest game version. These changes can modify the shapes and sizes of the model’s layers, the semantic meaning of categorical observation values, etc. When these changes occur, most aspects of the old model are likely relevant in the new envi-ronment.

But cherry-picking parts of the parameter vector to carry over is challenging and limits reproducibility. For these reasons training from scratch is the safe and common response to such changes. However, training OpenAI Five was a multi-month process with high capital expenditure, mo-tivating the need for methods that can persist models across domain and feature changes. It would have been prohibitive (in time and money) to train a fresh model to a high level of skill after each such change (approximately every two weeks). For example, we changed to Dota 2 version 7.21d, eight days before our match against the world champions (OG); this would not have been possible if we had not continued from the previous

agent. Our approach, which we term “surgery”, can be viewed as a collection of tools to perform offline operations to the old model πθ to obtain a new model ˆπˆθ compatible with the new environment, which performs at the same level of skill even if the parameter vectors ˆθ and θ have different sizes and semantics. We then begin training in the new environment using ˆπˆθ. In the simplest case where the environment, observation, and action spaces did not change, our standard reduces to insisting > 8Rollout machines produce 7.5 steps per second; they send data every 256 steps, or 34 seconds of game play.

Because our rollout games run at approximately half-speed, this means they push data approximately once per minute. 7that the new policy implements the same function from observed states to action probabilities as the old: ∀o ˆπˆθ(o) = πθ(o) (1) This case is a special case of Net2Net -style function preserving transformations [23]. We have developed tools to implement Equation 1 exactly when possible (adding observations, expanding layers, and other situations), and approximately when the type of modification to the environment, observation space, or action space precludes satisfying it exactly. See Appendix B for further discussion of surgery.

In the end, we performed over twenty surgeries (along with many unsuccessful surgery attempts) over the ten-month lifetime of OpenAI Five (see Table 1 in Appendix B for a full list). Surgery enabled continuous training without loss in performance (see Figure 4). In subsection 4.2 we discuss our experimental verification of this method. # 4 Experiments and Evaluation OpenAI Five is a single training run that ran from June 30th, 2018 to April 22nd, 2019. After ten months of training using 770 ±50 PFlops/s ·days of compute, it defeated the Dota 2 world champions in a best-of-three match and 99.4% of human players during a multi-day online showcase.

In order to utilize this level of compute effectively we had to scale up along three axes. First, we used batch sizes of 1 to 3 million timesteps (grouped in unrolled LSTM windows of length 16). Second, we used a model with over 150 million parameters. Finally, OpenAI Five trained for 180 days (spread over 10 months of real time due to restarts and reverts). Compared AlphaGo[4], we use 50 to 150 times larger batch size, 20 times larger model, and 25 times longer training time. Simultaneous works in recent months[7, 24] have matched or slightly exceeded our scale.

4.1 Human Evaluation Over the course of training, OpenAI Five played games against numerous amateur players, pro-fessional players, and professional teams in order to gauge progress. For a complete list of the professional teams OpenAI Five played against over time, see Appendix I. On April 13th, OpenAI Five played a high-profile game against OG, the reigning Dota 2 world champions, winning a best-of-three (2-0) and demonstrating that our system can learn to play at the highest levels of skill. For detailed analysis of our agent’s performance during this game and its overall understanding of the environment, see Appendix D.

Machine Learning systems often behave poorly when confronted with unexpected situations[25]. While winning a single high-stakes showmatch against the world champion indicates a very high level of skill, it does not prove a broad understanding of the variety of challenges the human community can present. To explore whether OpenAI Five could be consistently exploited by creative or out-of-distribution play, we ran OpenAI Five Arena, in which we opened OpenAI Five to the public for competitive online games from April 18-21, 2019. In total, Five played 3,193 teams in 7,257 total games, winning 99.4% 9. Twenty-nine teams managed to defeat OpenAI Five for a total of 42 games > 9Human players often abandoned losing games rather than playing them to the end, even abandoning games right after an unfavorable hero selection draft before the main game begins.

OpenAI Five does not abandon games, so we count abandoned games as wins for OpenAI Five. These abandoned games (3140 of the 7215 wins) likely includes a small number of games that were abandoned for technical or personal reasons. 8OG (world champions) > 0100 200 300 400 500 600 700 800 > Compute (PFLOPs/s-days) > 050 100 150 200 250 > TrueSkill > Benchmark (casters) Test team A (semi-pro) Test team B (amateur) Hand-scripted Random > OpenAI Five Pro matches won Calibration matches Figure 3: TrueSkill over the course of training for OpenAI Five. To provide informal context for how TrueSkill corresponds to human skill, we mark the level at which OpenAI Five begins to defeat various opponents, from random to world champions.

Note that this is biased against earlier models; this TrueSkill evaluation is performed using the final policy and environment (Dota 2 version 7.21d, all non-illusion items, etc), even though earlier models were trained in the earlier environment. We believe this contributes to the inflection point around 600 PFLOPs/s-days — around that point we gave the policy control of a new action (buyback) and performed a major Dota 2 version upgrade (7.20). We speculate that the rapid increase to TrueSkill 200 early in training is due to the exponential nature of the scale — a constant TrueSkill difference of approximately 8.3 corresponds to an 80% winrate, and it is easier to learn how to consistently defeat bad agents.

lost. In Dota 2, the key measure of human dexterity is reaction time 10. OpenAI Five can react to a game event in 217ms on average. This quantity does not vary depending on game state. It is difficult to find reliable data on Dota 2 professionals’ reaction times, but typical human visual reaction time is approximately 250ms[26]. See Appendix L for more details. While human evaluation is the ultimate goal, we also need to evaluate our agents continually during training in an automated way. We achieve this by comparing them to a pool of fixed reference agents with known skill using the TrueSkill rating system [27].

In our TrueSkill environment, a rating of 0 corresponds to a random agent, and a difference of approximately 8.3 TrueSkill between two agents roughly corresponds to an 80% winrate of one versus the other (see Appendix J for details of our TrueSkill setup). OpenAI Five’s TrueSkill rating over time can be seen in Figure 3. OpenAI Five’s “playstyle" is difficult to analyze rigorously (and is likely influenced by our shaped reward function) but we can discuss in broad terms the flavor of comments human players made to describe how our agent approached the game. Over the course of training, OpenAI Five developed > 10 Contrast with RTS games like Starcraft, where the key measure is actions per minute due to the large number of units that need to be supplied with actions.

9a distinct style of play with noticeable similarities and differences to human playstyles. Early in training, OpenAI Five prioritized large group fights in the game as opposed to accumulating resources for later, which led to games where they were significantly behind if the enemy team avoided fights early. This playstyle was risky and would result in quick wins in under 20 minutes if OpenAI Five got an early advantage, but had no way to recover from falling behind, leading to long and drawn out losses often over 45 minutes. As the agents improved, the playstyle evolved to align closer with human play while still main-taining many of the characteristics learned early on.

OpenAI Five began to concentrate resources in the hands of its strongest heroes, which is common in human play. Five relied heavily on large group battles, effectively applying pressure when holding a significant advantage, but also avoided fights and focused on gathering resources if behind. The final agent played similar to humans in many broad areas, but had a few interesting dif-ferences. Human players tend to assign heroes to different areas of the map and only reassign occasionally, but OpenAI Five moved heroes back and forth across the map much more frequently. Human players are often cautious when their hero has low health; OpenAI Five seemed to have a very finely-tuned understanding of when an aggressive attack with a low-health hero was worth a risk.

Finally OpenAI Five tended to more readily consume resources, as well as abilities with long cooldowns (time it takes to reload), while humans tend to hold on to those in case a better opportunity arises later. 4.2 Validating Surgery with Rerun In order to validate the time and resources saved by our surgery method (see subsection 3.3), we trained a second agent between May 18, 2019 and June 12, 2019, using only the final environment, model architecture, etc. This training run, called “Rerun”, did not go through a tortuous route of changing game rules, modifications to the neural network parameters, online experiments with hyperparameters, etc.

Rerun took 2 months and 150 ± 5 PFlops/s ·days of compute (see Figure 4). This timeframe is significantly longer than the frequency of our surgery changes (which happened every 1-2 weeks). As a naive comparison, if we had trained from scratch after each of our twenty major surgeries, the project would have taken 40 months instead of 10 (in practice we likely would have made fewer changes). Another benefit of surgery was that we had a very high-skill agent available for evaluation at all times, significantly tightening the iteration loop for experimental changes. In OpenAI Five’s regime — exploring a novel task and building a novel environment —

perpetual training is a significant benefit. Of course, in situations where the environment is pre-built and well-understood from the start, we see little need for surgery. Rerun took approximately 20% of the resources of OpenAI Five; if we had access to the final training environment ahead of time there would be no reason to start training g. on a different version of the game. Rerun continued to improve beyond OpenAI Five’s skill, and reached over 98% winrate against the final version of OpenAI Five. We wanted to validate that our final code and hyperparameters would reproduce OpenAI Five performance, so we ceased training at that point.

We believe Rerun would have continued improving, both because of its upward trend and because we had yet to fully anneal hyperparameters like learning rate and horizon to their final OpenAI Five settings. This process of surgery successfully allowed us to change the environment every week. However, the model ultimately plateaued at a weaker skill level than the from-scratch model was able to 10 0 200 400 600 800 050 100 150 200 250 TrueSkill > Final OpenAI Five TrueSkill = 254 Rerun OpenAI Five > 0200 400 600 800 Total project compute (PFLOPs/s-days) 050 100 150 200 250 TrueSkill > 15x more hypothetical always-restart run time spent retraining from scratch Figure 4: Training in an environment under development: In the top panel we see the full history of our project - we used surgery methods to continue training OpenAI Five at each environment or policy change without loss in performance; then we restarted once at the end to run Rerun.

On the bottom we see the hypothetical alternative, if we had restarted after each change and waited for the model to reach the same level of skill (assuming pessimistically that the curve would be identical to OpenAI Five). The ideal option would be to run Rerun-like training from the very start, but this is impossible — the OpenAI Five curve represents lessons learned that led to the final codebase, environment,, without which it would not be possible to train Rerun. 11 achieve. Learning how to continue long-running training without affecting final performance is a promising area for future work.

Ultimately, while surgery as currently conceived is far from perfect, with proper tooling it becomes a useful method for incorporating certain changes into long-running experiments without paying the cost of a restart for each. 4.3 Batch Size In this section, we evaluate the benefits of increasing the batch size using small scale experiments. Increasing the batch size in our case means two things: first, using twice as many optimizer GPUs to optimize over the larger batch, and second, using twice as many rollout machines and forward pass GPUs to produce twice as many samples to feed the increased optimizer pool.

One compelling benchmark to compare against when increasing the batch size is linear speedup: using 2x as much compute gets to the same skill level in 1/2 the time. If this scaling property holds, it is possible to use the same total amount of GPU-days (and thus dollars) to reach a given result[28]. In practice we see less than this ideal speedup, but the speedup from increasing batch size is still noticeable and allows us to reach the result in less wall time. To understand how batch size affects training speed, we calculate the “speedup” of an experiment to reach various TrueSkill thresholds, defined as: speedup (T) = Versions for baseline to first reach TrueSkill T Versions for experiment to first reach TrueSkill T (2) The results of varying batch size in the early part of training can be seen in Figure 5.

Full details of the experimental setup can be found in Appendix M. We find that increasing the batch size speeds up training through the regime we tested, up to batches of millions of observations. Using the scale of Rerun, we were able to reach superhuman performance in two months. In Figure 5a, we see that Rerun’s batch size (983k time steps) had a speedup factor of around 2.5x over the baseline batch size (123k). If we had instead used the smaller batch size, then, we might expect to wait 5 months for the same result.

We speculate that it would likely be longer, as the speedup factor of 2.5 applies at TrueSkill 175 early in training, but it appears to increase with higher TrueSkill. Per results in [28], we hoped to find (in the early part of training) linear speedup from increasing batch size; e. that it would be 2x faster to train an agent to certain thresholds if we use 2x the compute and data. Our results suggest that speedup is less than linear. However, we speculate that this may change later in training when the problem becomes more difficult.

Also, given the relevant compute costs, in this ablation study we did not tune hyperparameters such as learning rate separately for each batch size. 4.4 Data Quality One unusual feature of our task is the length of the games; each rollout can take up to two hours to complete. For this reason it is infeasible for us to optimize entirely on fully on-policy trajectories; if we waited to apply gradient updates for an entire rollout game to be played using the latest parameters, we could make only one update every two hours. Instead, our rollout workers and optimizers operate asynchronously: rollout workers download the latest parameters, play a small 12 0k 2k 5k 7k 10k 12k 15k > Parameter versions > 025 50 75 100 125 150 175 200 > Trueskill > Batch size 1966k Batch size 983k Batch size 492k Batch size 246k Batch size 123k (b) Batch size 61k 61k 123k 246k 492k 983k 1966k > Batch size (in frames, log scale) > 0.0 0.5 1.0 1.5 2.0 2.5 3.0 > Speedup > TS175 TS125 TS100 Linear speedup (a) Batch size: Larger batch size speeds up training.

In the early part of training studied here, the speedup is sublinear in the computation and samples re-quired. See subsection 1 for ex-periment details. 0k 2k 4k 6k 8k 10k > Parameter versions > 025 50 75 100 125 150 175 200 > Trueskill > Queue length 0 (b) Queue length 1 Queue length 2 Queue length 4 Queue length 8 Queue length 16 Queue length 32 24816 32 > Measured staleness (log) > 0.0 0.2 0.4 0.6 0.8 1.0 1.2 > Speedup > TS150 TS125 TS100 (b) Data Staleness: Training on stale rollout data causes significant losses in training speed.

Queue length estimates the amount of artificial staleness introduced; see subsection 2 for experiment details. 0k 2k 4k 6k > Parameter versions > 025 50 75 100 125 150 175 200 > Trueskill > Sample Reuse 0.5 Sample Reuse 1 (b) Sample Reuse 2 Sample Reuse 4 Sample Reuse 8 0.5 1.0 2.0 4.0 8.0 > Measured sample reuse (log) > 0.0 0.2 0.4 0.6 0.8 1.0 1.2 > Speedup > TS125 TS100 (c) Sample Reuse: Reusing each sample of training data causes significant slowdowns. See subsection 3 for experiment de-tails. Figure 5: Batch Size and data quality in early training: For each parameter, we ran multiple training runs varying only that parameter.

These runs cover early training (approximately one week) at small scale (8x smaller than Rerun). On the left we plot TrueSkill over time for each run. On the right, we plot the “speedup” to reach fixed TrueSkill thresholds of 100, 125, 150, and 175 as a function of the parameter under study compared to the baseline (marked with ‘b’); see Equation 2. Higher speedup means that training was faster and more efficient. These four thresholds are chosen arbitrarily; a few are omitted when the uncertainties are too large (for example in Figure 5c fewer than half the experiments reach 175, so that speedup curve would not be informative).

13 portion of the game, and upload data to the experience buffer, while optimizers continually sample from whatever data is present in the experience buffer to optimize (Figure 2). Early on in the project, we had rollout workers collect full episodes before sending it to the optimizers and downloading new parameters. This means that once the data finally enters the optimizers, it can be several hours old, corresponding to thousands of gradient steps. Gradients computed from these old parameters were often useless or destructive. In the final system rollout workers send data to optimizers after only 256 timesteps, but even so this can be a problem.

We found it useful to define a metric for this called staleness. If a sample was generated by parameter version N and we are now optimizing version M, then we define the staleness of that data to be M − N. In Figure 5b, we see that increasing staleness by ∼ 8 versions causes significant slowdowns. Note that this level of staleness corresponds to a few minutes in a multi-month experiment. Our final system design targeted a staleness between 0 and 1 by sending game data every 30 seconds of gameplay and updating to fresh parameters approximately once a minute, making the loop faster than the time it takes the optimizers to process a single batch (32 PPO gradient steps).

Because of the high impact of staleness, in future work it may be worth investigating whether optimization methods more robust to off-policy data could provide significant improvement in our asynchronous data collection regime. Because optimizers sample from an experience buffer, the same piece of data can be re-used many times. If data is reused too often, it can lead to overfitting on the reused data[18]. To diagnose this, we defined a metric called the sample reuse of the experiment as the instantaneous ratio between the rate of optimizers consuming data and rollouts producing data. If optimizers are consuming samples twice as fast as rollouts are producing them,

then on average each sample is being used twice and we say that the sample reuse is 2. In Figure 5c, we see that reusing the same data even 2-3 times can cause a factor of two slowdown, and reusing it 8 times may prevent the learning of a competent policy altogether. Our final system targets sample reuse ∼ 1 in all our experiments. These experiments on the early part of training indicate that high quality data matters even more than compute consumed; small degradations in data quality have severe effects on learning. Full details of the experiment setup can be found in Appendix M.

4.5 Long term credit assignment Dota 2 has extremely long time dependencies. Where many reinforcement learning environment episodes last hundreds of steps ([4, 29–31]), games of Dota 2 can last for tens of thousands of time steps. Agents must execute plans that play out over many minutes, corresponding to thousands of timesteps. This makes our experiment a unique platform to test the ability of these algorithms to understand long-term credit assignment. In Figure 6, we study the time horizon over which our agent discounts rewards, defined as H = T 1 − γ (3) Here γ is the discount factor [17] and T is the real game time corresponding to each step (0.133 seconds).

This measures the game time over which future rewards are integrated, and we use it as a proxy for the long-term credit assignment which the agent can perform. In Figure 6, we see that resuming training a skilled agent using a longer horizon makes it perform better, up to the longest horizons we explored (6-12 minutes). This implies that our optimization was capable of accurately assigning credit over long time scales, and capable of learning policies and 14 0 500 1000 1500 2000 > Parameter versions > 0.2 0.4 0.6 0.8 1.0 > Win rate > 45s 90s 180s 360s 720s Figure 6: Effect of horizon on agent performance.

We resume training from a trained agent using different horizons (we expect long-horizon planning to be present in highly-skilled agents, but not from-scratch agents). The base agent was trained with a horizon of 180 seconds (γ = 0.9993), and we include as a baseline continued training at horizon 180s. Increasing horizon increases win rate over the trained agent at the point training was resumed, with diminishing returns at high horizons. actions which maximize rewards 6-12 minutes into the future. As the environments we attempt to solve grow in complexity, long-term planning and thinking will become more and more important for intelligent behavior.

# 5 Related Work The OpenAI Five system builds upon several bodies of work combining deep reinforcement learning, large-scale optimization of deep learning models, and using self-play to explore environments and strategies. Competitive games have long served as a testbed for learning. Early systems mastered Backgam-mon [1], Checkers [32], and Chess [2]. Self-play was shown to be a powerful algorithm for learning skills within high-dimensional continuous environments [33] and a method for automatically gener-ating curricula [34]. Our use of self-play is similar in spirit to fictitious play [35], which has been successfully applied to poker [36] - in this work

we learn a distribution over opponents and use the latest policy rather than an average policy. Using a combination of imitation learning human games and self-play, Silver et al. demonstrated a master-level Go player [4]. Building upon this work, AlphaGoZero, AlphaZero, and ExIt discard imitation learning in favor of using Monte-Carlo Tree Search during training to obtain higher quality trajectories [12, 37, 38] and apply this to Go, Chess, Shogi, and Hex. Most recently, human-level play has been demonstrated in 3D first-person multi-player environments [30], professional-level play in the real-time strategy game StarCraft 2 using AlphaStar [7], and superhuman performance 15 in Poker [39].

AlphaStar is particularly relevant to this paper. In that effort, which ran concurrently to our own, researchers trained agents to play Starcraft 2, another complex game with real-time perfor-mance requirements, imperfect information, and long time horizons. The model for AlphaStar used a similar hand-designed architecture to embed observations and an autoregressive action decoder, with an LSTM core to handle partial observability. Both systems used actor critic reinforcement learning methods as part of the overall objective. OpenAI Five has certain sub-systems hard-coded (such as item buying), whereas AlphaStar handled similar decisions g. building order) by con-ditioning (during training) on statistics derived from human replays.

OpenAI Five trained using self play, while AlphaStar used a league consisting of multiple agents, where agents were trained to beat certain subsets of other agents. Finally, AlphaStar’s value network observed full information about the game state (including observations hidden from the policy); this method improved their training and exploring its application to Dota 2 is a promising direction for future work. Deep reinforcement learning has been successfully applied to learning control policies from high dimensional input. In 2013, Mnih et [3] show that it is possible to combine a deep convolutional neural network with a Q-learning algorithm[40] and a novel experience replay approach to learn policies that can reach superhuman performance on the Atari ALE games.

Following this work, a variety of efforts have pushed performance on the remaining Atari games[16], reduced the sample complexity, and introduced new challenges by focusing on intrinsic rewards [41–43]. As more computational resources have become available, a body of work has developed address-ing the use of distributed systems in training. Larger batch sizes were found to accelerate training of image models[44–46]. Proximal Policy Optimization[14] and A3C [47] improve the ability to asyn-chronously collect rollout data. Recent work has demonstrated the benefit of distributed learning on a wide array of problems including single-player video games[48] and robotics[5].

The motivation for our surgery method is similar to prior work on Net2Net style function preserv-ing transformations [23] which attempt to add model capacity without compromising performance, whereas our surgery technique was used in cases where the inputs, outputs, and recurrent layer size changed. Past methods have grown neural networks by incrementally training and freezing parts of the network [49], [50], [51]. Li & Hoiem [52] and Rusu et al. [53] use similar methods to use a trained model to quickly learn novel tasks. Distillation [54] and imitation learning [55, 56] offer an alternate approach to surgery for making model changes in response to a shifting environment.

In concurrent work, OpenAI et al. [24] has reported success using behavioral cloning for similar purposes. # 6 Conclusion When successfully scaled up, modern reinforcement learning techniques can achieve superhuman performance in competitive esports games. The key ingredients are to expand the scale of compute used, by increasing the batch size and total training time. In order to extend the training time of a single run to ten months, we developed surgery techniques for continuing training across changes to the model and environment. While we focused on Dota 2, we hypothesize these results will apply more generally and these methods can solve any zero-sum two-team continuous environment which can be simulated in parallel across hundreds of thousands of instances.

In the future, environments and tasks will continue to grow in complexity. Scaling will become even more important (for current methods) as the tasks become more challenging. 16 Acknowledgements Myriad individuals contributed to this work from within OpenAI, within the Dota 2 community, and elsewhere. We extend our utmost gratitude to everyone who helped us along the way! We would like to especially recognize the following contributions: • Technical discussions with numerous people within OpenAI including Bowen Baker, Paul Christiano, Danny Hernandez, Sam McCandlish, Alec Radford • Review of early drafts by Bowen Baker, Danny Hernandez, Jacob Hilton, Quoc Le, Luke Metz, Matthias Plappert, Alec Radford, Oriol Vinyals • Event Support from Larissa Schiavo, Diane Yoon, Loren Kwan • Communication, writing, and outreach support from Ben Barry, Justin Wang, Shan Carter, Ashley Pilipiszyn, Jack Clark • OpenAI infrastructure support from Eric Sigler • Google Cloud Support (Solomon Boulos, JS Riehl, Florent de Goriainoff, Somnath Roy, Win-ston Lee, Andrew Sallaway, Danny Hammo, Jignesh Naik) • Microsoft Azure Support (Jack Kabat, Jason Vallery, Niel Mackenzie, David Kalmin, Dina Frandsen) • Dota 2 Support from Valve (special thanks to Chris Carollo) • Dota 2 guides and builds from Tortedelini (Michael Cohen) and buyback saving strategy from Adam Michalik • Dota 2 expertise and community advice from Blitz (William Lee) • Dota 2 Casters: Blitz (William Lee), Capitalist (Austin Walsh), Purge (Kevin Godec), ODPixel (Owen Davies), Sheever (Jorien van der Heijden), Kyle Freedman • Dota 2 World Champions (OG): ana (Anathan Pham), Topson (Topias Taavitsainen), Ceb (Sébastien Debs), JerAx (Jesse Vainikka), N0tail (Johan Sundstein) • Dota 2 Professional Teams: Team Secret, Team Lithium, Alliance, SG E-sports • Benchmark Players: Moonmeander (David Tan), Merlini (Ben Wu), Capitalist (Austin Walsh), Fogged (Ioannis Loucas), Blitz (William Lee) • Playtesting: Alec Radford, Bowen Baker, Alex Botev, Pedja Marinkovic, Devin McGarry,

Ryan Perron, Garrett Fisher, Jordan Beeli, Aaron Wasnich, David Park, Connor Mason, James Timothy Herron, Austin Hamilton, Kieran Wasylyshyn, Jakob Roedel, William Rice, Joel Olazagasti, Samuel Anderson • We thank the entire Dota 2 community for their support and enthusiasm. We especially profusely thank all 39,356 Dota 2 players from 225 countries who participated in OpenAI Five Arena and all the players who played against the 1v1 agent during the LAN event at The International 2017! 17 Author Contributions This manuscript is the result of the work of the entire OpenAI Dota team. For each major area, we list the primary contributors in alphabetical order.

• Greg Brockman, Brooke Chan, Przemysław “Psyho" Dębiak, Christy Dennison, David Farhi, Scott Gray, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Szymon Sidor, Jie Tang, Filip Wolski, and Susan Zhang developed and trained OpenAI Five, including developing surgery, expanding tools for large-scale distributed RL, expanding the ca-pabilities to the 5v5 game, and running benchmarks against humans including the OG match and OpenAI Arena. • Christopher Berner, Greg Brockman, Vicki Cheung, Przemysław “Psyho" Dębiak, Quirin Fis-cher, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Catherine Olsson, Jakub Pachocki, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, and Jie Tang developed the 1v1 training system, including the Dota 2 gym interface, building the first Dota agent, and initial exploration of batch size scaling.

• Brooke Chan, David Farhi, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Jie Tang, and Filip Wolski wrote this manuscript, including running Rerun and all of the ab-lation studies. • Jakub Pachocki and Szymon Sidor set research direction throughout the project, including developing the first version of Rapid to demonstrate initial benefits of large scale computing in RL. • Greg Brockman and Rafal Józefowicz kickstarted the team. 18 References 1. Tesauro, G. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural computation 6, 215–219 (1994). 2. Campbell,, Hoane, A. J. & Hsu,

-h. Deep Blue. Artif. Intell. 134, 57–83. issn: 0004-3702 (Jan. 2002). 3. Mnih,, Kavukcuoglu,, Silver,, Graves,, Antonoglou,, Wierstra, D. & Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013). 4. Silver,, Huang,, Maddison, C., Guez,, Sifre,, Van Den Driessche,, Schrittwieser,, Antonoglou,, Panneershelvam,, Lanctot,, et al. Mastering the game of Go with deep neural networks and tree search. nature 529, 484 (2016). 5. OpenAI. Learning Dexterity https: / / openai. com / blog / learning - [Online; accessed 28-May-2019].

2018. 6. Paulus,, Xiong, C. & Socher, R. A Deep Reinforced Model for Abstractive Summarization 2017. arXiv: 1705.04304 7. Vinyals,, Babuschkin,, Czarnecki, W., Mathieu,, Dudzik,, Chung,, Choi, D., Powell,, Ewalds,, Georgiev,, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 1–5 (2019).