# data[].attributes.attachment_link.readable_text
The subject of this article is from the ExoMech update.
The information from this article is is up-to-date as of 10 June, 2020.

Procedural Generation is a mathematical design tool.

No Man's Sky is a game built on procedural generation; that is, each planet, creature, ship, multi-tool and other items are created procedurally using algorithms in the game itself, rendering diversity through each item created. The following is taken from Wikipedia.

What is Procedural Generation?

The term procedural refers to the process that computes a particular function. Fractals, an example of procedural generation, express this concept, around which a whole body of mathematics-fractal geometry-has evolved. Commonplace procedural content includes textures and meshes. Sound is often procedurally generated as well and has applications in both speech synthesis as well as music. It has been used to create compositions in various genres of electronic music by artists such as Brian Eno who popularized the term "generative music".

Previous use

While software developers have applied procedural generation techniques for years, few products have employed this approach extensively. Procedurally generated elements have appeared in earlier video games: The Elder Scrolls II: Daggerfall takes place on a mostly procedurally generated world, giving a world roughly twice the actual size of the British Isles. Soldier of Fortune from Raven Software uses simple routines to detail enemy models, while its sequel featured a randomly-generated level mode. Avalanche Studios employed procedural generation to create a large and varied group of tropical islands in great detail for Just Cause. "No Man's Sky," a game being developed by games studio Hello Games is all based upon Procedurally generated elements.

Uses of Procedural Generation

Almost all objects in the No Man's Sky universe are procedurally generated. The only things in the universe that are not procedurally generated are the names of the first five galaxies. These are some, but not by a long shot all, of the procedurally generated objects in the game:

The player's starting starship as of Beyond, Radiant Pillar BC1, is actually procedurally generated. It is always the same due to it being generated immediately on the start of a world, and the IGS (In-Game Seed) is always the same. The IGS differs from the LSS (Local System Seed) in that the LSS is essentially the system's RNG algorithm, whereas the IGS is run by the game itself.

Additional information

# data[].attributes.attachment_link.description
Procedural Generation is a mathematical design tool. 1 Summary 2 What is Procedural Generation? 2.1 Previous use 3 Uses of Procedural Generation 4 Additional information No Man's Sky is a game built on procedural generation; that is, each planet, creature, ship, multi-tool and other items are created procedurally using algorithms in the game itself, rendering diversity through each item created.

# data[].attributes.attachment_link.readable_text
Whether we're learning to cook an omelet or drive a car, the path to mastering new skills often begins by watching others. But can artificial intelligence learn the same way? A new challenge teaching AI agents to play Minecraft suggests it's much trickier for computers.

Announced earlier this year, the MineRL competition asked teams of researchers to create AI bots that could successfully mine a diamond in Minecraft. This isn't an impossible task, but it does require a mastery of the game's basics. Players need to know how to cut down trees, craft pickaxes, and explore underground caves while dodging monsters and lava. These are the sorts of skills that most adults could pick up after a few hours of experimentation or learn much faster by watching tutorials on YouTube.

But of the 660 entries in the MineRL competition, none were able to complete the challenge, according to results that will be announced at the AI conference NeurIPS and that were first reported by . Although bots were able to learn intermediary steps, like constructing a furnace to make durable pickaxes, none successfully found a diamond.

"The task we posed is very hard," Katja Hofmann, a principal researcher at Microsoft Research, which helped organize the challenge, told BBC News. "While no submitted agent has fully solved the task, they have made a lot of progress and learned to make many of the tools needed along the way."

This may be a surprise, especially when you think that AI has managed to best humans at games like chess, Go, and Dota 2. But it reflects important limitations of the technology as well as restrictions put in place by MineRL's judges to really challenge the teams.

The bots in MineRL had to learn using a combination of methods known as imitation learning and reinforcement learning. In imitation learning, agents are shown data of the task ahead of them, and they try to imitate it. In reinforcement learning, they're simply dumped into a virtual world and left to work things out for themselves using trial and error.

Often, AI is only able to take on big challenges by combining these two methods. The famous AlphaGo system, for example, first learned to play Go by being fed data of old games. It then honed its skills - and surpassed all humans - by playing itself over and over.

The MineRL bots took a similar approach, but the resources available to them were comparatively limited. While AI agents like AlphaGo are created with huge datasets, powerful computer hardware, and the equivalent of decades of training time, the MineRL bots had to make do with just 1,000 hours of recorded gameplay to learn from, a single Nvidia graphics processor to train with, and just four days to get up to speed.

It's the difference between the resources available to an MLB team - coaches, nutritionists, the finest equipment money can buy - and what a Little League squad has to make do with.

It may seem unfair to hamstring the MineRL bots in this way, but these constraints reflect the challenges of integrating AI into the real world. While bots like AlphaGo certainly push the boundary of what AI can achieve, very few companies and research labs can match the resources of Google-owned DeepMind.

The competition's lead organizer, Carnegie Mellon University PhD student William Guss, told BBC News that the challenge was meant to show that not every AI problem should be solved by throwing computing power at it. This mindset, said Guss, "works directly against democratizing access to these reinforcement learning systems, and leaves the ability to train agents in complex environments to corporations with swathes of compute."

So while AI may be struggling in Minecraft now, when it cracks this challenge, it'll hopefully deliver benefits to a wider audience. Just don't think about those poor Minecraft YouTubers who might be out of a job.

# data[].attributes.attachment_link.title
AI has bested chess and Go, but it struggles to find a diamond in Minecraft

# data[].attributes.attachment_link.description
Machine learning and AI have been used to create bots that can beat the best humans at chess and Go. But can they mine a diamond in Minecraft? A new challenge organized with the help of Microsoft put teams from around the world to the test.

# data[].attributes.attachment_link.readable_text
DeepMind Technologies is a British artificial intelligence company and research laboratory founded in September 2010, and acquired by Google in 2014. The company is based in London, with research centres in Canada, France, and the United States. In 2015, it became a wholly owned subsidiary of Alphabet Inc, Google's parent company.

DeepMind has created a neural network that learns how to play video games in a fashion similar to that of humans, as well as a Neural Turing machine, or a neural network that may be able to access an external memory like a conventional Turing machine, resulting in a computer that mimics the short-term memory of the human brain.

DeepMind made headlines in 2016 after its AlphaGo program beat a human professional Go player Lee Sedol, the world champion, in a five-game match, which was the subject of a documentary film. A more general program, AlphaZero, beat the most powerful programs playing go, chess and shogi (Japanese chess) after a few days of play against itself using reinforcement learning. In 2020, DeepMind made significant advances in the problem of protein folding.

The start-up was founded by Demis Hassabis, Shane Legg and Mustafa Suleyman in 2010. Hassabis and Legg first met at University College London's Gatsby Computational Neuroscience Unit.

During one of the interviews, Demis Hassabis said that the start-up began working on artificial intelligence technology by teaching it how to play old games from the seventies and eighties, which are relatively primitive compared to the ones that are available today. Some of those games included Breakout, Pong and Space Invaders. AI was introduced to one game at a time, without any prior knowledge of its rules. After spending some time on learning the game, AI would eventually become an expert in it. "The cognitive processes which the AI goes through are said to be very like those a human who had never seen the game would use to understand and attempt to master it." The goal of the founders is to create a general-purpose AI that can be useful and effective for almost anything.

Major venture capital firms Horizons Ventures and Founders Fund invested in the company, as well as entrepreneurs Scott Banister, Peter Thiel, and Elon Musk. Jaan Tallinn was an early investor and an adviser to the company. On 26 January 2014, Google announced the company had acquired DeepMind for $500 million, and that it had agreed to take over DeepMind Technologies. The sale to Google took place after Facebook reportedly ended negotiations with DeepMind Technologies in 2013. The company was afterwards renamed Google DeepMind and kept that name for about two years.

In 2014, DeepMind received the "Company of the Year" award from Cambridge Computer Laboratory.

In September 2015, DeepMind and the Royal Free NHS Trust signed their initial Information Sharing Agreement (ISA) to co-develop a clinical task management app, Streams.

After Google's acquisition the company established an artificial intelligence ethics board. The ethics board for AI research remains a mystery, with both Google and DeepMind declining to reveal who sits on the board. DeepMind, together with Amazon, Google, Facebook, IBM and Microsoft, is a founding member of Partnership on AI, an organization devoted to the society-AI interface. DeepMind has opened a new unit called DeepMind Ethics and Society and focused on the ethical and societal questions raised by artificial intelligence featuring prominent philosopher Nick Bostrom as advisor. In October 2017, DeepMind launched a new research team to investigate AI ethics.

In December 2019, Co-founder Suleyman announced he would be leaving DeepMind to join Google, working in a policy role.

According to the company's website, DeepMind Technologies' goal is to combine "the best techniques from machine learning and systems neuroscience to build powerful general-purpose learning algorithms".

Google Research has released a paper in 2016 regarding AI Safety and avoiding undesirable behaviour during the AI learning process. Deepmind has also released several publications via its website. In 2017 DeepMind released GridWorld, an open-source testbed for evaluating whether an algorithm learns to disable its kill switch or otherwise exhibits certain undesirable behaviours.

In July 2018, researchers from DeepMind trained one of its systems to play the computer game Quake III Arena.

As of 2020, DeepMind has published over a thousand papers, including thirteen papers that were accepted by or .[citation needed ] DeepMind received media attention during the AlphaGo period; according to a LexisNexis search, 1842 published news stories mentioned DeepMind in 2016, declining to 1363 in 2019.

As opposed to other AIs, such as IBM's Deep Blue or Watson, which were developed for a pre-defined purpose and only function within its scope, DeepMind claims that its system is not pre-programmed: it learns from experience, using only raw pixels as data input. Technically it uses deep learning on a convolutional neural network, with a novel form of Q-learning, a form of model-free reinforcement learning. They test the system on video games, notably early arcade games, such as Space Invaders or . Without altering the code, the AI begins to understand how to play the game, and after some time plays, for a few games (most notably Breakout), a more efficient game than any human ever could.

In 2013, DeepMind published research on an AI system that could surpass human abilities in games such as Pong, Breakout and Enduro, while surpassing state of the art performance on Seaquest, Beamrider, and Q*bert. This work reportedly led to the company's acquisition by Google. DeepMind's AI had been applied to video games made in the 1970s and 1980s; work was ongoing for more complex 3D games such as , which first appeared in the early 1990s.

In 2020, DeepMind published Agent57, an AI Agent which surpasses human level performance on all 57 games of the Atari2600 suite.

In 2014, the company published research on computer systems that are able to play Go.

In October 2015, a computer Go program called AlphaGo, developed by DeepMind, beat the European Go champion Fan Hui, a 2 dan (out of 9 dan possible) professional, five to zero. This is the first time an artificial intelligence (AI) defeated a professional Go player. Previously, computers were only known to have played Go at "amateur" level. Go is considered much more difficult for computers to win compared to other games like chess, due to the much larger number of possibilities, making it prohibitively difficult for traditional AI methods such as brute-force.

In March 2016 it beat Lee Sedol-a 9th dan Go player and one of the highest ranked players in the world-with a score of 4-1 in a five-game match.

In the 2017 Future of Go Summit, AlphaGo won a three-game match with Ke Jie, who at the time continuously held the world No. 1 ranking for two years. It used a supervised learning protocol, studying large numbers of games played by humans against each other.

In 2017, an improved version, AlphaGo Zero, defeated AlphaGo 100 games to 0. AlphaGo Zero's strategies were self-taught. AlphaGo Zero was able to beat its predecessor after just three days with less processing power than AlphaGo; in comparison, the original AlphaGo needed months to learn how to play.

Later that year, AlphaZero, a modified version of AlphaGo Zero but for handling any two-player game of perfect information, gained superhuman abilities at chess and shogi. Like AlphaGo Zero, AlphaZero learned solely through self-play.

Technology

AlphaGo technology was developed based on the deep reinforcement learning approach. This makes AlphaGo different from the rest of AI technologies on the market. With that said, AlphaGo's 'brain' was introduced to various moves based on the historical tournament data. The number of moves was increased gradually until it eventually processed over 30 million of them. The aim was to have the system mimic the human player and eventually become better. It played against itself and learned not only from its own defeats but wins as well; thus, it learned to improve itself over the time and increased its winning rate as a result.

AlphaGo used two deep neural networks: a policy network to evaluate move probabilities and a value network to assess positions. The policy network trained via supervised learning, and was subsequently refined by policy-gradient reinforcement learning. The value network learned to predict winners of games played by the policy network against itself. After training these networks employed a lookahead Monte Carlo tree search (MCTS), using the policy network to identify candidate high-probability moves, while the value network (in conjunction with Monte Carlo rollouts using a fast rollout policy) evaluated tree positions.

Zero trained using reinforcement learning in which the system played millions of games against itself. Its only guide was to increase its win rate. It did so without learning from games played by humans. Its only input features are the black and white stones from the board. It uses a single neural network, rather than separate policy and value networks. Its simplified tree search relies upon this neural network to evaluate positions and sample moves, without Monte Carlo rollouts. A new reinforcement learning algorithm incorporates lookahead search inside the training loop. AlphaGo Zero employed around 15 people and millions in computing resources. Ultimately, it needed much less computing power than AlphaGo, running on four specialized AI processors (Google TPUs), instead of AlphaGo's 48.

In 2016, DeepMind turned its artificial intelligence to protein folding, one of the toughest problems in science. In December 2018, DeepMind's AlphaFold won the 13th Critical Assessment of Techniques for Protein Structure Prediction (CASP) by successfully predicting the most accurate structure for 25 out of 43 proteins. "This is a lighthouse project, our first major investment in terms of people and resources into a fundamental, very important, real-world scientific problem," Hassabis said to The Guardian. In 2020, in the 14th CASP, AlphaFold's predictions achieved an accuracy score regarded as comparable with lab techniques. Dr Andriy Kryshtafovych, one of the panel of scientific adjudicators, described the achievement as "truly remarkable", and said the problem of predicting how proteins fold had been "largely solved".

In 2016, DeepMind introduced WaveNet, a text-to-speech system. It was originally too computationally intensive for use in consumer products, but in late 2017 it became ready for use in consumer applications such as Google Assistant. In 2018 Google launched a commercial text-to-speech product, Cloud Text-to-Speech, based on WaveNet.

In 2018, DeepMind introduced a more efficient model called WaveRNN co-developed with Google AI. In 2019, Google started to roll it out to Google Duo users.

In 2016, Hassabis discussed the game as a future challenge, since it requires strategic thinking and handling imperfect information.

In January 2019, DeepMind introduced AlphaStar, a program playing the real-time strategy game StarCraft II. AlphaStar used reinforcement learning based on replays from human players, and then played against itself to enhance its skills. At the time of the presentation, AlphaStar had knowledge equivalent to 200 years of playing time. It won 10 consecutive matches against two professional players, although it had the unfair advantage of being able to see the entire field, unlike a human player who has to move the camera manually. A preliminary version in which that advantage was fixed lost a subsequent match.

In July 2019, AlphaStar began playing against random humans on the public 1v1 European multiplayer ladder. Unlike the first iteration of AlphaStar, which played only Protoss v. Protoss, this one played as all of the game's races, and had earlier unfair advantages fixed. By October 2019, AlphaStar reached Grandmaster level on the StarCraft II ladder on all three StarCraft races, becoming the first AI to reach the top league of a widely popular esport without any game restrictions.

Google has stated that DeepMind algorithms have greatly increased the efficiency of cooling its data centers. In addition, DeepMind (alongside other Alphabet AI researchers) assists Google Play's personalized app recommendations. DeepMind has also collaborated with the Android team at Google for the creation of two new features which were made available to people with devices running Android Pie, the ninth installment of Google's mobile operating system. These features, Adaptive Battery and Adaptive Brightness, use machine learning to conserve energy and make devices running the operating system easier to use. It is the first time DeepMind has used these techniques on such a small scale, with typical machine learning applications requiring orders of magnitude more computing power.

In July 2016, a collaboration between DeepMind and Moorfields Eye Hospital was announced to develop AI applications for healthcare. DeepMind would be applied to the analysis of anonymised eye scans, searching for early signs of diseases leading to blindness.

In August 2016, a research programme with University College London Hospital was announced with the aim of developing an algorithm that can automatically differentiate between healthy and cancerous tissues in head and neck areas.

There are also projects with the Royal Free London NHS Foundation Trust and Imperial College Healthcare NHS Trust to develop new clinical mobile apps linked to electronic patient records. Staff at the Royal Free Hospital were reported as saying in December 2017 that access to patient data through the app had saved a 'huge amount of time' and made a 'phenomenal' difference to the management of patients with acute kidney injury. Test result data is sent to staff's mobile phones and alerts them to change in the patient's condition. It also enables staff to see if someone else has responded, and to show patients their results in visual form.[unreliable source? ]

In November 2017, DeepMind announced a research partnership with the Cancer Research UK Centre at Imperial College London with the goal of improving breast cancer detection by applying machine learning to mammography. Additionally, in February 2018, DeepMind announced it was working with the U.S. Department of Veterans Affairs in an attempt to use machine learning to predict the onset of acute kidney injury in patients, and also more broadly the general deterioration of patients during a hospital stay so that doctors and nurses can more quickly treat patients in need.

DeepMind developed an app called Streams, which sends alerts to doctors about patients at risk of acute risk injury. On 13 November 2018, DeepMind announced that its health division and the Streams app would be absorbed into Google Health. Privacy advocates said the announcement betrayed patient trust and appeared to contradict previous statements by DeepMind that patient data would not be connected to Google accounts or services. A spokesman for DeepMind said that patient data would still be kept separate from Google services or projects.

In April 2016, New Scientist obtained a copy of a data sharing agreement between DeepMind and the Royal Free London NHS Foundation Trust. The latter operates three London hospitals where an estimated 1.6 million patients are treated annually. The agreement shows DeepMind Health had access to admissions, discharge and transfer data, accident and emergency, pathology and radiology, and critical care at these hospitals. This included personal details such as whether patients had been diagnosed with HIV, suffered from depression or had ever undergone an abortion in order to conduct research to seek better outcomes in various health conditions.

A complaint was filed to the Information Commissioner's Office (ICO), arguing that the data should be pseudonymised and encrypted. In May 2016, New Scientist published a further article claiming that the project had failed to secure approval from the Confidentiality Advisory Group of the Medicines and Healthcare products Regulatory Agency.

In May 2017, Sky News published a leaked letter from the National Data Guardian, Dame Fiona Caldicott, revealing that in her "considered opinion" the data-sharing agreement between DeepMind and the Royal Free took place on an "inappropriate legal basis". The Information Commissioner's Office ruled in July 2017 that the Royal Free hospital failed to comply with the Data Protection Act when it handed over personal data of 1.6 million patients to DeepMind.

In October 2017, DeepMind announced a new research unit, DeepMind Ethics & Society. Their goal is to fund external research of the following themes: privacy, transparency, and fairness; economic impacts; governance and accountability; managing AI risk; AI morality and values; and how AI can address the world's challenges. As a result, the team hopes to further understand the ethical implications of AI and aid society to seeing AI can be beneficial.

This new subdivision of DeepMind is a completely separate unit from the partnership of leading companies using AI, academia, civil society organizations and nonprofits of the name Partnership on Artificial Intelligence to Benefit People and Society of which DeepMind is also a part. The DeepMind Ethics and Society board is also distinct from the mooted AI Ethics Board that Google originally agreed to form when acquiring DeepMind.

DeepMind sponsors two chairs of machine learning:

1. one at the University of Cambridge, held by Neil Lawrence, in the Department of Computer Science and Technology

2. another at the University of Oxford. in the Department of Computer Science

1. ^ Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David (26 February 2015). "Human-level control through deep reinforcement learning". Nature. 518 (7540): 529-33. Bibcode:2015Natur.518..529M. doi:10.1038/nature14236. PMID 25719670. S2CID 205242740.

2. "The Last AI Breakthrough DeepMind Made Before Google Bought It". The Physics arXiv Blog. 29 January 2014.

3. ^ Graves, Alex; Wayne, Greg; Danihelka, Ivo (2014). "Neural Turing Machines". arXiv:1410.5401 [cs.NE].

4. ^ Best of 2014: Google's Secretive DeepMind Startup Unveils a "Neural Turing Machine", MIT Technology Review

5. ^ Graves, Alex; Wayne, Greg; Reynolds, Malcolm; Harley, Tim; Danihelka, Ivo; Grabska-Barwińska, Agnieszka; Colmenarejo, Sergio Gómez; Grefenstette, Edward; Ramalho, Tiago (12 October 2016). "Hybrid computing using a neural network with dynamic external memory". Nature. 538 (7626): 471-476. Bibcode:2016Natur.538..471G. doi:10.1038/nature20101. ISSN 1476-4687. PMID 27732574. S2CID 205251479.

6. ^ Silver, David; Hubert, Thomas; Schrittwieser, Julian; Antonoglou, Ioannis; Lai, Matthew; Guez, Arthur; Lanctot, Marc; Sifre, Laurent; Kumaran, Dharshan; Graepel, Thore; Lillicrap, Timothy; Simonyan, Karen; Hassabis, Demis (5 December 2017). "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm". arXiv:1712.01815 [cs.AI].

7. ^ Shead, Sam. "How DeepMind convinced billionaire Peter Thiel to invest without moving the company to Silicon Valley". Business Insider.

8. ^ Cuthbertson, Anthony (4 August 2014). "Elon Musk: Artificial Intelligence 'Potentially More Dangerous Than Nukes'". International Business Times UK.

9. Madhumita Murgia, ^ "DeepMind co-founder leaves for policy role at Google", Financial Times, 5 December 2019

10. ^

11. ^ "DeepMind AI's new trick is playing 'Quake III Arena' like a human". Engadget. 3 July 2018.

12. ^

13. ^

14. ^

15. ^ Huang, Shih-Chieh; Müller, Martin (12 July 2014). Investigating the Limits of Monte-Carlo Tree Search Methods in Computer Go. Lecture Notes in Computer Science. 8427. Springer. pp. 39-48. CiteSeerX 10.1.1.500.1701. doi:10.1007/978-3-319-09165-5_4. ISBN 978-3-319-09164-8.

16. ^

17. ^ "World's Go Player Ratings". May 2017.

18. ^ "柯洁迎19岁生日 雄踞人类世界排名第一已两年" (in Chinese). May 2017.

19. Silver, David; Schrittwieser, Julian; Simonyan, Karen; Antonoglou, Ioannis; Huang, Aja; Guez, Arthur; Hubert, Thomas; Baker, Lucas; Lai, Matthew; Bolton, Adrian; Chen, Yutian; Lillicrap, Timothy; Fan, Hui; Sifre, Laurent; Driessche, George van den; Graepel, Thore; Hassabis, Demis (19 October 2017). "Mastering the game of Go without human knowledge" (PDF). Nature. 550 (7676): 354-359. Bibcode:2017Natur.550..354S. doi:10.1038/nature24270. ISSN 0028-0836. PMID 29052630. S2CID 205261034.

20. ^ "Google DeepMind's Streams technology branded 'phenomenal'". Digital Health. 4 December 2017.

21. ^ Vincent, James (13 November 2018). "Google is absorbing DeepMind's health care unit to create an 'AI assistant for nurses and doctors'". The Verge.

22. ^ Hodson, Hal (29 April 2016). "Revealed: Google AI has access to huge haul of NHS patient data". New Scientist.

23. ^ "Leader: If Google has nothing to hide about NHS data, why so secretive?". New Scientist. 4 May 2016.

24. ^ Donnelly, Caroline (12 May 2016). "ICO probes Google DeepMind patient data-sharing deal with NHS Hospital Trust". Computer Weekly.

25. ^ Martin, Alexander J (15 May 2017). "Google received 1.6 million NHS patients' data on an 'inappropriate legal basis'". Sky News.

# data[].attributes.attachment_link.description
DeepMind Technologies is a British artificial intelligence company and research laboratory founded in September 2010, and acquired by Google in 2014. The company is based in London, with research centres in Canada, France, and the United States. In 2015, it became a wholly owned subsidiary of Alphabet Inc, Google's parent company.

# data[].attributes.attachment_link.readable_text
In video games, various artificial intelligence techniques have been used in a variety of ways, ranging from non-player character (NPC) control to procedural content generation (PCG). Machine learning is a subset of artificial intelligence that focuses on using algorithms and statistical models to make machines act without specific programming. This is in sharp contrast to traditional methods of artificial intelligence such as search trees and expert systems.

Information on machine learning techniques in the field of games is mostly known to public through research projects as most gaming companies choose not to publish specific information about their intellectual property. The most publicly known application of machine learning in games is likely the use of deep learning agents that compete with professional human players in complex strategy games. There has been a significant application of machine learning on games such as Atari/ALE, , , , and car racing. Other games that did not originally exists as video games, such as chess and Go have also been affected by the machine learning.

Deep learning is a subset of machine learning which focuses heavily on the use of artificial neural networks (ANN) that learn to solve complex tasks. Deep learning uses multiple layers of ANN and other techniques to progressively extract information from an input. Due to this complex layered approach, deep learning models often require powerful machines to train and run on.

Convolutional neural networks (CNN) are specialized ANNs that are often used to analyze image data. These types of networks are able to learn translation invariant patterns, which are patterns that are not dependent on location. CNNs are able to learn these patterns in a hierarchy, meaning that earlier convolutional layers will learn smaller local patterns while later layers will learn larger patterns based on the previous patterns. A CNN's ability to learn visual data has made it a commonly used tool for deep learning in games.

Recurrent neural networks are a type of ANN that are designed to process sequences of data in order, one part at a time rather than all at once. An RNN runs over each part of a sequence, using the current part of the sequence along with memory of previous parts of the current sequence to produce an output. These types of ANN are highly effective at tasks such as speech recognition and other problems that depend heavily on temporal order. There are several types of RNNs with different internal configurations; the basic implementation suffers from a lack of long term memory due to the vanishing gradient problem, thus it is rarely used over newer implementations.

A long short-term memory (LSTM) network is a specific implementation of a RNN that is designed to deal with the vanishing gradient problem seen in simple RNNs, which would lead to them gradually "forgetting" about previous parts of an inputted sequence when calculating the output of a current part. LSTMs solve this problem with the addition of an elaborate system that uses an additional input/output to keep track of long term data. LSTMs have achieved very strong results across various fields, and were used by several monumental deep learning agents in games.

Reinforcement learning is the process of training an agent using rewards and/or punishments. The way an agent is rewarded or punished depends heavily on the problem; such as giving an agent a positive reward for winning a game or a negative one for losing. Reinforcement learning is used heavily in the field of machine learning and can be seen in methods such as Q-learning, policy search, Deep Q-networks and others. It has seen strong performance in both the field of games and robotics.

Neuroevolution involves the use of both neural networks and evolutionary algorithms. Instead of using gradient descent like most neural networks, neuroevolution models make use of evolutionary algorithms to update neurons in the network. Researchers claim that this process is less likely to get stuck in a local minimum and is potentially faster than state of the art deep learning techniques.

Machine learning agents have been used to take the place of a human player rather than function as NPCs, which are deliberately added into video games as part of designed gameplay. Deep learning agents have achieved impressive results when used in competition with both humans and other artificial intelligence agents.

Chess is a turn-based strategy game that is considered a difficult AI problem due to the computational complexity of its board space. Similar strategy games are often solved with some form of a Minimax Tree Search. These types of AI agents have been known to beat professional human players, such as the historic 1997 Deep Blue versus Garry Kasparov match. Since then, machine learning agents have shown ever greater success than previous AI agents.

Go is another turn-based strategy game which is considered an even more difficult AI problem than chess. The state space of is Go is around 10^170 possible board states compared to the 10^120 board states for Chess. Prior to recent deep learning models, AI Go agents were only able to play at the level of a human amateur.

Google's 2015 AlphaGo was the first AI agent to beat a professional Go player. AlphaGo used a deep learning model to train the weights of a Monte Carlo tree search (MCTS). The deep learning model consisted of 2 ANN, a policy network to predict the probabilities of potential moves by opponents, and a value network to predict the win chance of a given state. The deep learning model allows the agent to explore potential game states more efficiently than a vanilla MCTS. The network were initially trained on games of humans players and then were further trained by games against itself.

AlphaGo Zero, another implementation of AlphaGo, was able to train entirely by playing against itself. It was able to quickly train up to the capabilities of the previous agent.

and its sequel StarCraft II: Wings of Liberty are real-time strategy (RTS) video games that have become popular environments for AI research. Blizzard and DeepMind have worked together to release a public StarCraft 2 environment for AI research to be done on. Various deep learning methods have been tested on both games, though most agents usually have trouble outperforming the default AI with cheats enabled or skilled players of the game.

Alphastar was the first AI agent to beat professional StarCraft 2 players without any in-game advantages. The deep learning network of the agent initially received input from a simplified zoomed out version of the gamestate, but was later updated to play using a camera like other human players. The developers have not publicly released the code or architecture of their model, but have listed several state of the art machine learning techniques such as relational deep reinforcement learning, long short-term memory, auto-regressive policy heads, pointer networks, and centralized value baseline. Alphastar was initially trained with supervised learning, it watched replays of many human games in order to learn basic strategies. It then trained against different versions of itself and was improved through reinforcement learning. The final version was hugely successful, but only trained to play on a specific map in a protoss mirror matchup.

is a multiplayer online battle arena (MOBA) game. Like other complex games, traditional AI agents have not been able to compete on the same level as professional human player. The only widely published information on AI agents attempted on Dota 2 is OpenAI's deep learning Five agent.

OpenAI Five utilized separate LSTM networks to learn each hero. It trained using a reinforcement learning technique known as Proximal Policy Learning running on a system containing 256 GPUs and 128,000 CPU cores. Five trained for months, accumulating 180 years of game experience each day, before facing off with professional players. It was eventually able to beat the 2018 Dota 2 esports champion team in a 2019 series of games.

Planetary Annihilation is a real-time strategy game which focuses on massive scale war. The developers use ANNs in their default AI agent.

Supreme Commander 2 is a real-time strategy (RTS) video game. The game uses Multilayer Perceprtrons (MLPs) to control a platoon's reaction to encountered enemy units. Total of four MLPs are used, one for each platoon type: land, naval, bomber, and fighter.

There have been attempts to make machine learning agents that are able to play more than one game. These "general" gaming agents are trained to understand games based on shared properties between them.

AlphaZero is a modified version of AlphaGo Zero which is able to play Shogi, chess, and Go. The modified agent starts without only information basic rules of the game, and is also trained entirely through self-learning. DeepMind was able to train this generalized agent to be competitive with previous versions of itself on Go, as well as top agents in the other two games.

Machine learning agents are often not covered in many game design courses. Previous use of machine learning agents in games may not have been very practical, as even the 2015 version of AlphaGo took hundreds of CPUs and GPUs to train to a strong level. This potentially limits the creation of highly effective deep learning agents to large corporations or extremely wealthy individuals. The extensive training time of neural network based approaches can also take weeks on these powerful machines.

The problem of effectively training ANN based models extends beyond powerful hardware environments; finding a good way to represent data and learn meaningful things from it is also often a difficult problem. ANN models often overfit to very specific data and perform poorly in more generalized cases. AlphaStar shows this weakness, despite being able to beat professional players, it is only able to do so on a single map when playing a mirror protoss matchup. OpenAI Five also shows this weakness, it was only able to beat professional player when facing a very limited hero pool out of the entire game. This example show how difficult it can be to train a deep learning agent to perform in more generalized situations.

Machine learning agents have shown great success in a variety of different games. However, agents that are too competent also risk making games too difficult for new or casual players. Research has shown that challenge that is too far above a player's skill level will ruin lower player enjoyment. These highly trained agents are likely only desirable against very skilled human players who have many of hours of experience in a given game. Given these factors, highly effective deep learning agents are likely only a desired choice in games that have a large competitive scene, where they can function as an alternative practice option to a skilled human player.

Computer vision focuses on training computers to gain a high-level understanding of digital images or videos. Many computer vision techniques also incorporate forms of machine learning, and have been applied on various video games. This application of computer vision focuses on interpreting game events using visual data. In some cases, artificial intelligence agents have used model-free techniques to learn to play games without any direct connection to internal game logic, solely using video data as input.

Andrej Karpathy has demonstrated that relatively trivial neural network with just one hidden layer is capable of being trained to play based on screen data alone.

(1993) is a first-person shooter (FPS) game. Student researchers from Carnegie Mellon University used computer vision techniques to create an agent that could play the game using only image pixel input from the game. The students used convolutional neural network (CNN) layers to interpret incoming image data and output valid information to a recurrent neural network which was responsible for outputting game moves.

Other uses of vision-based deep learning techniques for playing games have included playing Super Mario Bros. only using image input, using deep Q-learning for training.

Machine learning has seen research for use in content recommendation and generation. Procedural content generation is the process of creating data algorithmically rather than manually. This type of content is used to add replayability to games without relying on constant additions by human developers. PCG has been used in various games for different types of content generation, examples of which include weapons in Borderlands 2, all world layouts in Minecraft and entire universes in . Common approaches to PCG include techniques that involve grammars, search-based algorithms, and logic programming. These approaches require humans to manually define the range of content possible, meaning that a human developer decides what features make up a valid piece of generated content. Machine learning is theoretically capable of learning these features when given examples to train off of, thus greatly reducing the complicated step of developers specifying the details of content design. Machine learning techniques used for content generation include Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNN), Generative Adversarial networks (GAN), and K-means clustering. Not all of these techniques make use of ANNs, but the rapid development of deep learning has greatly increased the potential of techniques that do.

Galactic Arms Race is a space shooter video game that uses neuroevolution powered PCG to generate unique weapons for the player. This game was a finalist in the 2010 Indie Game Challenge and its related research paper won the Best Paper Award at the 2009 IEEE Conference on Computational Intelligence and Games. The developers use a form of neuroevolution called cgNEAT to generate new content based on each player's personal preferences.

Each generated item is represented by a special ANN known as a Compositional Pattern Producing Network (CPPNs). During the evolutionary phase of the game cgNEAT calculates the fitness of current items based on player usage and other gameplay metrics, this fitness score is then used decide which CPPNs will reproduce to create a new item. The ending result is the generation of new weapon effects based on the player's preference.

Super Mario Bros. has been used by several researchers to simulate PCG level creation. Various attempts having used different methods. A version in 2014 used n-grams to generate levels similar to the ones it trained on, which was later improved by making use of MCTS to guide generation. These generations were often not optimal when taking gameplay metrics such as player movement into account, a separate research project in 2017 tried to resolve this problem by generating levels based on player movement using Markov Chains. These projects were not subjected to human testing and may not meet human playability standards.

PCG level creation for The Legend of Zelda has been attempted by researchers at the University of California, Santa Cruz. This attempt made use of a Bayesian Network to learn high level knowledge from existing levels, while Principal Component Analysis (PCA) was used to represent the different low level features of these levels. The researchers used PCA to compare generated levels to human made levels and found that they were considered very similar. This test did not include playability or human testing of the generated levels.

Music is often seen in video games and can be a crucial element for influencing the mood of different situations and story points. Machine learning has seen use in the experimental field of music generation; it is uniquely suited to processing raw unstructured data and forming high level representations that could be applied to the diverse field of music. Most attempted methods have involved the use of ANN in some form. Methods include the use of basic feedforward neural networks, autoencoders, restricted boltzmann machines, recurrent neural networks, convolutional neural networks, generative adversarial networks (GANs), and compound architectures that use multiple methods.

The 2014 research paper on "Variational Recurrent Auto-Encoders" attempted to generate music based on songs from 8 different video games. This project is one of the few conducted purely on video game music. The neural network in the project was able to generate data that was very similar to the data of the games it trained off of. The generated data did not translate into good quality music.

# data[].attributes.attachment_link.description
In video games, various artificial intelligence techniques have been used in a variety of ways, ranging from non-player character (NPC) control to procedural content generation (PCG). Machine learning is a subset of artificial intelligence that focuses on using algorithms and statistical models to make machines act without specific programming.

# data[].attributes.attachment_link.readable_text
First team to destroy a tower wins.

Team with the higher net worth after 7 minutes wins.

Mirror match of 5 fixed heroes utilizing 5 invulnerable couriers. No neutrals, runes, shrines, wards, invisibility, summons, illusions, or Scan. No Divine Rapier, Bottle, Quelling Blade, Boots of Travel, Tome of Knowledge, or Infused Raindrop.

#AI bots just beat humans at the video game Dota 2. That's a big deal, because their victory required teamwork and collaboration - a huge milestone in advancing artificial intelligence.

Drafting from an 18 hero pool, 5 invulnerable couriers. No summons, illusions, or Scan. No Divine Rapier or Bottle.

Handpicked lineup from an 18 hero pool. No summons or illusions. No Divine Rapier or Bottle.

Drafted from a 17 hero pool. No summons or illusions.

You play against [OpenAI Five] and you realize it has a playstyle that is different. It's doing things that you've never done and you've never seen. Sometimes it looks extremely silly. But then again, are you going to be human and be like "Hey, this looks very stupid, this is bad" or [do] you try to take it to next steps, like "Why is it doing this?"

One key learning that we took is how it was allocating resources. It's just allocating resources as efficiently as possible. And then you realize that we're guilty of being stuck in a team dynamic, whereas sometimes we have to be way more flexible. [...] If OpenAI does that dynamic switch at 100%, we maybe went from 5% to 10%? But that is already a difference-we've noticed it.

I don't believe in comparing OpenAI Five to human performance, since it's like comparing the strength we have to hydraulics. Instead of looking at how inhuman and absurd its reaction time is, or how it will never get tired or make the mistakes you'll make as a human, we looked at the patterns it showed moving around the map and allocating resources.

In terms of what OpenAI has done for us and how it influenced our run at TI9, one of the many curious patterns was the buyback and pressure play that happened in most of the games. We had a lot of talks about fighting and pressuring and how it used a different approach from any human in the past. As people, it's about being realistic and learning from the brain of the AI and not the hydraulic strength that machines have.

# data[].attributes.attachment_link.description
At OpenAI, we've used the multiplayer video game Dota 2 as a research platform for general-purpose AI systems. Our Dota 2 AI, called OpenAI Five, learned by playing over 10,000 years of games against itself. It demonstrated the ability to achieve expert-level performance, learn human-AI cooperation, and

# data[].attributes.attachment_link.readable_text
Long short-term memory ( LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points (such as images), but also entire sequences of data (such as speech or video). For example, LSTM is applicable to tasks such as unsegmented, connected handwriting recognition, speech recognition and anomaly detection in network traffic or IDSs (intrusion detection systems).

A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell.

LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series. LSTMs were developed to deal with the vanishing gradient problem that can be encountered when training traditional RNNs. Relative insensitivity to gap length is an advantage of LSTM over RNNs, hidden Markov models and other sequence learning methods in numerous applications.[citation needed ]

The advantage of an LSTM cell compared to a common recurrent unit is its cell memory unit. The cell vector has the ability to encapsulate the notion of forgetting part of its previously stored memory, as well as to add part of the new information. To illustrate this, one has to inspect the equations of the cell and the way it processes sequences under the hood.

1995- 1997: LSTM was proposed by Sepp Hochreiter and Jürgen Schmidhuber. By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.

1999: Felix Gers and his advisor Jürgen Schmidhuber and Fred Cummins introduced the forget gate (also called "keep gate") into LSTM architecture, enabling the LSTM to reset its own state.

2000: Gers & Schmidhuber & Cummins added peephole connections (connections from the cell to the gates) into the architecture. Additionally, the output activation function was omitted.

2009: An LSTM based model won the ICDAR connected handwriting recognition competition. Three such models were submitted by a team lead by Alex Graves. One was the most accurate model in the competition and another was the fastest.

2013: LSTM networks were a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.

2014: Kyunghyun Cho et al. put forward a simplified variant called Gated recurrent unit (GRU).

2015: Google started using an LSTM for speech recognition on Google Voice. According to the official blog post, the new model cut transcription errors by 49%.

2016: Google started using an LSTM to suggest messages in the Allo conversation app. In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.

Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype in the iPhone and for Siri.

Amazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.

2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.

Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference. Their study describes a novel neural network that performs better on certain data sets than the widely used long short-term memory neural network.

Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".

2019: Researchers from the University of Waterloo proposed a related RNN architecture which represents continuous windows of time. It was derived using the Legendre polynomials and outperforms the LSTM on some memory-related benchmarks.

An LSTM model climbed to third place on the in Large Text Compression Benchmark.

In theory, classic (or "vanilla") RNNs can keep track of arbitrary long-term dependencies in the input sequences. The problem with vanilla RNNs is computational (or practical) in nature: when training a vanilla RNN using back-propagation, the gradients which are back-propagated can "vanish" (that is, they can tend to zero) or "explode" (that is, they can tend to infinity), because of the computations involved in the process, which use finite-precision numbers. RNNs using LSTM units partially solve the vanishing gradient problem, because LSTM units allow gradients to also flow unchanged. However, LSTM networks can still suffer from the exploding gradient problem.

There are several architectures of LSTM units. A common architecture is composed of a cell (the memory part of the LSTM unit) and three "regulators", usually called gates, of the flow of information inside the LSTM unit: an input gate, an output gate and a forget gate. Some variations of the LSTM unit do not have one or more of these gates or maybe have other gates. For example, gated recurrent units (GRUs) do not have an output gate.

Intuitively, the cell is responsible for keeping track of the dependencies between the elements in the input sequence. The input gate controls the extent to which a new value flows into the cell, the forget gate controls the extent to which a value remains in the cell and the output gate controls the extent to which the value in the cell is used to compute the output activation of the LSTM unit. The activation function of the LSTM gates is often the logistic sigmoid function.

There are connections into and out of the LSTM gates, a few of which are recurrent. The weights of these connections, which need to be learned during training, determine how the gates operate.

In the equations below, the lowercase variables represent vectors. Matrices and contain, respectively, the weights of the input and recurrent connections, where the subscript can either be the input gate , output gate , the forget gate or the memory cell , depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, is not just one cell of one LSTM unit, but contains LSTM unit's cells.

The compact forms of the equations for the forward pass of an LSTM unit with a forget gate are:

where the initial values are and and the operator denotes the Hadamard product (element-wise product). The subscript indexes the time step.

: input vector to the LSTM unit

: forget gate's activation vector

: input/update gate's activation vector

: output gate's activation vector

: hidden state vector also known as output vector of the LSTM unit

: cell input activation vector

: cell state vector

, and : weight matrices and bias vector parameters which need to be learned during training

where the superscripts and refer to the number of input features and number of hidden units, respectively.

The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM).[36][37] Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state.[38] is not used, is used instead in most places.

Peephole convolutional LSTM.[39] The denotes the convolution operator.

A RNN using LSTM units can be trained in a supervised fashion, on a set of training sequences, using an optimization algorithm, like gradient descent, combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.

A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to if the spectral radius of is smaller than 1.[40][41]

However, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.

Many applications use stacks of LSTM RNNs and train them by connectionist temporal classification (CTC) to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

Sometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution or by policy gradient methods, especially when there is no "teacher" (that is, training labels).

There have been several successful stories of training, in a non-supervised fashion, RNNs with LSTM units.

In 2018, Bill Gates called it a "huge milestone in advancing artificial intelligence" when bots developed by OpenAI were able to beat humans in the game of Dota 2. OpenAI Five consists of five independent but coordinated neural networks. Each network is trained by a policy gradient method without supervising teacher and contains a single-layer, 1024-unit Long-Short-Term-Memory that sees the current game state and emits actions through several possible action heads.

In 2018, OpenAI also trained a similar LSTM by policy gradients to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.

In 2019, DeepMind's program AlphaStar used a deep LSTM core to excel at the complex video game Starcraft II. This was viewed as significant progress towards Artificial General Intelligence.

Applications of LSTM include:

# data[].attributes.attachment_link.description
Long short-term memory ( LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points (such as images), but also entire sequences of data (such as speech or video).

# data[].attributes.attachment_link.readable_text
Written by Laura E. Shummon Maass and Andy Luc

Most people probably imagine that the majority of games released in the last couple of years have highly sophisticated A.I. for any non-player controlled characters, creatures, or animals (often referred to as bots in this article). However, many video game developers are hesitant to build advanced A.I. into their games in fear of losing control of the overall player experience. In fact, the goal of A.I. in video games is not to create an unbeatable entity for players to struggle against, but is instead meant to maximize player participation and enjoyment over long periods of time.

If you picked up a new game and started playing, would you have more enjoyment being completely destroyed over and over? Or would you rather be paired against someone/something more your level so that you could learn and improve over time? The vast majority of players would likely choose the later. That doesn't mean there is no place for A.I. in the modern gaming industry, it just means that it's purpose is different from what we would initially expect. We don't want to create the best possible A.I., we want to create the most enjoyable A.I. for players to interact/compete with.

A.I. is a very broad term. It doesn't necessarily need to be a model that learns from a player's actions. Wolfenstein 3D was released back in 1992, and even the soldiers in that game had a rudimentary form of A.I. The F inite State Machine (FSM) algorithm is a relatively simple A.I. where designers create a list of all possible events a bot can experience. The designers then assign specific responses the bot would have to each situation (Lou, 2017). We can imagine in 1992 the developers of Wolfenstein 3D considered all of the possible situations an enemy soldier could experience. A door could open in their view, Blazkowicz (the hero of the Wolfenstein series) could walk into view, they could get shot from the back, they could lose sight of Blazkowicz, etc. The developers would compile this list and, for each situation, they would tell the bot what it should do. Here is a helpful visual from Lou's article:

This is obviously a simplistic example. We can imagine it getting much more complex the more detail we build in. How long should the bot search for Blazkowicz before giving up? When/if they give up, should they stay where they are or go back to their spawning point? The list can quickly become very long and quite detailed. In FSM, each situation would be assigned a specific action by the developers creating the game.

The FSM algorithm is not feasible to use in every game. Imagine using FSM in a strategy game for example. If a bot were pre-programmed to respond the same way every time, the player would quickly learn how to outsmart the computer. This creates a repetitive gaming experience which, as you might expect, would not be enjoyable for the player. The M onte Carlo Search Tree (MCST) algorithm was created to prevent the repeatability aspect of FSM. The way MCST works is by first visualizing all of the possible moves a bot has available to it currently. Then, for each of those possible moves, it analyzes all of the moves a player could respond with, then it considers all of it's possible responding moves it could make in response, etc. (Lou, 2017). You can visualize how quickly this tree would become massive in size. Here is a good chart to visualize how MCST works:

Figure 2 highlights the process that a computer using MCST goes through before making a move against a human component. It first looks at all of the options it has, in the above example these options are to either defend, build technology, or attack. Then it builds out a tree that predicts the likelihood of success for each potential move thereafter. Above we can see that the option with the highest likelihood of success is 'attack' (because dark red equals a higher probability of reward), therefore the computer chooses to attack. When the player makes their next move the computer repeats the tree building process all over again.

Imagine a game like Civilization where there are a huge amount of choices a computer could make. The computer would take an extremely long time if it were to build out a detailed tree for every possible choice and every scenario possible for the entire game. It would never make a move. So, to avoid this huge calculation, the MCST algorithm will randomly select a handful of possible options and build out the trees for only the ones selected. That way, the calculation is much quicker and the computer can analyze which selected option would have the highest likelihood of reward.

One of the more popular forms of advanced A.I. in video games recently is the Alien from Creative Assembly's Alien: Isolation. There are some misunderstandings about how the A.I. works behind the scenes. However, it is a remarkable display of the ways in which A.I. can be used to create an engaging and unpredictable environment for the player.

The alien in Alien: Isolation has two driving A.I. forces controlling its movements and behavior: Director A.I. and Alien A.I. The Director A.I. is a passive controller that is in charge of creating an enjoyable player experience. In order to achieve this, the Director A.I. knows where both the player and alien are at all times. However, it does not share this knowledge with the alien. The Director A.I. keeps an eye on what is referred to as the Menace Gauge, which is essentially just a measure of expected player stress levels determined by a multitude of factors such as the alien's proximity to the player, the amount of time the alien spends near the player, the amount of time spent in sight of the player, the amount of time spent visible on the motion tracker device, etc. This Menace Gauge informs the alien's Job Systems, which is essentially just a task tracker for the alien. If the Menace Gauge reaches a certain level, the priority of the task "search new location zone" will grow until the alien moves away from the player into a separate area.
Behavioral Decision Trees:
Before diving into how the Alien A.I. works in action, it is important to first highlight the structure that informs the decision-making process. The Alien A.I. uses an extensive behavioral decision tree with over 100 nodes and 30 selector nodes. Imagine the simple example below:

The way a behavior tree works is by asking questions from left to right. A success allows progression along the tree, while a fail would revert back to the sequence node. Here is the process: Sequence -> Am I Hungry?(success) -> Sequence(running) -> Do I Have Food?(success) -> Sequence(running) -> No Enemies Around(success) -> Sequence(running) -> Eat Food(success) -> Sequence(success) -> Parent Node (Simpson, 2014). If, at any point, one of the nodes returns a (fail), the entire sequence would fail. For example, if it turned out that "Do I Have Food?" failed, it would not check to see if there were any enemies around and it would not eat the food. Instead, the sequence would fail and that would be the end of that sequence.

Sequences can obviously get much more complex, and become multilayered in depth. Here is a deeper example:

Remember, when a sequence either succeeds or fails it returns the result to it's Parent node. In the example above, let's assume that we have succeeded in approaching the door, but failed to open the door as it was locked and we had no key. The sequence node was marked as a fail. As a result, the behavior tree path reverted to the parent node of that sequence. Here is what this parent node might look like:

So, we have failed at opening the door, but we haven't given up yet. Our parent node has another sequence for us to try. This time it involves entering through a window instead. The Alien A.I. has 30 different selector nodes and 100 total nodes, so it is exponentially more complicated than this example, but I hope this gives you an idea about how the Alien A.I. works under the hood.
Back to Alien A.I.
As we know, the Alien A.I. is the system that controls the alien's actions. It is never provided information about the player's location. The only information it receives from the Director A.I. is which general area it should search. Beyond that, it must find the player on its own. It does have some tools to use that help it in hunting down the player. The first is the Sensor System which allows the alien to pick up on audio and visual cues in the environment. Noises such as footsteps, gunshots, the opening of doors, even the beeping of the motion tracker, all of these help the alien to track down the player. The audio range depends on the type of noise that was created. In addition to audio sensors, the alien can also pick up on visual sensors such as glimpsing Ripley running past, or seeing a door open in view, etc.

Another tool the alien has to hunt down the player is the Searching System. There are specific areas that the developers have determined to be good hiding spots that the alien is pre-programmed to search. However, it does not search them in any particular order, and will even double check areas that have already been visited. Of course, if the alien hears a noise or sees a visual cue, it will search an area that the developers have not specifically outlined.

The most commonly discussed topic about Alien: Isolation is how the alien seems to learn more about the player as the game progresses. The actions it makes seem to become more complex as it learns certain traits about a player's play style. Surprisingly to some, the way the developers achieved this was not by building in a complex neural network into the alien's A.I. system. To show how the game accomplishes this sense of alien learning we need to refer back to the Alien A.I.'s behavioral decision tree.

At the start of the game, there are sections of this behavioral tree that are blocked out to the alien. The areas that are blocked out are inaccessible to the alien, meaning it cannot access certain behaviors and actions. For example, at the start of the game the section of the tree that responds to the sound of a door opening in the distance may not be active. If a player opens a door in the alien's view it could unlock that section of the behavioral tree so that, in the future, the sound of a door opening will trigger a response. As the player progresses through the game, more and more of the alien's behavioral tree is unlocked. This gives the illusion that the alien is learning and adapting to the player's gaming style.

This article would not be complete without at least some mention of neural networks being applied to video games. There are some very famous recent examples, one being the A.I. that beat a professional Dota 2 team. However, the best way to cover this topic is to start small and build a basic understanding of how a neural network learns the goals and strategies of a video game.

The game we will use for this basic understanding purpose is Snake. For those of you unfamiliar, Snake is a 2D game where you control a line of squares (referred to as a snake). You have three choices for movement: left, right, or straight ahead. If you hit a wall or run into your tail you die instantly and restart. There is a dot for you to collect (referred to as food) that will grow your tail by one square. So the more you eat, the longer you become.

Let's imagine we want to teach our snake how to get as high of a score as possible. For our snake to survive in this world it needs to learn a few things. For our snake to learn it needs to be provided information about the environment. We will refer to this information we provide it as inputs. These inputs can be anything that we have information about. For example, our inputs could be the following 6 Yes/No questions: is it clear straight, is it clear left, is it clear right, is food straight, is food left, is food right (Designing AI, 2017). This would provide 6 input nodes with 1 or 0 depending on the answer to each question. However, these inputs could also be measures of the distance between the head of the snake and the wall, or it's tail, or the food. For simplicity, let's stay at our 6 input nodes example.

The next thing we need to tell our snake is what we want it to achieve. To communicate our desired goal, we implement a reward system. For example, we might give our snake 1 point every time it moves 1 step towards the food, and maybe 10 points every time it eats the food and grows in length. However, when Binggeser (Designing AI, 2017) implemented these rewards for his snake, he realized his snake would only move in a very small circle. This way his snake was able to rack up points while avoiding the hazards posed by walls and a long tail. Obviously, this wasn't the intended result. There needed to be some type of punishment built into the initial model that removed points whenever the snake moved away from the food. This encouraged the snake to move primarily in the direction of the food.

So now we have a snake that has information from the environment and a reward system that defines what its goal is. Where do we go from here? How does our snake actually learn how to play the game? At this point, it will be helpful to give a quick run through of how a neural network actually works.
Generational Neural Network
A generational neural network is structured the same way as a standard neural network. It starts off with a certain number of input nodes, which then feed into one or more hidden layers, eventually providing an output. Here is a good example visual:

For our snake example, we would have 6 input nodes which are the 6 Yes/No questions we defined earlier: is it clear straight, is it clear left, is it clear right, is food straight, is food left, is food right. Each input node connects to each of the first hidden layer nodes through what we refer to as weights. In figure 8 we see all of the lines (weights) connecting to each of the nodes. These weights are what our model will be adjusting as it learns which inputs to strengthen or weaken to provide the most accurate outputs. In our case, "the most accurate output" is defined as "the snake that collects the highest amount of points". Remember, our snake receives points for moving towards the food, it receives even more points for eating the food, and it receives negative points for moving away from the food.

The way a generational neural network 'learns' is by first deciding the size of each generation (let's say we want each generation to contain 200 snakes). Next, it creates micro-variations in the weights for each of the 200 snakes in the first generation, then it runs each of the 200 snakes in the first generation and selects the most successful snakes (the snakes that received the most points). Let's say we pick the top 10 snakes (top 5%) that received the most points in our first generation. These 10 snakes then become the 'parents' of the second generation. The weights of these 10 snakes are used to define the second generation's starting point. The second generation of 200 snakes will again create micro-variations in these weights and the top performers will be selected as 'parents' of the third generation, and so on.
Back to Snake
So, as we saw above, we can run our first generation snake model over and over (above we ran it 200 times) to see a wide range of variations the snake comes up with through changing each of the weights slightly. We then select the top performers that will go on to influence the neural network weightings in the second generation. We repeat this process for every subsequent generation until the snake's learning rate begins to level out (in other words, until the generational improvement slows or stops).

Maybe in the first, second, and third generations none of the snakes ever ate a piece of food, and therefore never learned that food rewards 10 points. However, maybe on the fourth generation one snake eats a piece of food. This snake will likely have the highest amount of points from its generation and will, therefore, be selected to influence future generations. The weights of future generations will be altered based on the most successful snake ancestors. After 10, 100, or even 1,000 generations, you can imagine how much learning will take place.

The same type of reinforcement learning that is being used in the video game industry is also being successfully applied to other industries. For example, the Grand Theft Auto games, which have pre-programmed "traffic rules, roads, and car physics" (Luzgin, 2018) have been used to provide a safe and realistic environment for testing self-driving car algorithms. Not only is it safe and realistic, but it is also up to 1,000 times faster to collect data in a virtual environment compared to the real world (Luzgin, 2018).

"Video games are a great way of training AI algorithms because they are designed to give human minds gradual progression into harder and harder challenges." (Luzgin, 2018)

One of the latest advancements of A.I. in video games was made by researchers at Open AI. Open AI created a game based on an algorithm whose sole purpose was simply to explore with a sense of natural curiosity. The reward system focused on rewarding exploration over progressing further into the game. The researchers placed this curiosity-driven model into a game of Super Mario Bros. and it successfully passed 11 levels out of pure curiosity. Obviously, there are downsides to this, as it takes immense computing power and the machine can get easily distracted. However, this would also be the same for a human player playing the game for the first time. As Luzgin quoted in his article, "babies appear to employ goal-less exploration to learn skills that will be useful later on in life." This sense of goal-less exploration is continued throughout life, but the most obvious example is in the exploration of virtual environments through video games.

There are many forms of A.I. in use across the video game industry today. Whether it's a simplistic FSM model or an advanced neural network learning from feedback in it's the environment, the possibilities that these virtual environments provide to the advancement of A.I. (both in the gaming industry and beyond) are endless.

Works Cited

Binggeser, Peter. Designing AI: Solving Snake with Evolution. 2017. https://becominghuman.ai/designing-ai-solving-snake-with-evolution-f3dd6a9da867

Comi, Mauro. How to teach AI to play Games: Deep Reinforcement Learning. 2017. https://towardsdatascience.com/how-to-teach-an-ai-to-play-games-deep-reinforcement-learning-28f9b920440a

Lou, Harbing. AI in Video Games: Toward a More Intelligent Game. 2017. http://sitn.hms.harvard.edu/flash/2017/ai-video-games-toward-intelligent-game/

Luzgin, Roman. Video Games as a Perfect Playground for Artificial Intelligence. 2018. https://towardsdatascience.com/video-games-as-a-perfect-playground-for-artificial-intelligence-3b4ebeea36ce

Simpson, Chris. Behavior Trees for AI: How They Work. 2014. https://www.gamasutra.com/blogs/ChrisSimpson/20140717/221339/Behavior_trees_for_AI_How_they_work.php

Thompson, Tommy. The Perfect Organism: The AI of Alien: Isolation. 2017. https://www.gamasutra.com/blogs/TommyThompson/20171031/308027/The_Perfect_Organism_The_AI_of_Alien_Isolation.php

# data[].attributes.attachment_link.description
Written by Laura E. Shummon Maass and Andy Luc Most people probably imagine that the majority of games released in the last couple of years have highly sophisticated A.I. for any non-player controlled characters, creatures, or animals (often referred to as bots in this article).

# data[].attributes.attachment_link.readable_text
A paper on today's Artificial Intelligence progress in Video-Games

Fig. 0 - Black and White circles placed randomly

It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable
John McCarthy

In short, A.I. is about making a computer thinking by itself, without needing any particular help. The range goes from deducting your Google Search sentence, to predicting the Trajectory of a Comet passing by the Moon in 57 years.

Even though both these examples are different, they both relyon the same techniques: Learning. These two A.I.s learn by themselves, they use inputs to calculate outputs, i.e. Google Search knows that 78% of people starting their query with "fac" will be looking for Facebook, hence it shows "Facebook" as a possible search. The same applies to the A.I. that calculates Comet Trajectory: it relies on maths and algorithms, as well as patterns from previous Comets. Learning.

But there are several kinds of Learning, A.I.-wise: Machine Learning, Deep Learning,to name a few. How are they different ? Well, they are all sub-types of Machine Learning (ML). They all depend on training data, they all need to train and to improve by failing and succeeding, and trying to give the best output possible. The goal here is to show the A.I. possible outputs- or not, there's also a type of learning that relies on not knowing the outputs, Unsupervised Learning - , and to let the A.I. train. How the A.I. trains is where the differences lie.

Deep Learning, however, is based on Artificial Neural Networks (ANN), which are representations of our Neural Networks, that examine layers of the input to determine an output. In other words, lots of Neurons giving information to each others, based on layers of the input - for an image, the ANN will split the image in several layers, such as the colours, the shapes, if there's something looking like a tail or pointy ears, and so on - to determine an output: if this image is Cat or Dog. Deep Learning tends to suffice to itself, meaning that it has to train, to learn "through example" a lot before having reliable outputs.

We will see that, through the years, video-game A.I. became more and more intelligent, less and less scripted.

# data[].attributes.attachment_link.title
Evolution of AI in Video-Games from 1980 to Today

# data[].attributes.attachment_link.description
It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable John McCarthy In short, A.I.

# data[].attributes.attachment_link.provider_name
Civilization V | News | The Greatest Game of Civilization V No One Played

# data[].attributes.attachment_link.title
Civilization V | News | The Greatest Game of Civilization V No One Played

# data[].attributes.attachment_link.description
The story of an AI-only world domination game pitting 42 civilizations against one another

# data[].attributes.attachment_link.title
This Guy Made a Machine Learning RuneScape PVP Bot

# data[].attributes.attachment_link.readable_text
is an artificial intelligence research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAI Inc. The company, considered a competitor to DeepMind, conducts research in the field of artificial intelligence (AI) with the stated goal of promoting and developing friendly AI in a way that benefits humanity as a whole. The organization was founded in San Francisco in late 2015 by Elon Musk, Sam Altman, and others, who collectively pledged US$1 billion. Musk resigned from the board in February 2018 but remained a donor. In 2019, OpenAI LP received a US$1 billion investment from Microsoft.

In October 2015, Elon Musk, Sam Altman, and other investors announced the formation of OpenAI and pledged over US$1 billion to the venture. The organization stated they would "freely collaborate" with other institutions and researchers by making its patents and research open to the public.

On April 27, 2016, OpenAI released a public beta of "OpenAI Gym", its platform for reinforcement learning research.

On December 5, 2016, OpenAI released Universe, a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications.

On February 21, 2018, Musk resigned his board seat, citing "a potential future conflict (of interest)" with Tesla AI development for self driving cars, but remained a donor.

As of 2020, OpenAI is headquartered in San Francisco's Mission District, and shares the former Pioneer Trunk Factory building with Neuralink, another company co-founded by Musk.

Following OpenAI's transition from non-profit to for-profit the company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment package of US$ 1 billion into the company. OpenAI then announced its intention to commercially license its technologies, with Microsoft as its preferred partner.

In June 2020, OpenAI announced GPT-3, a language model trained on trillions of words from the Internet. It also announced that an associated API, named simply "the API", would form the heart of its first commercial product. GPT-3 is aimed at natural language answering of questions, but it can also translate between languages and coherently generate improvised text.

Other backers of the project include:

Companies:

The group started in early January 2016 with nine researchers. According to , Brockman met with Yoshua Bengio, one of the "founding fathers" of the deep learning movement, and drew up a list of the "best researchers in the field". Microsoft's Peter Lee stated that the cost of a top AI researcher exceeds the cost of a top NFL quarterback prospect. While OpenAI pays corporate-level (rather than nonprofit-level) salaries, it doesn't currently pay AI researchers salaries comparable to those of Facebook or Google. Nevertheless, Sutskever stated that he was willing to leave Google for OpenAI "partly of because of the very strong group of people and, to a very large extent, because of its mission." Brockman stated that "the best thing that I could imagine doing was moving humanity closer to building real AI in a safe way." OpenAI researcher Wojciech Zaremba stated that he turned down "borderline crazy" offers of two to three times his market value to join OpenAI instead.

Some scientists, such as Stephen Hawking and Stuart Russell, have articulated concerns that if advanced AI someday gains the ability to re-design itself at an ever-increasing rate, an unstoppable " intelligence explosion" could lead to human extinction. Musk characterizes AI as humanity's "biggest existential threat." OpenAI's founders structured it as a non-profit so that they could focus its research on creating a positive long-term human impact.

Musk and Altman have stated they are motivated in part by concerns about the existential risk from artificial general intelligence. OpenAI states that "it's hard to fathom how much human-level AI could benefit society," and that it is equally difficult to comprehend "how much it could damage society if built or used incorrectly". Research on safety cannot safely be postponed: "because of AI's surprising history, it's hard to predict when human-level AI might come within reach." OpenAI states that AI "should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible...", and which sentiment has been expressed elsewhere in reference to a potentially enormous class of AI-enabled products: "Are we really willing to let our society be infiltrated by autonomous software and hardware agents whose details of operation are known only to a select few? Of course not." Co-chair Sam Altman expects the decades-long project to surpass human intelligence.

Vishal Sikka, former CEO of Infosys, stated that an "openness" where the endeavor would "produce results generally in the greater interest of humanity" was a fundamental requirement for his support, and that OpenAI "aligns very nicely with our long-held values" and their "endeavor to do purposeful work". Cade Metz of suggests that corporations such as Amazon may be motivated by a desire to use open-source software and data to level the playing field against corporations such as Google and Facebook that own enormous supplies of proprietary data. Altman states that Y Combinator companies will share their data with OpenAI.

In 2019, OpenAI became a for profit company called OpenAI LP to secure additional funding while staying controlled by a non-profit called OpenAI Inc in a structure that OpenAI calls "capped-profit", having previously been a 501(c)(3) nonprofit organization.

Musk posed the question: "what is the best thing we can do to ensure the future is good? We could sit on the sidelines or we can encourage regulatory oversight, or we could participate with the right structure with people who care deeply about developing AI in a way that is safe and is beneficial to humanity." Musk acknowledged that "there is always some risk that in actually trying to advance (friendly) AI we may create the thing we are concerned about"; nonetheless, the best defense is "to empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower."

Musk and Altman's counter-intuitive strategy of trying to reduce the risk that AI will cause overall harm, by giving AI to everyone, is controversial among those who are concerned with existential risk from artificial intelligence. Philosopher Nick Bostrom is skeptical of Musk's approach: "If you have a button that could do bad things to the world, you don't want to give it to everyone." During a 2016 conversation about the technological singularity, Altman said that "we don't plan to release all of our source code" and mentioned a plan to "allow wide swaths of the world to elect representatives to a new governance board". Greg Brockman stated that "Our goal right now... is to do the best thing there is to do. It's a little vague."

Conversely, OpenAI's initial decision to withhold GPT-2 due to a wish to "err on the side of caution" in the presence of potential misuse, has been criticized by advocates of openness. Delip Rao, an expert in text generation, stated "I don't think [OpenAI] spent enough time proving [GPT-2] was actually dangerous." Other critics argued that open publication is necessary to replicate the research and to be able to come up with countermeasures.

In the 2017 tax year, OpenAI spent US$7.9 million, or a quarter of its functional expenses, on cloud computing alone. In comparison, DeepMind's total expenses in 2017 were much larger, measuring US$442 million. In Summer 2018, simply training OpenAI's Dota 2 bots required renting 128,000 CPUs and 256 GPUs from Google for multiple weeks. According to OpenAI, the capped-profit model adopted in March 2019 allows OpenAI LP to legally attract investment from venture funds, and in addition, to grant employees stakes in the company, the goal being that they can say "I'm going to Open AI, but in the long term it's not going to be disadvantageous to us as a family." Many top researchers work for Google Brain, DeepMind, or Facebook, Inc., which offer stock options that a nonprofit would be unable to. In June 2019, OpenAI LP raised a billion dollars from Microsoft, a sum which OpenAI plans to have spent "within five years, and possibly much faster". Altman has stated that even a billion dollars may turn out to be insufficient, and that the lab may ultimately need "more capital than any non-profit has ever raised" to achieve AGI.

The transition from a nonprofit to a capped-profit company was viewed with skepticism by Oren Etzioni of the nonprofit Allen Institute for AI, who agreed that wooing top researchers to a nonprofit is difficult, but stated "I disagree with the notion that a nonprofit can't compete" and pointed to successful low-budget projects by OpenAI and others. "If bigger and better funded was always better, then IBM would still be number one." Following the transition, public disclosure of the compensation of top employees at OpenAI LP is no longer legally required. The nonprofit, OpenAI Inc., is the sole controlling shareholder of OpenAI LP. OpenAI LP, despite being a for-profit company, retains a formal fiduciary responsibility to OpenAI's Inc.'s nonprofit charter. A majority of OpenAI Inc.'s board is barred from having financial stakes in OpenAI LP. In addition, minority members with a stake in OpenAI LP are barred from certain votes due to conflict of interest. Some researchers have argued that OpenAI LP's switch to for-profit status is inconsistent with OpenAI's claims to be "democratizing" AI. A journalist in Vice News wrote that "generally, we've never been able to rely on venture capitalists to better humanity".

OpenAI's research tend to focus on reinforcement learning. OpenAI is viewed as an important competitor to DeepMind.

Gym aims to provide an easy to set up, general-intelligence benchmark with a wide variety of different environments-somewhat akin to, but broader than, the ImageNet Large Scale Visual Recognition Challenge used in supervised learning research-and that hopes to standardize the way in which environments are defined in AI research publications, so that published research becomes more easily reproducible. The project claims to provide the user with a simple interface. As of June2017, Gym can only be used with Python. As of September 2017, the Gym documentation site was not maintained, and active work focused instead on its GitHub page.

In "RoboSumo", virtual humanoid " metalearning" robots initially lack knowledge of how to even walk, and given the goals of learning to move around, and pushing the opposing agent out of the ring. Through this adversarial learning process, the agents learn how to adapt to changing conditions; when an agent is then removed from this virtual environment and placed in a new virtual environment with high winds, the agent braces to remain upright, suggesting it had learned how to balance in a generalized way. OpenAI's Igor Mordatch argues for that competition between agents can create an intelligence "arms race" that can increase an agent's ability to function, even outside the context of the competition.

In 2018, OpenAI launched the Debate Game, which teaches machines to debate toy problems in front of a human judge. The purpose is to research whether such an approach may assist in auditing AI decisions and in developing explainable AI.

Dactyl uses machine learning to train a robot Shadow Hand from scratch, using the same reinforcement learning algorithm code that OpenAI Five uses. The robot hand is trained entirely in physically inaccurate simulation.

The original paper on generative pre-training (GPT) of a language model was written by Alec Radford and colleagues, and published in preprint on OpenAI's website on June 11, 2018. It showed how a generative model of language is able to acquire world knowledge and process long-range dependencies by pre-training on a diverse corpus with long stretches of contiguous text.

Generative Pre-trained Transformer 2, commonly known by its abbreviated form GPT-2, is an unsupervised transformer language model and the successor to GPT. GPT-2 was first announced in February 2019, with only limited demonstrative versions initially released to the public. The full version of GPT-2 was not immediately released out of concern over potential misuse, including applications for writing fake news. Some experts expressed skepticism that GPT-2 posed a significant threat. The Allen Institute for Artificial Intelligence responded to GPT-2 with a tool to detect "neural fake news". Other researchers, such as Jeremy Howard, warned of "the technology to totally fill Twitter, email, and the web up with reasonable-sounding, context-appropriate prose, which would drown out all other speech and be impossible to filter". In November 2019, OpenAI released the complete version of the GPT-2 language model. Several websites host interactive demonstrations of different instances of GPT-2 and other transformer models.

GPT-2's authors argue unsupervised language models to be general-purpose learners, illustrated by GPT-2 achieving state-of-the-art accuracy and perplexity on 7 of 8 zero-shot tasks (i.e. the model was not further trained on any task-specific input-output examples). The corpus it was trained on, called WebText, contains slightly over 8 million documents for a total of 40 GB of text from URLs shared in Reddit submissions with at least 3 upvotes. It avoids certain issues encoding vocabulary with word tokens by using byte pair encoding. This allows to represent any string of characters by encoding both individual characters and multiple-character tokens.

Generative Pre-trained Transformer 3, commonly known by its abbreviated form GPT-3, is an unsupervised Transformer language model and the successor to GPT-2. It was first described in May 2020. OpenAI stated that full version of GPT-3 contains 175 billion parameters, two orders of magnitude larger than the 1.5 billion parameters in the full version of GPT-2 (although GPT-3 models with as few as 125 million parameters were also trained).

OpenAI stated that GPT-3 succeeds at certain " meta-learning" tasks. It can generalize the purpose of a single input-output pair. The paper gives an example of translation and cross-linguistic transfer learning between English and Romanian, and between English and German.

GPT-3 dramatically improved benchmark results over GPT-2. OpenAI cautioned that such scaling up of language models could be approaching or encountering the fundamental capability limitations of predictive language models. Pre-training GPT-3 required several thousand petaflop/s-days of compute, compared to tens of petaflop/s-days for the full GPT-2 model. Like that of its predecessor, GPT-3's fully trained model was not immediately released to the public on the grounds of possible abuse, though OpenAI planned to allow access through a paid cloud API after a two-month free private beta that began in June 2020.

On September 23, 2020, GPT-3 was licensed exclusively to Microsoft.

OpenAI's MuseNet (2019) is a deep neural net trained to predict subsequent musical notes in MIDI music files. It can generate songs with ten different instruments in fifteen different styles. According to , a song generated by MuseNet tends to start out reasonably but then fall into chaos the longer it plays.

OpenAI's Jukebox (2020) is an open-sourced algorithm to generate music with vocals. After training on 1.2 million samples, the system accepts a genre, artist, and a snippet of lyrics, and outputs song samples. OpenAI stated the songs "show local musical coherence, follow traditional chord patterns" but acknowledged that the songs lack "familiar larger musical structures such as choruses that repeat" and that "there is a significant gap" between Jukebox and human-generated music. The Verge stated "It's technologically impressive, even if the results sound like mushy versions of songs that might feel familiar", while Business Insider stated "surprisingly, some of the resulting songs are catchy and sound legitimate".

In June 2020, OpenAI announced a multi-purpose API which it said was "for accessing new AI models developed by OpenAI" to let developers call on it for "any English language AI task."

OpenAI Five is the name of a team of five OpenAI-curated bots that are used in the competitive five-on-five video game , who learn to play against human players at a high skill level entirely through trial-and-error algorithms. Before becoming a team of five, the first public demonstration occurred at The International 2017, the annual premiere championship tournament for the game, where Dendi, a professional Ukrainian player, lost against a bot in a live 1v1 matchup. After the match, CTO Greg Brockman explained that the bot had learned by playing against itself for two weeks of real time, and that the learning software was a step in the direction of creating software that can handle complex tasks like a surgeon. The system uses a form of reinforcement learning, as the bots learn over time by playing against themselves hundreds of times a day for months, and are rewarded for actions such as killing an enemy and taking map objectives.

By June 2018, the ability of the bots expanded to play together as a full team of five and they were able to defeat teams of amateur and semi-professional players. At The International 2018, OpenAI Five played in two exhibition matches against professional players, but ended up losing both games. In April 2019, OpenAI Five defeated OG, the reigning world champions of the game at the time, 2:0 in a live exhibition match in San Francisco. The bots' final public appearance came later that month, where they played in 42,729 total games in a four-day open online competition, winning a percentage of 99.4% of those games.

Gym Retro is a platform for reinforcement learning research on games. Gym Retro is used to conduct research on RL algorithms and study generalization. Prior research in RL has mostly focused on optimizing agents to solve single tasks. Gym Retro gives the ability to generalize between games with similar concepts but different appearances.

1. ^ Vincent, James (February 21, 2018). "Elon Musk leaves board of AI safety group to avoid conflict of interest with Tesla". The Verge.

2. ^ Vance, Ashlee (2020-06-11). "Trillions of Words Analyzed, OpenAI Sets Loose AI Language Colossus". Bloomberg News.

3. ^ Mendoza, Jessica. "Tech leaders launch nonprofit to save the world from killer robots". The Christian Science Monitor.

4. Metz, Cade (15 December 2015). "Elon Musk's Billion-Dollar AI Plan Is About Far More Than Saving the World". . Altman said they expect this decades-long project to surpass human intelligence.

5. ^ Levy, Steven (December 11, 2015). "How Elon Musk and Y Combinator Plan to Stop Computers From Taking Over". Medium/Backchannel. Elon Musk: ...we came to the conclusion that having a 501(c)(3)... would probably be a good thing to do

6. ^ Brockman, Greg (April 3, 2017). "Yes, we're a 501(c)(3). As you mention in /r/ControlProblem, we will file our 990 later this year as required. Not yet sure of exact date".

7. ^ Ganesh, Prakhar (December 17, 2019). "Pre-trained Language Models: Simplified". The intuition behind pre-trained language models is to create a black box which understands the language and can then be asked to do any specific task in that language.

8. ^ Brown, Tom; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini (2020-06-01). "Language Models are Few-Shot Learners". p. appendix. arXiv: [cs.CL].

9. ^ Brown, Tom; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini (2020-06-01). "Language Models are Few-Shot Learners". arXiv: [cs.CL]. Since we increase the capacity by over two orders of magnitude from GPT-2 to GPT-3

10. ^ Amodei, Dario; Hernandez, Danny (May 16, 2018). "AI and Compute". A petaflop/s-day (pfs-day) consists of performing 10¹⁵ neural net operations per second for one day, or a total of about 10²⁰ operations. The compute-time product serves as a mental convenience, similar to kW-hr for energy.

11. ^ "TechCrunch - Startup and Technology News". TechCrunch. If you've ever wanted to try out OpenAI's vaunted machine learning toolset, it just got a lot easier. The company has released an API that lets developers call its AI tools in on "virtually any English language task."

# data[].attributes.attachment_link.description
is an artificial intelligence research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAI Inc. The company, considered a competitor to DeepMind, conducts research in the field of artificial intelligence (AI) with the stated goal of promoting and developing friendly AI in a way that benefits humanity as a whole.

# data[].attributes.attachment_link.readable_text
In video games, artificial intelligence (AI) is used to generate responsive, adaptive or intelligent behaviors primarily in non-player characters (NPCs) similar to human-like intelligence. Artificial intelligence has been an integral part of video games since their inception in the 1950s. AI in video games is a distinct subfield and differs from academic AI. It serves to improve the game-player experience rather than machine learning or decision making. During the golden age of arcade video games the idea of AI opponents was largely popularized in the form of graduated difficulty levels, distinct movement patterns, and in-game events dependent on the player's input. Modern games often implement existing techniques such as pathfinding and decision trees to guide the actions of NPCs. AI is often used in mechanisms which are not immediately visible to the user, such as data mining and procedural-content generation.

However, "game AI" does not, in general, as might be thought and sometimes is depicted to be the case, mean a realization of an artificial person corresponding to an NPC, in the manner of say, the Turing test or an artificial general intelligence.

The term "game AI" is used to refer to a broad set of algorithms that also include techniques from control theory, robotics, computer graphics and computer science in general, and so video game AI may often not constitute "true AI" in that such techniques do not necessarily facilitate computer learning or other standard criteria, only constituting "automated computation" or a predetermined and limited set of responses to a predetermined and limited set of inputs.

Many industries and corporate voices claim that so-called video game AI has come a long way in the sense that it has revolutionized the way humans interact with all forms of technology, although many expert researchers are skeptical of such claims, and particularly of the notion that such technologies fit the definition of "intelligence" standardly used in the cognitive sciences. Industry voices make the argument that AI has become more versatile in the way we use all technological devices for more than their intended purpose because the AI allows the technology to operate in multiple ways, allegedly developing their own personalities and carrying out complex instructions of the user.

However, many in the field of AI have argued that video game AI is not true intelligence, but an advertising buzzword used to describe computer programs that use simple sorting and matching algorithms to create the illusion of intelligent behavior while bestowing software with a misleading aura of scientific or technological complexity and advancement. Since game AI for NPCs is centered on appearance of intelligence and good gameplay within environment restrictions, its approach is very different from that of traditional AI.

Game playing was an area of research in AI from its inception. One of the first examples of AI is the computerized game of Nim made in 1951 and published in 1952. Despite being advanced technology in the year it was made, 20 years before Pong, the game took the form of a relatively small box and was able to regularly win games even against highly skilled players of the game. In 1951, using the Ferranti Mark 1 machine of the University of Manchester, Christopher Strachey wrote a checkers program and Dietrich Prinz wrote one for chess. These were among the first computer programs ever written. Arthur Samuel's checkers program, developed in the middle 50s and early 60s, eventually achieved sufficient skill to challenge a respectable amateur. Work on checkers and chess would culminate in the defeat of Garry Kasparov by IBM's Deep Blue computer in 1997. The first video games developed in the 1960s and early 1970s, like , , and (1973), were games implemented on discrete logic and strictly based on the competition of two players, without AI.

Games that featured a single player mode with enemies started appearing in the 1970s. The first notable ones for the arcade appeared in 1974: the Taito game ( racing video game) and the Atari games (duck hunting light gun shooter) and ( fighter aircraft dogfighting simulator). Two text-based computer games from 1972, Hunt the Wumpus and , also had enemies. Enemy movement was based on stored patterns. The incorporation of microprocessors would allow more computation and random elements overlaid into movement patterns.

It was during the golden age of video arcade games that the idea of AI opponents was largely popularized, due to the success of Space Invaders (1978), which sported an increasing difficulty level, distinct movement patterns, and in-game events dependent on hash functions based on the player's input. (1979) added more complex and varied enemy movements, including maneuvers by individual enemies who break out of formation. (1980) introduced AI patterns to maze games, with the added quirk of different personalities for each enemy. Karate Champ (1984) later introduced AI patterns to fighting games, although the poor AI prompted the release of a second version. First Queen (1988) was a tactical action RPG which featured characters that can be controlled by the computer's AI in following the leader. The role-playing video game Dragon Quest IV (1990) introduced a "Tactics" system, where the user can adjust the AI routines of non-player characters during battle, a concept later introduced to the action role-playing game genre by Secret of Mana (1993).

Games like Madden Football, Earl Weaver Baseball and Tony La Russa Baseball all based their AI in an attempt to duplicate on the computer the coaching or managerial style of the selected celebrity. Madden, Weaver and La Russa all did extensive work with these game development teams to maximize the accuracy of the games. Later sports titles allowed users to "tune" variables in the AI to produce a player-defined managerial or coaching strategy.

The emergence of new game genres in the 1990s prompted the use of formal AI tools like finite state machines. Real-time strategy games taxed the AI with many objects, incomplete information, pathfinding problems, real-time decisions and economic planning, among other things. The first games of the genre had notorious problems. Herzog Zwei (1989), for example, had almost broken pathfinding and very basic three-state state machines for unit control, and (1992) attacked the players' base in a beeline and used numerous cheats. Later games in the genre exhibited more sophisticated AI.

Later games have used bottom-up AI methods, such as the emergent behaviour and evaluation of player actions in games like or Black & White. Façade (interactive story) was released in 2005 and used interactive multiple way dialogs and AI as the main aspect of game.

Games have provided an environment for developing artificial intelligence with potential applications beyond gameplay. Examples include Watson, a Jeopardy!-playing computer; and the RoboCup tournament, where robots are trained to compete in soccer.

Many experts complain that the "AI" in the term "game AI" overstates its worth, as game AI is not about intelligence, and shares few of the objectives of the academic field of AI. Whereas "real AI" addresses fields of machine learning, decision making based on arbitrary data input, and even the ultimate goal of strong AI that can reason, "game AI" often consists of a half-dozen rules of thumb, or heuristics, that are just enough to give a good gameplay experience. Historically, academic game-AI projects have been relatively separate from commercial products because the academic approaches tended to be simple and non-scalable. Commercial game AI has developed its own set of tools, which have been sufficient to give good performance in many cases.

Game developers' increasing awareness of academic AI and a growing interest in computer games by the academic community is causing the definition of what counts as AI in a game to become less idiosyncratic. Nevertheless, significant differences between different application domains of AI mean that game AI can still be viewed as a distinct subfield of AI. In particular, the ability to legitimately solve some AI problems in games by cheating creates an important distinction. For example, inferring the position of an unseen object from past observations can be a difficult problem when AI is applied to robotics, but in a computer game a NPC can simply look up the position in the game's scene graph. Such cheating can lead to unrealistic behavior and so is not always desirable. But its possibility serves to distinguish game AI and leads to new problems to solve, such as when and how to use cheating.

The major limitation to strong AI is the inherent depth of thinking and the extreme complexity of the decision making process. This means that although it would be then theoretically possible to make "smart" AI the problem would take considerable processing power.

Game AI/heuristic algorithms are used in a wide variety of quite disparate fields inside a game. The most obvious is in the control of any NPCs in the game, although "scripting" ( decision tree) is currently the most common means of control. These handwritten decision trees often result in "artificial stupidity" such as repetitive behavior, loss of immersion, or abnormal behavior in situations the developers did not plan for.

Pathfinding, another common use for AI, is widely seen in real-time strategy games. Pathfinding is the method for determining how to get a NPC from one point on a map to another, taking into consideration the terrain, obstacles and possibly " fog of war". Commercial videogames often use fast and simple "grid-based pathfinding", wherein the terrain is mapped onto a rigid grid of uniform squares and a pathfinding algorithm such as A* or IDA* is applied to the grid. Instead of just a rigid grid, some games use irregular polygons and assemble a navigation mesh out of the areas of the map that NPCs can walk to. As a third method, it is sometimes convenient for developers to manually select "waypoints" that NPCs should use to navigate; the cost is that such waypoints can create unnatural-looking movement. In addition, waypoints tend to perform worse than navigation meshes in complex environments. Beyond static pathfinding, navigation is a sub-field of Game AI focusing on giving NPCs the capability to navigate in a dynamic environment, finding a path to a target while avoiding collisions with other entities (other NPC, players...) or collaborating with them (group navigation). Navigation in dynamic strategy games with large numbers of units, such as Age of Empires (1997) or Civilization V (2010), often performs poorly; units often get in the way of other units.

Rather than improve the Game AI to properly solve a difficult problem in the virtual environment, it is often more cost-effective to just modify the scenario to be more tractable. If pathfinding gets bogged down over a specific obstacle, a developer may just end up moving or deleting the obstacle. In Half-Life (1998), the pathfinding algorithm sometimes failed to find a reasonable way for all the NPCs to evade a thrown grenade; rather than allow the NPCs to attempt to bumble out of the way and risk appearing stupid, the developers instead scripted the NPCs to crouch down and cover in place in that situation.

Video game combat AI

Many contemporary video games fall under the category of action, first-person shooter, or adventure. In most of these types of games, there is some level of combat that takes place. The AI's ability to be efficient in combat is important in these genres. A common goal today is to make the AI more human or at least appear so.

One of the more positive and efficient features found in modern-day video game AI is the ability to hunt. AI originally reacted in a very black and white manner. If the player were in a specific area then the AI would react in either a complete offensive manner or be entirely defensive. In recent years, the idea of "hunting" has been introduced; in this 'hunting' state the AI will look for realistic markers, such as sounds made by the character or footprints they may have left behind. These developments ultimately allow for a more complex form of play. With this feature, the player can actually consider how to approach or avoid an enemy. This is a feature that is particularly prevalent in the stealth genre.

Another development in recent game AI has been the development of "survival instinct". In-game computers can recognize different objects in an environment and determine whether it is beneficial or detrimental to its survival. Like a user, the AI can look for cover in a firefight before taking actions that would leave it otherwise vulnerable, such as reloading a weapon or throwing a grenade. There can be set markers that tell it when to react in a certain way. For example, if the AI is given a command to check its health throughout a game then further commands can be set so that it reacts a specific way at a certain percentage of health. If the health is below a certain threshold then the AI can be set to run away from the player and avoid it until another function is triggered. Another example could be if the AI notices it is out of bullets, it will find a cover object and hide behind it until it has reloaded. Actions like these make the AI seem more human. However, there is still a need for improvement in this area.

Another side-effect of combat AI occurs when two AI-controlled characters encounter each other; first popularized in the id Software game , so-called 'monster infighting' can break out in certain situations. Specifically, AI agents that are programmed to respond to hostile attacks will sometimes attack each other if their cohort's attacks land too close to them. In the case of Doom, published gameplay manuals even suggest taking advantage of monster infighting in order to survive certain levels and difficulty settings.

Game AI often amounts to pathfinding and finite state machines. Pathfinding gets the AI from point A to point B, usually in the most direct way possible. State machines permit transitioning between different behaviors. The Monte Carlo Tree Search method provides a more engaging game experience by creating additional obstacles for the player to overcome. The MCTS consists of a tree diagram in which the AI essentially plays tic-tac-toe. Depending on the outcome, it selects a pathway yielding the next obstacle for the player. In complex video games, these trees may have more branches, provided that the player can come up with several strategies to surpass the obstacle.

Georgios N. Yannakakis suggests that academic AI developments may play roles in-game AI beyond the traditional paradigm of AI controlling NPC behavior. He highlights four other potential application areas:

1. Player-experience modeling: Discerning the ability and emotional state of the player, so as to tailor the game appropriately. This can include dynamic game difficulty balancing, which consists in adjusting the difficulty in a video game in real-time based on the player's ability. Game AI may also help deduce player intent (such as gesture recognition).

2. Procedural-content generation: Creating elements of the game environment like environmental conditions, levels, and even music in an automated way. AI methods can generate new content or interactive stories.

3. Data mining on user behavior: This allows game designers to explore how people use the game, what parts they play most, and what causes them to stop playing, allowing developers to tune gameplay or improve monetization.

4. Alternate approaches to NPCs: These include changing the game set-up to enhance NPC believability and exploring social rather than individual NPC behavior.

Rather than procedural generation, some researchers have used generative adversarial networks (GANs) to create new content. In 2018 researchers at Cornwall University trained a GAN on a thousand human-created levels for DOOM (1993); following training, the neural net prototype was able to design new playable levels on its own. Similarly, researchers at the University of California prototyped a GAN to generate levels for Super Mario. In 2020 Nvidia displayed a GAN-created clone of ; the GAN learned how to recreate the game by watching 50,000 (mostly bot-generated) playthroughs.

Dani Bunten was once asked how to play-balance a game. Her one word answer was "Cheat." Asked what to do if gamers complained, she said, "Lie!"

Gamers always ask if the AI cheats (presumably so they can complain if they lose)

In the context of artificial intelligence in video games, cheating refers to the programmer giving agents actions and access to information that would be unavailable to the player in the same situation. Believing that the Atari 8-bit could not compete against a human player, Chris Crawford did not fix a bug in Eastern Front (1941) that benefited the computer-controlled Russian side. Computer Gaming World in 1994 reported that "It is a well-known fact that many AIs 'cheat' (or, at least, 'fudge') in order to be able to keep up with human players".

For example, if the agents want to know if the player is nearby they can either be given complex, human-like sensors (seeing, hearing, etc.), or they can cheat by simply asking the game engine for the player's position. Common variations include giving AIs higher speeds in racing games to catch up to the player or spawning them in advantageous positions in first-person shooters. The use of cheating in AI shows the limitations of the "intelligence" achievable artificially; generally speaking, in games where strategic creativity is important, humans could easily beat the AI after a minimum of trial and error if it were not for this advantage. Cheating is often implemented for performance reasons where in many cases it may be considered acceptable as long as the effect is not obvious to the player. While cheating refers only to privileges given specifically to the AI-it does not include the inhuman swiftness and precision natural to a computer-a player might call the computer's inherent advantages "cheating" if they result in the agent acting, unlike a human player. Sid Meier stated that he omitted multiplayer alliances in Civilization because he found that the computer was almost as good as humans in using them, which caused players to think that the computer was cheating. Developers say that most are honest but they dislike players erroneously complaining about "cheating" AI. In addition, humans use tactics against computers that they would not against other people.

Creatures is an artificial life program where the user "hatches" small furry animals and teaches them how to behave. These "Norns" can talk, feed themselves, and protect themselves against vicious creatures. It was the first popular application of machine learning in an interactive simulation. Neural networks are used by the creatures to learn what to do. The game is regarded as a breakthrough in artificial life research, which aims to model the behavior of creatures interacting with their environment.

A first-person shooter where the player assumes the role of the Master Chief, battling various aliens on foot or in vehicles. Enemies use cover very wisely, and employ suppressing fire and grenades. The squad situation affects the individuals, so certain enemies flee when their leader dies. A lot of attention is paid to the little details, with enemies notably throwing back grenades or team-members responding to you bothering them. The underlying "behavior tree" technology has become very popular in the games industry (especially since Halo 2).

A psychological horror first-person shooter with a gloomy and fascinating storyline. There the player fighting against battalion of clone super-soldiers, robots and paranormal creatures. The AI uses a planner to generate context-sensitive behaviors, the first time in a mainstream game. This technology is still used as a reference for many studios. The enemies are capable of using the environment very cleverly, finding cover behind tables, tipping bookshelves, opening doors, crashing through windows, and so on. Squad tactics are used to great effect. The enemies perform flanking maneuvers, use suppressing fire, etc. As mentioned earlier, what sets it apart is the opponent AI which is extremely good and acts on even minute details. For example, in the game if a player takes cover behind a barricade, the enemy AI's quickly processes and throws a grenade to eliminate the player. Another aspect is the game AI's communication in the game. The enemies talk and detail their movements according to that of the player. In the game has achieved what AI could not achieved in the earlier years, namely, to give a person a similar experience. The remarkable enemy AI's still impresses to this day. The game is one of the greatest shooters with game AI's. That's not to mention the weapons physics and movement animations in this game, where they were groundbreaking for their time. This game won GameSpot's "2005 Best AI Award".

A first-person shooter survival horror game where the player must face man-made experiments, military soldiers, and mercenaries known as Stalkers. The various encountered enemies (if the difficulty level is set to its highest) use combat tactics and behaviors such as healing wounded allies, giving orders, out-flanking the player or using weapons with pinpoint accuracy.

A real-time strategy game where a player takes control of one of three factions in a 1v1, 2v2, or 3v3 battle arena. The player must defeat their opponents by destroying all their units and bases. This is accomplished by creating units that are effective at countering your opponents' units. Players can play against multiple different levels of AI difficulty ranging from very easy to Cheater 3 (insane). The AI is able to cheat at the difficulty Cheater 1 (vision), where it can see units and bases when a player in the same situation could not. Cheater 2 gives the AI extra resources, while Cheater 3 gives an extensive advantage over its opponent.

# data[].attributes.attachment_link.description
In video games, artificial intelligence (AI) is used to generate responsive, adaptive or intelligent behaviors primarily in non-player characters (NPCs) similar to human-like intelligence. Artificial intelligence has been an integral part of video games since their inception in the 1950s. AI in video games is a distinct subfield and differs from academic AI.

# data[].attributes.body
In the 1950's TV presenters and actors kept forgetting their lines, so people were hired to hold a roll of paper next to the camera with the actors lines so they could easily read off the roll with no mistakes. Autocue then released a system where 2 way mirrors and a screen were used to display the presenters lines directly in front of the camera so they were able to look directly into the camera while reading their lines, this can be controlled by the presenter if they have their own scroll wheel or by the Autocue operator in the gallery. This evolution in tele-prompting has only benefited the TV industry as it has streamlined the process of tele prompting because instead of writing out the presenters script by hand and holding a big scroll of text, operators of teleprompters simply copy and paste the script into the Autocue software and scroll through it with a scroll wheel