# data[].attributes.attachment_link.title
How Do Machines Learn? Artificial Intelligence as a New Era in Medicine

# data[].attributes.attachment_link.description
With an increased number of medical data generated every day, there is a strong need for reliable, automated evaluation tools. With high hopes and expectations, machine learning has the potential to revolutionize many fields of medicine, helping to make faster and more correct decisions and improvin…

# data[].attributes.body
Artificial intelligence successful learning process is based on connectivity, where for example, self-driving cars will be connected to each other so that when one makes a mistake, the learning can be quickly shared with the network, improving the service's security and function. Another fundamental quality that increases artificial intelligences' knowledge pool, is it's interactive algorithms, which will stimulate the machine to mimic human preference and expand its knowledge specifically to the individual or entity that uses it.

# data[].attributes.attachment_link.title
How does artificial intelligence learn? - Briana Brownell

# data[].attributes.attachment_link.readable_text
Machine learning is behind chatbots and predictive text, language translation apps, the shows Netflix suggests to you, and how your social media feeds are presented. It powers autonomous vehicles and machines that can diagnose medical conditions based on images.

When companies today deploy artificial intelligence programs, they are most likely using machine learning - so much so that the terms are often used interchangeably, and sometimes ambiguously. Machine learning is a subfield of artificial intelligence that gives computers the ability to learn without explicitly being programmed.

"In just the last five or 10 years, machine learning has become a critical way, arguably the most important way, most parts of AI are done," said MIT Sloan professor Thomas W. Malone, the founding director of the MIT Center for Collective Intelligence. "So that's why some people use the terms AI and machine learning almost as synonymous ... most of the current advances in AI have involved machine learning."

With the growing ubiquity of machine learning, everyone in business is likely to encounter it and will need some working knowledge about this field. A 2020 Deloitte survey found that 67% of companies are using machine learning, and 97% are using or planning to use it in the next year.

From manufacturing to retail and banking to bakeries, even legacy companies are using machine learning to unlock new value or boost efficiency. "Machine learning is changing, or will change, every industry, and leaders need to understand the basic principles, the potential, and the limitations," said MIT computer science professor Aleksander Madry, director of the MIT Center for Deployable Machine Learning.

While not everyone needs to know the technical details, they should understand what the technology does and what it can and cannot do, Madry added. "I don't think anyone can afford not to be aware of what's happening."

That includes being aware of the social, societal, and ethical implications of machine learning. "It's important to engage and begin to understand these tools, and then think about how you're going to use them well. We have to use these [tools] for the good of everybody," said Dr. Joan LaRovere, MBA '16, a pediatric cardiac intensive care physician and co-founder of the nonprofit The Virtue Foundation. "AI has so much potential to do good, and we need to really keep that in our lenses as we're thinking about this. How do we use this to do good and better the world?"

What is machine learning?

Machine learning is a subfield of artificial intelligence, which is broadly defined as the capability of a machine to imitate intelligent human behavior. Artificial intelligence systems are used to perform complex tasks in a way that is similar to how humans solve problems.

The goal of AI is to create computer models that exhibit "intelligent behaviors" like humans, according to Boris Katz, a principal research scientist and head of the InfoLab Group at CSAIL. This means machines that can recognize a visual scene, understand a text written in natural language, or perform an action in the physical world.

Machine learning is one way to use AI. It was defined in the 1950s by AI pioneer Arthur Samuel as "the field of study that gives computers the ability to learn without explicitly being programmed."

The definition holds true, according to Mikey Shulman, a lecturer at MIT Sloan and head of machine learning at Kensho, which specializes in artificial intelligence for the finance and U.S. intelligence communities. He compared the traditional way of programming computers, or "software 1.0," to baking, where a recipe calls for precise amounts of ingredients and tells the baker to mix for an exact amount of time. Traditional programming similarly requires creating detailed instructions for the computer to follow.

But in some cases, writing a program for the machine to follow is time-consuming or impossible, such as training a computer to recognize pictures of different people. While humans can do this task easily, it's difficult to tell a computer how to do it. Machine learning takes the approach of letting computers learn to program themselves through experience.

Machine learning starts with data - numbers, photos, or text, like bank transactions, pictures of people or even bakery items, repair records, time series data from sensors, or sales reports. The data is gathered and prepared to be used as training data, or the information the machine learning model will be trained on. The more data, the better the program.

From there, programmers choose a machine learning model to use, supply the data, and let the computer model train itself to find patterns or make predictions. Over time the human programmer can also tweak the model, including changing its parameters, to help push it toward more accurate results. (Research scientist Janelle Shane's website AI Weirdness is an entertaining look at how machine learning algorithms learn and how they can get things wrong - as happened when an algorithm tried to generate recipes and created Chocolate Chicken Chicken Cake.)

Some data is held out from the training data to be used as evaluation data, which tests how accurate the machine learning model is when it is shown new data. The result is a model that can be used in the future with different sets of data.

Successful machine learning algorithms can do different things, Malone wrote in a recent research brief about AI and the future of work that was co-authored by MIT professor and CSAIL director Daniela Rus and Robert Laubacher, the associate director of the MIT Center for Collective Intelligence.

"The function of a machine learning system can be descriptive, meaning that the system uses the data to explain what happened; predictive, meaning the system uses the data to predict what will happen; or prescriptive, meaning the system will use the data to make suggestions about what action to take," the researchers wrote.

There are three subcategories of machine learning:

Supervised machine learning models are trained with labeled data sets, which allow the models to learn and grow more accurate over time. For example, an algorithm would be trained with pictures of dogs and other things, all labeled by humans, and the machine would learn ways to identify pictures of dogs on its own. Supervised machine learning is the most common type used today.

In unsupervised machine learning, a program looks for patterns in unlabeled data. Unsupervised machine learning can find patterns or trends that people aren't explicitly looking for. For example, an unsupervised machine learning program could look through online sales data and identify different types of clients making purchases.

Reinforcement machine learning trains machines through trial and error to take the best action by establishing a reward system. Reinforcement learning can train models to play games or train autonomous vehicles to drive by telling the machine when it made the right decisions, which helps it learn over time what actions it should take.

In the Work of the Future brief, Malone noted that machine learning is best suited for situations with lots of data - thousands or millions of examples, like recordings from previous conversations with customers, sensor logs from machines, or ATM transactions. For example, Google Translate was possible because it "trained" on the vast amount of information on the web, in different languages.

In some cases, machine learning can gain insight or automate decision-making in cases where humans would not be able to, Madry said. "It may not only be more efficient and less costly to have an algorithm do this, but sometimes humans just literally are not able to do it," he said.

Google search is an example of something that humans can do, but never at the scale and speed at which the Google models are able to show potential answers every time a person types in a query, Malone said. "That's not an example of computers putting people out of work. It's an example of computers doing things that would not have been remotely economically feasible if they had to be done by humans."

Machine learning is also associated with several other artificial intelligence subfields:
Natural language processing
Natural language processing is a field of machine learning in which machines learn to understand natural language as spoken and written by humans, instead of the data and numbers normally used to program computers. This allows machines to recognize language, understand it, and respond to it, as well as create new text and translate between languages. Natural language processing enables familiar technology like chatbots and digital assistants like Siri or Alexa.
Neural networks
Neural networks are a commonly used, specific class of machine learning algorithms. Artificial neural networks are modeled on the human brain, in which thousands or millions of processing nodes are interconnected and organized into layers.

In an artificial neural network, cells, or nodes, are connected, with each cell processing inputs and producing an output that is sent to other neurons. Labeled data moves through the nodes, or cells, with each cell performing a different function. In a neural network trained to identify whether a picture contains a cat or not, the different nodes would assess the information and arrive at an output that indicates whether a picture features a cat.
Deep learning
Deep learning networks are neural networks with many layers. The layered network can process extensive amounts of data and determine the "weight" of each link in the network - for example, in an image recognition system, some layers of the neural network might detect individual features of a face, like eyes, nose, or mouth, while another layer would be able to tell whether those features appear in a way that indicates a face.

Like neural networks, deep learning is modeled on the way the human brain works and powers many machine learning uses, like autonomous vehicles, chatbots, and medical diagnostics.

"The more layers you have, the more potential you have for doing complex things well," Malone said.

Deep learning requires a great deal of computing power, which raises concerns about its economic and environmental sustainability.

How businesses are using machine learning

Machine learning is the core of some companies' business models, like in the case of Netflix's suggestions algorithm or Google's search engine. Other companies are engaging deeply with machine learning, though it's not their main business proposition.

Others are still trying to determine how to use machine learning in a beneficial way. "In my opinion, one of the hardest problems in machine learning is figuring out what problems I can solve with machine learning," Shulman said. "There's still a gap in the understanding."

In a 2018 paper, researchers from the MIT Initiative on the Digital Economy outlined a 21-question rubric to determine whether a task is suitable for machine learning. The researchers found that no occupation will be untouched by machine learning, but no occupation is likely to be completely taken over by it. The way to unleash machine learning success, the researchers found, was to reorganize jobs into discrete tasks, some which can be done by machine learning, and others that require a human.

Companies are already using machine learning in several ways, including:

Recommendation algorithms. The recommendation engines behind Netflix and YouTube suggestions, what information appears on your Facebook feed, and product recommendations are fueled by machine learning. "[The algorithms] are trying to learn our preferences," Madry said. "They want to learn, like on Twitter, what tweets we want them to show us, on Facebook, what ads to display, what posts or liked content to share with us."

Image analysis and object detection. Machine learning can analyze images for different information, like learning to identify people and tell them apart - though facial recognition algorithms are controversial. Business uses for this vary. Shulman noted that hedge funds famously use machine learning to analyze the number of cars in parking lots, which helps them learn how companies are performing and make good bets.

Fraud detection. Machines can analyze patterns, like how someone normally spends or where they normally shop, to identify potentially fraudulent credit card transactions, log-in attempts, or spam emails.

Automatic helplines or chatbots. Many companies are deploying online chatbots, in which customers or clients don't speak to humans, but instead interact with a machine. These algorithms use machine learning and natural language processing, with the bots learning from records of past conversations to come up with appropriate responses.

Self-driving cars. Much of the technology behind self-driving cars is based on machine learning, deep learning in particular.

Medical imaging and diagnostics. Machine learning programs can be trained to examine medical images or other information and look for certain markers of illness, like a tool that can predict cancer risk based on a mammogram.
Read report: Artificial Intelligence and the Future of Work
How machine learning works: promises and challenges

While machine learning is fueling technology that can help workers or open new possibilities for businesses, there are several things business leaders should know about machine learning and its limits.
Explainability
One area of concern is what some experts call explainability, or the ability to be clear about what the machine learning models are doing and how they make decisions. "Understanding why a model does what it does is actually a very difficult question, and you always have to ask yourself that," Madry said. "You should never treat this as a black box, that just comes as an oracle ... yes, you should use it, but then try to get a feeling of what are the rules of thumb that it came up with? And then validate them."

This is especially important because systems can be fooled and undermined, or just fail on certain tasks, even those humans can perform easily. For example, adjusting the metadata in images can confuse computers - with a few adjustments, a machine identifies a picture of a dog as an ostrich.

Madry pointed out another example in which a machine learning algorithm examining X-rays seemed to outperform physicians. But it turned out the algorithm was correlating results with the machines that took the image, not necessarily the image itself. Tuberculosis is more common in developing countries, which tend to have older machines. The machine learning program learned that if the X-ray was taken on an older machine, the patient was more likely to have tuberculosis. It completed the task, but not in the way the programmers intended or would find useful.

The importance of explaining how a model is working - and its accuracy - can vary depending on how it's being used, Shulman said. While most well-posed problems can be solved through machine learning, he said, people should assume right now that the models only perform to about 95% of human accuracy. It might be okay with the programmer and the viewer if an algorithm recommending movies is 95% accurate, but that level of accuracy wouldn't be enough for a self-driving vehicle or a program designed to find serious flaws in machinery.
Bias and unintended outcomes
Machines are trained by humans, and human biases can be incorporated into algorithms - if biased information, or data that reflects existing inequities, is fed to a machine learning program, the program will learn to replicate it and perpetuate forms of discrimination. Chatbots trained on how people converse on Twitter can pick up on offensive and racist language, for example.

In some cases, machine learning models create or exacerbate social problems. For example, Facebook has used machine learning as a tool to show users ads and content that will interest and engage them - which has led to models showing people extreme content that leads to polarization and the spread of conspiracy theories when people are shown incendiary, partisan, or inaccurate content.

Ways to fight against bias in machine learning including carefully vetting training data and putting organizational support behind ethical artificial intelligence efforts, like making sure your organization embraces human-centered AI, the practice of seeking input from people of different backgrounds, experiences, and lifestyles when designing AI systems. Initiatives working on this issue include the Algorithmic Justice League and The Moral Machine project.

Putting machine learning to work

Shulman said executives tend to struggle with understanding where machine learning can actually add value to their company. What's gimmicky for one company is core to another, and businesses should avoid trends and find business use cases that work for them.

Machine learning is changing, or will change, every industry, and leaders need to understand the basic principles, the potential, and the limitations.
Aleksander Madry MIT Computer Science Professor
The way machine learning works for Amazon is probably not going to translate at a car company, Shulman said - while Amazon has found success with voice assistants and voice-operated speakers, that doesn't mean car companies should prioritize adding speakers to cars. More likely, he said, the car company might find a way to use machine learning on the factory line that saves or makes a great deal of money.

"The field is moving so quickly, and that's awesome, but it makes it hard for executives to make decisions about it and to decide how much resourcing to pour into it," Shulman said.

It's also best to avoid looking at machine learning as a solution in search of a problem, Shulman said. Some companies might end up trying to backport machine learning into a business use. Instead of starting with a focus on technology, businesses should start with a focus on a business problem or customer need that could be met with machine learning.

A basic understanding of machine learning is important, LaRovere said, but finding the right machine learning use ultimately rests on people with different expertise working together. "I'm not a data scientist. I'm not doing the actual data engineering work - all the data acquisition, processing, and wrangling to enable machine learning applications - but I understand it well enough to be able to work with those teams to get the answers we need and have the impact we need," she said. "You really have to work in a team."

Sign-up for a Machine Learning in Business Course.

Watch an Introduction to Machine Learning through MIT OpenCourseWare.

Read about how an AI pioneer thinks companies can use machine learning to transform.

Watch a discussion with two AI experts about machine learning strides and limitations.

Take a look at the seven steps of machine learning.
Read next: 7 lessons for successful machine learning projects

# data[].attributes.attachment_link.description
Machine learning is a powerful form of artificial intelligence that is affecting every industry. Here's what you need to know about its potential and limitations and how it's being used.

# data[].attributes.attachment_link.readable_text
ChatGPT is the latest language model from OpenAI and represents a significant improvement over its predecessors. Similarly to many Large Language Models, ChatGPT is capable of generating text in a wide range of styles and for different purposes, but with remarkably greater precision, detail, and coherence. It represents the next generation in OpenAI's line of Large Language Models, and it is designed with a strong focus on interactive conversations.

The creators have used a combination of both Supervised Learning and Reinforcement Learning to fine-tune ChatGPT, but it is the Reinforcement Learning component specifically that makes ChatGPT unique. The creators use a particular technique called Reinforcement Learning from Human Feedback (RLHF), which uses human feedback in the training loop to minimize harmful, untruthful, and/or biased outputs.

We are going to examine GPT-3's limitations and how they stem from its training process, before learning how RLHF works and understand how ChatGPT uses RLHF to overcome these issues. We will conclude by looking at some of the limitations of this methodology.

Capability vs Alignment in Large Language Models

In the context of machine learning, the term capability refers to a model's ability to perform a specific task or set of tasks. A model's capability is typically evaluated by how well it is able to optimize its objective function, the mathematical expression that defines the goal of the model. For example, a model designed to predict stock market prices might have an objective function that measures the accuracy of the model's predictions. If the model is able to accurately predict the movement of stock prices over time, it would be considered to have a high level of capability for this task.

Alignment, on the other hand, is concerned with what we actually want the model to do versus what it is being trained to do. It asks the question "is that objective function consistent with our intentions?" and refers to the extent to which a model's goals and behavior align with human values and expectations. For a simple concrete example, say we train a bird classifier to classify birds as either "sparrows" or "robins" and we use log loss (which measures the difference between the predicted probability distribution of the model and the true distribution) as the training objective, even though our ultimate goal is a high classification accuracy. The model might have low log loss, i.e. the model's capability is high, but poor accuracy on the test set. In fact, the log loss is not perfectly correlated with accuracy in classification tasks. This is an example of misalignment, where the model is capable of optimizing the training objective but poorly aligned with our ultimate goal.

Models like the original GPT-3 are misaligned

Large Language Models, such as GPT-3, are trained on vast amounts of text data from the internet and are capable of generating human-like text, but they may not always produce output that is consistent with human expectations or desirable values. In fact, their objective function is a probability distribution over word sequences (or token sequences) that allows them to predict what the next word is in a sequence (more details on this below).

In practical applications, however, these models are intended to perform some form of valuable cognitive work, and there is a clear divergence between the way these models are trained and the way we would like to use them. Even though a machine calculated statistical distribution of word sequences might be, mathematically speaking, a very effective choice to model language, we as humans generate language by choosing text sequences that are best for the given situation, using our background knowledge and common sense to guide this process. This can be a problem when language models are used in applications that require a high degree of trust or reliability, such as dialogue systems or intelligent personal assistants.

While these powerful, complex models trained on huge amounts of data have become extremely capable in the last few years, when used in production systems to make human lives easier they often fall short of this potential. The alignment problem in Large Language Models typically manifests as:

Lack of helpfulness: not following the user's explicit instructions.

Hallucinations: model making up unexisting or wrong facts.

Lack of interpretability: it is difficult for humans to understand how the model arrived at a particular decision or prediction.

Generating biased or toxic output: a language model that is trained on biased/toxic data may reproduce that in its output, even if it was not explicitly instructed to do so.

But where does this alignment problem stem from, concretely? Is it the very way language models are trained inherently prone to misalignment?

How language model training strategies can produce misalignment

Next-token-prediction and masked-language-modeling are the core techniques used for training language models, such as transformers. In the first approach, the model is given a sequence of words (or "tokens", i.e. parts of words) as input and is asked to predict the next word in the sequence. For example, if the model is given the input sentence
"The cat sat on the"
it might predict the next word as "mat", "chair", or "floor", because of the high-probability of occurrence of these words given the previous context; the language model is in fact able to estimate the likelihood of each possible word (in its vocabulary) given the previous sequence.

The masked language modeling approach is a variant of next token prediction, in which some of the words in the input sentence are replaced with a special token, such as [MASK]. The model is then asked to predict the correct word that should be inserted in place of the mask. For example, if the model is given the sentence

as input, it might predict the next word as "cat", "dog", or "rabbit".

One advantage of these objective functions is that it allows the model to learn the statistical structure of language, such as common word sequences and patterns of word usage. This generally helps the model generate more natural and fluent text, and it is an essential step in the pre-training phase of every language model.

However, these objective functions can also lead to problems, essentially because the model is not capable of distinguishing between an important error and an unimportant one. To make a very simple example, if the model is given the input sentence:

"The Roman Empire [MASK] with the reign of Augustus."

it might predict "began" or "ended", as both words score high likelihood of occurrence (indeed, both sentences are historically correct), even though the second choice implies a very different meaning.

More generally, these training strategies can lead to a misalignment of the language model for some more complex tasks, because a model which is only trained to predict the next word (or a masked word) in a text sequence, may not necessarily be learning some higher-level representations of its meaning. As a result, the model struggles to generalize to tasks or contexts that require a deeper understanding of language.

Researchers and developers are working on various approaches to address the alignment problem in Large Language Models. ChatGPT is based on the original GPT-3 model, but has been further trained by using human feedback to guide the learning process with the specific goal of mitigating the model's misalignment issues. The specific technique used, called Reinforcement Learning from Human Feedback, is based on previous academic research. ChatGPT represents the first case of use of this technique for a model put into production.

But how exactly do the creators of ChatGPT use human feedback to attack the alignment problem?

Reinforcement Learning from Human Feedback

In this section, we are going to explain how RLHF was used to align ChatGPT to human preferences. For updated content on RLHF and its current limitations, you may want to read the dedicated article:

How RLHF Preference Model Tuning Works (And How Things May Go Wrong)

Fine-tuning ChatGPT with RLHF consisted of three distinct steps:

1. Supervised fine-tuning - A pre-trained language model is fine-tuned on a relatively small amount of demonstration data curated by labelers, to learn a supervised policy (the SFT model) that generates outputs from a selected list of prompts. This represents the baseline model.

2. "Mimic human preferences" - Labelers are asked to vote on a relatively large number of the SFT model outputs, this way creating a new dataset consisting of comparison data. A new model is trained on this dataset. This is referred to as the reward model (RM).

3. Proximal Policy Optimization (PPO) - The reward model is used to further fine-tune and improve the SFT model. The outcome of this step is the so-called policy model.

Step 1 takes place only once, while steps 2 and 3 can be iterated continuously: more comparison data is collected on the current best policy model, which is used to train a new reward model and then a new policy.

Let's now dive into the details of each step!

Note: The rest of this article is based on the content of the InstructGPT paper. According to OpenAI, ChatGPT has been trained "using the same methods as InstructGPT, but with slight differences in the data collection setup" ( source). Unfortunately, exact quantitative reports have yet to be made publicly available for ChatGPT.

Step 1: The Supervised Fine-Tuning (SFT) model

The first step consists in collecting demonstration data in order to train a supervised policy model, referred to as the SFT model.

Data collection: a list of prompts is selected and a group of human labelers are asked to write down the expected output response. For ChatGPT, two different sources of prompts have been used: some have been prepared directly from the labelers or developers, some have been sampled from OpenAI's API requests (i.e. from their GPT-3 customers). As this whole process is slow and expensive, the result is a relatively small, high-quality curated dataset (of approximately 12-15k data points, presumably) that is to be used to fine-tune a pretrained language model.

Choice of model: instead of fine-tuning the original GPT-3 model, the developers of ChatGPT opted for a pretrained model in the so-called GPT-3.5 series. Presumably the baseline model used is the latest one text-davinci-003, a GPT-3 model which was fine-tuned mostly on programming code.

Quite interestingly, therefore, in order to create a general purpose chatbot like ChatGPT, the developers decided to fine-tune on top of a "code model" rather than a pure text model.

Due to the limited amount of data for this step, the SFT model obtained after this process is likely to output text which is still (probabilistically) not very user-attentive and generally suffers from misalignment, in the sense explained in the above sections. The problem here is that the supervised learning step suffers from high scalability costs.

To overcome this problem, instead of asking human labelers to create a much bigger curated dataset, a slow and costly process, the strategy is now to have the labelers rank different outputs of the SFT model to create a reward model -let's explain this in more detail in the following section.

Step 2: The reward model (RM)

The goal is to learn an objective function (the reward model) directly from the data. The purpose of this function is to give a score to the SFT model outputs, proportional to how desirable these outputs are for humans. In practice, this will strongly reflect the specific preferences of the selected group of human labelers and the common guidelines which they agreed to follow. In the end, this process will extract from the data an automatic system that is supposed to mimic human preferences.

Here's how it works:

A list of prompts is selected and the SFT model generates multiple outputs (anywhere between 4 and 9) for each prompt.

Labelers rank the outputs from best to worst. The result is a new labeled dataset, where the rankings are the labels. The size of this dataset is approximately 10 times bigger than the curated dataset used for the SFT model.

This new data is used to train a reward model (RM). This model takes as input a few of the SFT model outputs and ranks them in order of preference.

As for labelers it is much easier to rank the outputs than to produce them from scratch, this process scales up much more efficiently. In practice, this dataset has been generated from a selection of 30-40k prompts, and a variable number of the generated outputs (for each prompt) is presented to the each labeler during the ranking phase.

Step 3: Fine-tuning the SFT model via Proximal Policy Optimization (PPO)

Reinforcement Learning is now applied to fine-tune the SFT policy by letting it optimize the reward model. The specific algorithm used is called Proximal Policy Optimization (PPO) and the fine-tuned model is referred to as the PPO model.

What is PPO? Here are the main takeaways of this method:

PPO is an algorithm that is used to train agents in reinforcement learning. It is called an "on-policy" algorithm because it learns from and updates the current policy directly, rather than learning from past experiences as in "off-policy" algorithms like DQN (Deep Q-Network). This means that PPO is continuously adapting the current policy based on the actions that the agent is taking and the rewards it is receiving.

PPO uses a trust region optimization method to train the policy, which means that it constrains the change in the policy to be within a certain distance of the previous policy in order to ensure stability. This is in contrast to other policy gradient methods which can sometimes make large updates to the policy that can destabilize learning.

PPO uses a value function to estimate the expected return of a given state or action. The value function is used to compute the advantage function, which represents the difference between the expected return and the current return. The advantage function is then used to update the policy by comparing the action taken by the current policy to the action that would have been taken by the previous policy. This allows PPO to make more informed updates to the policy based on the estimated value of the actions being taken.

In this step, the PPO model is initialized from the SFT model, and the value function is initialized from the reward model. The environment is a bandit environment which presents a random prompt and expects a response to the prompt. Given the prompt and response, it produces a reward (determined by the reward model) and the episode ends. A per-token KL penalty is added from the SFT model at each token to mitigate over optimization of the reward model.

Performance Evaluation

Because the model is trained on human labelers input, the core part of the evaluation is also based on human input, i.e. it takes place by having labelers rate the quality of the model outputs. To avoid overfitting to the judgment of the labelers involved in the training phase, the test set uses prompts from held-out OpenAI customers which are not represented in the training data.

The model is evaluated on three high-level criteria:

Helpfulness: judging the model's ability to follow user instructions, as well as infer instructions.

Truthfulness: judging the model's tendency for hallucinations (making up facts) on closed-domain tasks. The model is evaluated on the TruthfulQA dataset.

Harmlessness: the labelers evaluate whether the model's output is appropriate, denigrates a protected class, or contains derogatory content. The model is also benchmarked on the RealToxicityPrompts and CrowS-Pairs datasets.

The model is also evaluated for zero-shot performance on traditional NLP tasks like question answering, reading comprehension, and summarization, on some of which the developers observed performance regressions compared to GPT-3. This is an example of an "alignment tax" where the RLHF-based alignment procedure comes at the cost of lower performance on certain tasks.

The performance regressions on these datasets can be greatly reduced with a trick called pre-train mix: during training of the PPO model via gradient descent, the gradient updates are computed by mixing the gradients of the SFT model and the PPO model.

Shortcomings of the methodology

A very clear limitation of the methodology, as discussed in the InstructGPT paper (on which ChatGPT is based, according to its creators) is the fact that, in the process of aligning language models with human intentions, the data for fine-tuning the models is influenced by an intricate variety of subjective factors, including:

The preferences of the labelers who produce the demonstration data.

The researchers who design the study and write the labeling instructions.

The choice of prompts crafted by the developers or provided by the OpenAI customers.

The labelers bias is both included in the reward model training (by ranking outputs) and in the model evaluation.

In particular, the authors point out the obvious fact that the labelers and researchers taking part in the training process may not be representative of all potential end users of the language model.

Apart from this clear "intrinsic" limitation, we want to point out a few other possible shortcomings of the method, problems not explicitly addressed, as well as some open questions:

Lack of control study: The reported results measure the performance of the final PPO model, taking the SFT model as the baseline. This can be misleading: how can we know the improvements are actually due to the RLHF? A proper (yet, expensive) control study would consist in investing the exact same amount of labeler-hours as those used to train the reward model into creating a larger curated SFT dataset with high-quality demonstration data. One would then be in the position to objectively measure the performance improvements of the RLHF methodology versus the supervised approach. In very simple terms, the lack of such a control study is leaving a fundamental question completely open: is RLHF actually doing a good job in aligning language models?

Lack of ground truth for the comparison data: labelers can often disagree on the ranking of the model's outputs. Technically speaking, the risk is to add a high potential variance to the comparison data without any ground truth.

Human preferences are just not homogeneous: The RLHF method treats human preferences as if they were homogeneous and static. Assuming that all people share the same values would be an obvious stretch, at least on a large amount of topics of human knowledge. Some recent research is starting to tackle this open problem differently.

Prompt-stability testing for the reward model (RM): There seem to be no experiments investigating the sensitivity of the reward model in terms of changes in the input prompt. If two prompts are syntactically different but are semantically equivalent, can the RM show significant differences in the ranking of the model outputs? In simpler terms, how much does the quality of the prompt matter for the RM?

Wireheading-type issues: In RL approaches, the model can sometimes learn to manipulate its own reward system to achieve a desired outcome, leading to an "over optimized policy". This can push the model recreating some patterns that for some unknown reason make the reward model score high (see Table 29 in this paper from OpenAI for an explicit example of this behavior in language modeling). ChatGPT puts a patch on this with the KL penalty term in the reward function. Note that one is trying to optimize the RM input (i.e. the PPO output) in order to improve its output (the reward score), all the while constraining the input itself to be not too far away from some reference input (the SFT output). More details on the limitations of this approach in this preprint.

Selected references for further reading

# data[].attributes.attachment_link.description
Since its release, the public has been playing with ChatGPT and seeing what it can do, but how does ChatGPT actually work? While the details of its inner workings have not been published, we can piece together its functioning principles from recent research.

# data[].attributes.attachment_link.readable_text
AI: Optimism, pessimism, and a middle road for managers

When asked what he thought about artificial intelligence (AI), the late, great Amos Tversky joked that he did not know much about it; rather, his specialty was natural stupidity (Lewis, 2016). Until very recently, AI has been regarded as a good idea, but one more suited to the realms of science fiction. Years of funding setbacks and a public perception that AI was failing to live up to its many promises (McCorduck, 2009) caused AI to languish. In the past 2 or 3 years, however, AI has reemerged

AI building blocks

We employ Poole and Mackworth's (2010, p. 3) definition of AI as "computational agents that act intelligently " and perceive their environments in order to take actions that maximize chances of success. This is a departure from previous views that emphasized AI displaying humanlike intelligence. This definition also notably involves an agent in the process. As computational agents, AI agents perceive and act upon their environment, solving problems in practice and not just in theory.

Any

A typology of AI-enabled innovations and their effects on competencies

AI, as part of systems that act intelligently, can be used in any combination of the building blocks described in Section 2 to help organizations find creative solutions to business problems and exploit worthwhile opportunities. The examples show how AI can introduce significant changes to organizational functions, processes, and product functionalities, as well as how it can alter firms' competencies and thus change how they compete within their industries.

Our core argument is that by

Applications of the AI typology

The typology introduced above is useful in at least four important exercises for decision makers. First, executives can use the typology as an analytic tool to anticipate how AI-enabled innovations might affect their organization's competencies. When managers investigate which of their processes or products could be affected by AI and whether the outcomes would be competence-enhancing or -destroying, they can understand more fully their firm's exposure to AI-based risks or opportunities and

Limitations of AI

AI systems can introduce profound and pervasive changes to firms and can fundamentally reinvent how they run, compete, and thrive. Today's substantial research and development efforts in AI are split broadly into two fields (Aleksander, 2017, Armstrong et al., 2014). Artificial generalized intelligence, also known as 'strong' AI, aims to develop machine intelligences capable of turning their hands to any task, much like a person can. This goal is extremely ambitious, and some might argue it is

Summary

Notwithstanding these limitations, executives and entrepreneurs eager to benefit from the potential that applied AI offers need to understand its foundational building blocks and its implications for organizational competencies. We hope this article provides insightful perspectives on both and that it will help executives and general managers to capitalize on opportunities to innovate with AI, while also alerting them to its risks and limitations.

In describing the six building blocks of AI and

# data[].attributes.attachment_link.title
Artificial intelligence: Building blocks and an innovation typology

# data[].attributes.attachment_link.description
The range of topics and the opinions expressed on artificial intelligence (AI) are so broad that clarity is needed on the the field's central tenets, ...

# data[].attributes.attachment_link.description
About ScienceDirect Shopping cart Contact and supportTerms and conditionsPrivacy policy

# data[].attributes.attachment_link.readable_text
Artificial intelligence is transforming all sectors of the economy, but there's no reason to fear that robots will replace all human employees. In fact, companies that automate their operations mainly to cut their workforces will see only short-term productivity gains, say the authors. Their research, involving 1,500 firms in a range of industries, shows that the biggest performance improvements come when humans and smart machines work together, enhancing each other's strengths.

People need to train AI agents, explain their outputs, and make sure they are used responsibly. AI agents, in turn, can assist people with information gathering, data crunching, routine customer service, and physical labor, thereby freeing them for higher-level tasks that require leadership, creative thinking, judgment, and other human skills.

To get the most out of AI, companies need to redesign their business processes. After deciding what needs improvement-their operational flexibility, speed, or scalability; their decision making; or their ability to personalize products and services-they can devise appropriate solutions. That will mean not only implementing AI technology but also developing employees who can work effectively at the human-machine interface.

The authors describe how a number of firms are already taking these steps and optimizing collaborative intelligence. But many more should follow their example.

# data[].attributes.attachment_link.title
How Humans and AI Are Working Together in 1,500 Companies

# data[].attributes.attachment_link.description
Artificial intelligence is transforming all sectors of the economy, but there's no reason to fear that robots will replace all human employees. In fact, companies that automate their operations mainly to cut their workforces will see only short-term productivity gains, say the authors.