Large Language Models

4.1 What is LLM

In the previous three chapters, we started from the definition of NLP and its main tasks, introducing the core idea that triggered a major transformation in the field of NLP - the attention mechanism and the Transformer architecture. With the emergence of the Transformer architecture, the field of NLP gradually entered the pre-training-finetuning paradigm, with a variety of pre-trained language models based on Transformers that have powerful text representation capabilities, pushing various classic NLP tasks to a new height.

With the release of ChatGPT at the end of 2022, which once again broke the upper limit of NLP capabilities, large language models (LLMs) began to replace traditional pre-trained language models (PLMs) as the mainstream direction in NLP. A new research paradigm based on LLMs is also refreshing the pre-training-finetuning paradigm popularized by BERT. NLP has thus undergone another significant transformation. From the end of 2022 to now, the capabilities of LLMs have been continuously updated, the number of general foundation large models has increased exponentially, and the concepts and applications based on LLMs have been constantly evolving, indicating the arrival of the era of large models.

In Chapter 3, we analyzed the classic models and their training processes under three architectures: Encoder-Only, Encoder-Decoder, and Decoder-Only. Some of these models were milestones in the era before LLMs (such as BERT), while others are the stars of the LLM era, being strong contenders for Artificial General Intelligence (AGI). So, what exactly is an LLM? What is the core difference between LLMs and traditional PLMs? And what makes researchers so enthusiastic and optimistic about LLMs?

In this chapter, we will combine the previous discussion on model architectures to deeply analyze the definition, characteristics, and capabilities of LLMs, revealing the core differences between LLMs and traditional deep learning models. On this basis, we will demonstrate the actual three-stage training process of LLMs, helping readers understand how LLMs acquire such unique capabilities conceptually, thereby providing a theoretical foundation for further practical training of complete LLMs.

4.1.1 Definition of LLM

LLM, or Large Language Model, is known as a large language model or a large-scale language model, which is a language model with more parameters than traditional language models and is pre-trained on larger-scale corpora.

In Chapter 1, we introduced the concept of a language model, which is an NLP model trained by predicting the next token. LLMs use similar architectures and pre-training tasks as traditional pre-trained language models (e.g., Decoder-Only architecture and CLM pre-training tasks), but they have a much larger number of parameters and are pre-trained on massive corpora, thus showing significantly different capabilities from traditional pre-trained language models.

Generally, LLM refers to a language model containing hundreds of billions (or more) of parameters, which are usually pre-trained on tens of T tokens of data using multi-card distributed clusters, possessing text understanding and generation capabilities far exceeding those of traditional pre-trained models. However, as LLM research continues to deepen, various sizes of LLMs have gradually emerged. Broadly speaking, LLMs generally cover all large-scale language models from billion-parameter models (such as Qwen-1.5B) to trillion-parameter models (such as Grok-314B). As long as the model demonstrates emergent abilities, meaning it shows significantly superior capabilities and potential compared to traditional pre-trained models (such as BERT, T5), it can be called an LLM.

It is generally believed that GPT-3 (175 billion parameters) marks the beginning of LLMs. The ChatGPT, which was obtained through the three-stage training of pretraining, supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF) based on GPT-3, has dominated the arrival of the LLM era. Since the release of ChatGPT by OpenAI in November 2022, less than two years have passed, and hundreds of LLMs with unique features and varying capabilities have emerged. The following table lists some of the large models released domestically and internationally from November 2022 to November 2023:

Time	Open-source LLM	Closed-source LLM
2022.11	None	OpenAI-ChatGPT
2023.02	Meta-LLaMA; Fudan-MOSS	None
2023.03	Stanford-Alpaca, Vicuna; Zhipu-ChatGLM	OpenAI-GPT4; Baidu-Wenxin Yanyi; Anthropic-Claude; Google-Bard
2023.04	Alibaba-Tongyi Qianwen; Stability AI-StableLM	SenseTime-RiRiXin
2023.05	Microsoft-Pi; Tll-Falcon	iFLYTEK-Xinghuo Large Model; Google-PaLM2
2023.06	Zhipu-ChatGLM2; Shanghai AI Lab-ShuRen PuYu; BaiChuan-BaiChuan; HuBo-TigerBot	360-ZhiNao Large Model
2023.07	Meta-LLaMA2	Anthropic-Claude2; Huawei-Pangu Large Model 3
2023.08	None	ByteDance-DouBao
2023.09	BaiChuan-BaiChuan2	Google-Gemini; Tencent-HunYuan Large Model
2023.11	ZeroOne万物-Yi; HuanFang-DeepSeek	xAI-Grok

At present, enterprises and research institutes at home and abroad are continuously launching more powerful LLMs, exploring the path to AGI.

4.1.2 Capabilities of LLM

(1) Emergent Abilities

The most significant feature distinguishing LLMs from traditional PLMs is that LLMs possess emergent abilities. Emergent abilities refer to certain capabilities that are not obvious in small models but are particularly prominent in large models under the same model architecture and pre-training tasks. It can be analogized to the phenomenon of phase transitions in physics, where the emergence of abilities is like the rapid improvement of model performance with increasing scale, surpassing random levels, which is often referred to as quantitative changes leading to qualitative changes.

Specifically, emergent abilities can be defined as capabilities related to certain complex tasks. However, generally speaking, NLP focuses more on their general capabilities, that is, the ability to solve various NLP tasks. Emergent abilities are the core reason for the high enthusiasm and attention of the industry and academia towards LLMs. Although LLMs currently have capabilities and can solve tasks that are still far from what humans ultimately expect from general artificial intelligence, we believe that with the continuous deepening of research, the emergence of high-quality data, and the appearance of more efficient model architectures and training frameworks, LLMs will eventually possess the capabilities required for general artificial intelligence, bringing about a qualitative change in human life.

(2) In-context Learning

The capability of in-context learning was first introduced by GPT-3. Specifically, in-context learning refers to allowing a language model to perform tasks by understanding the context and generating corresponding outputs when provided with natural language instructions or multiple task examples, without requiring additional training or parameter updates.

For traditional PLMs, after the costly pre-training, it is often necessary to perform supervised fine-tuning for specific downstream tasks. Although traditional PLMs are smaller in size and require less computing power, for example, models like BERT (0.5B parameters) usually require over 10G of memory for supervised fine-tuning, which involves certain computing costs. At the same time, the cost of training data for supervised fine-tuning is higher. Depending on the difficulty of the downstream task, the number of training samples needed is usually between 1k to tens of thousands, all requiring manual annotation, resulting in considerable data acquisition costs. LLMs with in-context learning capabilities often do not require high-cost additional training or fine-tuning, but can handle most tasks by providing a few examples or adjusting natural language instructions, thus greatly saving computing and data costs.

The capability of in-context learning is also causing a transformation in the research paradigm of NLP. In the era of traditional PLMs, the general paradigm for solving downstream NLP tasks was pre-training-finetuning, that is, selecting an appropriate pre-trained model and preparing supervised data for the specific downstream task for fine-tuning. By using LLMs with in-context learning capabilities, the general paradigm has begun to shift toward Prompt Engineering, i.e., adjusting prompts to elicit the capabilities of LLMs. For example, currently, most NLP tasks can achieve better results than traditional PLM fine-tuning by adjusting prompts or providing 1~5 natural language examples, enabling GPT-4 to achieve superior performance.

(3) Instruction Following

Through fine-tuning with multi-task data described in natural language, also known as instruction tuning, LLMs have been proven to perform well on unseen tasks described in instruction form. That is, instruction-tuned LLMs can understand and follow unseen instructions and execute tasks according to the task instructions without having seen specific examples beforehand, demonstrating their strong generalization ability.

The capability of instruction following means that we don't need to teach the model everything before it can do something. We just need to mix various instructions during the instruction tuning phase to train its generalization ability, and the LLM can handle most of the instructions from humans, i.e., flexibly solve problems encountered by users. This is particularly evident in ChatGPT. The core reason why ChatGPT has become so popular is that it is not only a theoretical model used in academia and industry, but can also widely serve users in various industries. By inputting instructions to ChatGPT, it can write essays, code programs, correct exams, read newspapers, etc.

The instruction following capability allows LLMs to truly integrate with multiple industries, empowering various aspects of human life through artificial intelligence technology, thereby bringing about a qualitative change for humanity. Whether it's the current hot topics of Agents and Workflows, or the all-around assistants and super-intelligence that may appear in the near future, their essence relies on the instruction following capability of LLMs.

(4) Step-by-Step Reasoning

Logical reasoning, especially complex reasoning tasks involving multiple reasoning steps, has always been a difficult point in NLP and one of the important reasons why artificial intelligence has not been widely recognized. After all, if a model cannot answer the basic "chicken and rabbit in a cage" problem or identify logical traps in language, it is hard to consider it "intelligent" rather than "stupid".

However, traditional NLP models often struggle with complex tasks involving multiple reasoning steps, such as mathematical problems. LLMs, however, can solve these tasks by using a chain-of-thought (CoT) reasoning strategy, utilizing a prompt mechanism that includes intermediate reasoning steps to arrive at the final answer. It is speculated that this capability may be acquired through training on code.

The capability of step-by-step reasoning means that LLMs can handle complex logical tasks, that is, they can solve most of the problems that require logical judgment in daily life, taking a solid step forward towards becoming a "reliable" intelligent assistant.

These unique capabilities are important advantages that distinguish LLMs from traditional PLMs, making them perform excellently in handling various tasks and becoming powerful tools for solving complex problems and applying to multiple fields. Because of the existence of emergent abilities, in-context learning capabilities, instruction following capabilities, and step-by-step reasoning capabilities, NLP researchers believe that LLMs are an important way to move towards general artificial intelligence and help human society achieve a qualitative change in productivity. In fact, there are already many applications based on LLMs aimed at significantly improving productivity using the unique capabilities of LLMs. For example, Microsoft's Copilot based on GPT-4 leverages the LLM's strong instruction following and step-by-step reasoning capabilities to provide functions such as code completion, code hints, and code writing, assisting programmers to write programs more efficiently, conveniently, and accurately, greatly improving the productivity of programmers.

4.1.3 Characteristics of LLM

In addition to the core capabilities of LLMs discussed above, LLMs also have some additional interesting or dangerous characteristics, which are important research directions for LLMs currently. Here, we discuss a few of them:

(1) Multilingual Support

Multilingual and cross-lingual models were once an important research direction in NLP. However, due to the need for massive corpora for pre-training, the training corpus itself is multilingual, so LLMs inherently have multilingual and cross-lingual capabilities, although the capabilities vary in different languages due to differences in training corpus and instruction tuning. Since high-quality English corpora currently dominate, most models represented by GPT-4 have significantly superior capabilities in English. Although they can process multiple languages, domestic models that have been additionally trained and optimized for Chinese (such as Wenxin Yanyi, Tongyi Qianwen, etc.) often show better performance in Chinese environments.

(2) Long Text Processing

The ability to process long context texts determines part of the model's capability to some extent, so LLMs pay more attention to long text processing capabilities than traditional PLMs. Compared to traditional PLMs (such as BERT, T5, etc.), which usually have a maximum context length of 512 tokens, LLMs have come up with many ingenious methods to expand the maximum context length. Due to training on massive distributed training clusters, LLMs often support context lengths of 4k, 8k, or even 32k during training. At the same time, LLMs mostly use rotary positional encoding (Rotary Positional Encoding, RoPE) (or similarly extrapolatable AliBi) as positional encoding, which has a certain length extrapolation capability, meaning that during inference, it can process texts significantly longer than the training length. For example, InternLM was pre-trained on a context length of 32k, but through RoPE, it can process a context length of 200k. By continuously enhancing the capability of long text processing, LLMs often have stronger information reading and summarizing capabilities, thus solving the "century problem" of requiring LLMs to read "Dream of the Red Chamber" and write a corresponding college entrance exam essay.

(3) Expanding to Multimodal

The powerful capabilities of LLMs have also given them strong performance across modalities. With the continuous improvement of LLMs, adding extra parameters to LLMs to represent images and then using the powerful capabilities of LLMs to create dual-modal models of text and images has become a successful method. By introducing Adapter layers and image encoders, and performing supervised fine-tuning on text-image data, the model can have good text-image question answering and generation capabilities. In the future, how to align the representations of text and images to build more powerful multimodal large models, extending the capabilities of LLMs to more modalities, will be an important research direction.

(4) Persistent Hallucinations

Hallucinations refer to the phenomenon where LLMs fabricate false or incorrect information based on prompts. For example, when we ask LLMs to generate an academic paper and a list of references, they often fabricate numerous seemingly "serious" but completely non-existent papers and research. Hallucination issues are inherent defects of LLMs and are major challenges in the research and application of LLMs. Especially in fields such as medicine and finance, where precision and correctness are highly emphasized, the presence of hallucinations may lead to very serious consequences. There are currently many studies that provide methods to reduce hallucinations, such as limiting prompts or guiding generation through RAG (Retrieval-Augmented Generation), but they can only partially reduce hallucinations and cannot completely eliminate them.

In addition to the above points, LLMs have many other characteristics worth studying, such as the three-stage training process of LLMs and the self-reflection of LLMs discussed in the next section. Here, we will not list them one by one.

4.2 How to Train an LLM

In the previous section, we analyzed the definition of LLMs and their unique powerful capabilities. Through a larger scale of parameters and massive training corpora, LLMs have achieved remarkable emergent capabilities that far exceed traditional pre-trained models, showcasing strong in-context learning, instruction following, and step-by-step reasoning capabilities, bringing a new transformation to the field of NLP. So, through what steps can we train an LLM with emergent capabilities? What are the differences between training an LLM and training traditional pre-trained models?

Figure 4.1 Three stages of training an LLM

Generally, training a complete LLM requires going through three stages as shown in Figure 1 - Pretrain, SFT, and RLHF. In this section, we will elaborate on the three stages of training an LLM and analyze the process and core difficulties and precautions of each stage, helping readers understand theoretically what steps are needed to train an LLM.

4.2.1 Pretrain

Pretrain, or pre-training, is the most crucial and engineering-intensive first step in training an LLM. The pre-training of LLMs is very similar to that of traditional pre-trained models, both using massive unsupervised text to train randomly initialized model parameters. As we saw in Chapter 3, the majority of current LLMs have adopted the Decoder-Only GPT-like architecture (LLaMA architecture), and their pre-training tasks have inherited the classical pre-training task of the GPT model - Causal Language Model (CLM).

Causal language modeling, which is consistent with the original language model, trains the model to predict the next token based on the preceding text. The process and principles of CLM have been detailed in Chapter 3, so we will not elaborate further here. The core difference between LLM pre-training and traditional pre-trained models lies in the scale and resource consumption of pre-training.

According to the definition, the core characteristic of LLMs is that they have far more parameters than traditional pre-trained models and are pre-trained on much larger corpora. Traditional pre-trained models such as BERT have base and large versions. The BERT-base model consists of 12 encoder layers, with a hidden_size of 768, uses 12 heads as multi-head attention layers, and has a total of 110 million (110M) parameters; while the BERT-large model consists of 24 encoder layers, with a hidden_size of 1024, 16 heads, and a total of 340 million (340M) parameters. At the same time, BERT's pre-training used 3 billion (3B) tokens of corpus, and was trained on 64 TPUs for 4 days. In fact, relative to traditional deep learning models, BERT with 300 million parameters and 3 billion training data is already a formidable giant in terms of capability and resource consumption.

However, as mentioned earlier, LLMs generally have hundreds of billions or even trillions of parameters. Even the smallest LLMs in a broad sense usually have more than 1 billion (1B) parameters. For example, taking the pioneering work GPT-3 as an example, it has 96 decoder layers, a hidden_size of 12288, and 96 heads, with a total of 175 billion (175B) parameters, which is nearly three orders of magnitude larger than BERT. Even the currently popular small LLMs, such as Qwen-1.8B, have 24 decoder layers, a hidden_size of 2048, and 16 attention heads, with a total parameter count of 1.8 billion (1.8B).

Model	hidden_layers	hidden_size	heads	Total Parameters	Pre-training Data
BERT-base	12	768	12	0.1B	3B
BERT-large	24	1024	16	0.3B	3B
Qwen-1.8B	24	2048	16	1.8B	2.2T
LLaMA-7B	32	4096	32	7B	1T
GPT-3	96	12288	96	175B	300B

More importantly, LLMs often require larger-scale pre-training corpora. According to the Scaling Law proposed by OpenAI: C ~ 6ND, where C is the computational power, N is the model parameters, and D is the number of training tokens, it can be experimentally concluded that the number of training tokens should be 1.7 times the model parameters. Therefore, for a 175B GPT-3, 300B tokens of training data are needed. LLaMA further proposes that using 20 times the number of tokens to train the model can achieve optimal performance, so a 175B GPT-3 can be pre-trained with 3.5T tokens of data to achieve optimal performance.

Such a massive model parameter and pre-training data make the computational resources required for pre-training an LLM extremely large. In fact, even pre-training a 1B large model requires a multi-card distributed GPU cluster, and the model parameters, intermediate parameters of training, and training data must be partitioned through a distributed framework to complete the training over a period of days. Generally, a hundred-billion-level LLM requires 1024 A100 GPUs for over a month, and a ten-billion-level LLM generally requires 256 A100 GPUs for two to three days, with very high computational resource consumption.

Because of this, distributed training frameworks have also become an indispensable part of LLM training. The core idea of distributed training frameworks is data parallelism and model parallelism. Data parallelism refers to the situation where the model size can be accommodated by a single GPU memory, but increasing the batch size for training increases the memory usage, making it impossible to use a large batch size for training. At the same time, the amount of training data is very large, and using a single GPU for training would take too long.

Figure 4.2 Model and Data Parallelism

As shown in Figure 4.2, the model instances can run on different GPUs and different batches of data. After each forward pass, the gradients from all instances are collected and the gradient update is calculated. After updating the model parameters, the updated parameters are passed to all instances. That is, in data parallelism, the model parameters on each GPU remain consistent, and the total batch size for training is the sum of the batch sizes on each card.

However, when LLMs scale to hundreds of billions of parameters, the single GPU memory is often unable to store the entire model parameters. As shown in Figure 4.3, in this case, the model can be split across multiple GPUs, with each GPU storing different layers or different parts, thus achieving model parallelism.

Figure 4.3 Model Parallelism

Based on the ideas of data parallelism and model parallelism, various more efficient distributed methods have evolved, such as tensor parallelism, 3D parallelism, and ZeRO (Zero Redundancy Optimizer, zero redundancy optimizer). Currently, the mainstream distributed training frameworks include Deepspeed, Megatron-LM, and ColossalAI, among which Deepspeed is the most widely used.

The core strategy of Deepspeed is ZeRO and CPU-offload. ZeRO is a data parallelism scheme that optimizes memory usage on each card during data parallelism, thus supporting larger-scale models. ZeRO divides the memory occupied on each card during the model training phase into two categories:

• Model States, including model parameters, model gradients, and Adam state parameters of the optimizer. Assuming the model has 1M parameters, in mixed-precision training, this part requires 16M of space for storage, where Adam state parameters occupy 12M of storage space.
• Residual States, the memory occupancy other than model states, including activations, various caches, and memory fragments.

Regarding the above memory occupation, ZeRO proposes three progressively increasing optimization strategies:

1. ZeRO-1, which shards the Adam state parameters in the model states, i.e., each card stores only $\frac{1}{N}$ of the Adam state parameters, while other parameters remain stored on each card.
2. ZeRO-2, which continues to shard the model gradients, each card stores only $\frac{1}{N}$ of the model gradients and Adam state parameters, while the model parameters remain stored on each card.
3. ZeRO-3, which also shards the model parameters, each card stores only $\frac{1}{N}$ of the model gradients, model parameters, and Adam state parameters.

It can be seen that as the amount of sharded parameters increases, the memory occupied on each card also decreases. Of course, the increase in sharding also means an increase in communication overhead during training. Generally, the GPU utilization of ZeRO-1 is the highest, while that of ZeRO-3 is the lowest. The specific strategy used needs to be dynamically determined based on the available computing resources and the size of the model being trained.

In addition to the requirements for computing resources, the training data itself is also a major challenge in pre-training LLMs. Training an LLM requires at least hundreds of B or even up to T of pre-training corpus. According to research, the vast majority of knowledge that LLMs possess is learned during the pre-training process. Therefore, to make the trained LLM cover as wide a range of knowledge as possible, the pre-training corpus needs to be composed of data from various sources and mixed in certain proportions. Currently, the main open-source pre-training corpora include CommonCrawl, C4, Github, Wikipedia, etc. Different LLMs often add some private high-quality corpora to the open-source pre-training corpora, and then construct the pre-training dataset based on the best proportion obtained through their own experiments. In fact, data ratio has always been the "core secret" of pre-training LLMs, and different ratios can significantly affect the performance of the final model. For example, the following table shows the pre-training data and ratios of LLaMA:

Dataset	Proportion	Data Size (Disk size)
CommonCrawl	67.0%	3.3 TB
C4	15.0%	783 GB
Github	4.5%	328 GB
Wikipedia	4.5%	83 GB
Books	4.5%	85 GB
ArXiv	2.5%	92 GB
StackExchange	2.0%	78 GB

Training a Chinese LLM is more challenging in terms of training data. Currently, high-quality data is still mostly concentrated in the English domain, for example, the aforementioned Wikipedia, Arxiv, etc., are all English datasets; and in multi-language datasets like C4, English data also dominates. Currently, open-source Chinese LLMs such as ChatGLM and Baichuan have not opened their pre-training datasets. The only open-source Chinese pre-training datasets currently available are SkyPile (150B) and yayi2 (100B) released by Kunlun Tian Gong and Wenge Research, respectively, which are significantly behind the English open-source datasets.

The processing and cleaning of pre-training data is also an important part of LLM pre-training. Many studies have shown that the quality of pre-training data is often more important than the volume. Pre-training data processing generally includes the following steps:

1. Document preparation. Since the massive pre-training corpus is often obtained from the Internet, it usually requires obtaining natural language documents from crawled websites. Document preparation mainly includes URL filtering (filtering out harmful content based on web URLs), document extraction (extracting plain text from HTML), and language selection (determining the language of the extracted text).
2. Corpus filtering. The core purpose of corpus filtering is to remove low-quality, meaningless, and harmful content, such as garbled text and advertisements. Corpus filtering generally has two methods: one is model-based, which involves training a text classifier using a high-quality corpus for filtering; the other is heuristic-based, which generally defines quality metrics for web content manually and calculates the metric values of the corpus for filtering.
3. Corpus deduplication. Experiments show that a large amount of duplicate text can significantly affect the model's generalization ability, so deduplication of training corpus, i.e., removing documents with high similarity, is also an essential step. Deduplication is generally based on hash algorithms to calculate the similarity of documents within or across datasets, removing documents with similarity greater than a specified threshold; it can also be based on substring matching at the sequence level for precise deduplication.

Currently, there are many high-quality pre-training corpora that have been processed and frameworks specifically designed for pre-training data processing. For example, there is a pre-training dataset called RedPajama-1T, which collects and cleanses data based on the LLaMA approach, and a dataset called SlimPajama-627B, which is filtered and deduplicated based on RedPajama. Experiments have shown that the high-quality 627B Slimpajama dataset can achieve better results than the 1T RedPajama dataset.

4.2.2 SFT

Pretraining is the fundamental source of the powerful capabilities of LLMs. In fact, the vast amount of knowledge covered by LLMs is largely derived from the pre-training corpus. The core of LLM performance is also based on pretraining work. However, pretraining endows LLMs with capabilities, but we still need the second step to activate them. An LLM that has been pre-trained is like a scholar who reads extensively but does not seek to understand deeply. He can fluently continue the text for any strange question, but he doesn't know the meaning of the question itself, only "mechanically reciting."

This phenomenon essentially stems from the fact that the pre-training task of LLMs is the classic CLM, which trains the model to predict the next token. Without further fine-tuning, it cannot adapt to other downstream tasks or user instructions.

Therefore, we still need a second step to teach this bookish student how to use his knowledge, which is SFT (Supervised Fine-Tuning, supervised fine-tuning). Supervised fine-tuning is essentially the fine-tuning mentioned in Chapter 3, with a slight difference. For traditionally limited pre-trained models, we need to fine-tune them individually for each downstream task to train the model's performance on that task. For example, to solve the problem of text classification, we need to fine-tune BERT for text classification; to solve the problem of entity recognition, we need to fine-tune for the entity recognition task.

However, for powerful LLMs, we often no longer construct supervised data for specific downstream tasks for fine-tuning, but instead choose to train the model's "general instruction following ability," which is usually done through "instruction tuning" for SFT.

Instruction tuning refers to the input we train being various types of user instructions, and the output we want the model to fit is the response we hope the model gives after receiving the instruction. For example, a training sample can be:

input: Tell me the weather forecast today? output: According to the weather forecast, the weather today is sunny with clouds, the highest temperature is 26 degrees Celsius, the lowest temperature is 9 degrees Celsius, and the temperature difference between day and night is large. Please pay attention to keeping warm.

In other words, the main goal of SFT is to enable the model to gain generalized instruction following ability from various types and styles of instructions, i.e., to understand and respond to user instructions. Therefore, similar to Pretrain, the data quality and data ratio of SFT are also important factors determining the model's instruction following ability.

First, the quantity and coverage of instruction data. To enable the LLM to gain generalized instruction following ability, i.e., perform well on untrained instructions, it is necessary to collect a large number of diverse user instructions and corresponding responses for training the LLM. Generally, 500~1000 training samples per task can achieve good fine-tuning results. However, to enable the LLM to perform well on various task instructions, it is necessary to cover various types of task instructions in the training dataset, and also require a relatively large amount of training data. The SFT data volume of a well-performing open-source LLM is generally around several B tokens.

To improve the generalization ability of the LLM, the coverage of the instruction dataset is naturally better. However, the ratio of different types of instruction data is also a major challenge in LLM training. OpenAI's InstructGPT (the predecessor of ChatGPT) used ten types of instructions derived from user use of its API:

Instruction Type	Proportion
Text Generation	45.6%
Open-domain QA	12.4%
Brainstorming	11.2%
Chatting	8.4%
Text Transcription	6.6%
Text Summary	4.2%
Text Classification	3.5%
Others	3.5%
Domain-specific QA	2.6%
Text Extraction	1.9%

High-quality instruction datasets are difficult to obtain. Unlike the unsupervised corpora used in pretraining, SFT uses supervised corpora. In addition to designing a wide range of reasonable instructions, it also requires manual annotation of instruction responses and ensures the high quality of annotations. In fact, a large part of ChatGPT's success comes from its high-quality manually annotated data. However, manually annotated data is extremely expensive, and it is rare for companies to open-source manually annotated instruction datasets. To reduce data costs, some scholars have proposed using ChatGPT or GPT-4 to generate instruction datasets. For example, the classic open-source instruction dataset Alpaca is built by generating more instructions and responses based on some seed prompts using ChatGPT.

Generally, the instruction dataset used in SFT includes the following three keys:

{
    "instruction":"The user's instruction",
    "input":"Supplementary input required to execute the instruction, leave empty if none",
    "output":"What the model should respond"
}

For example, if our instruction is to translate the target text "Today is a nice day" into English, the sample can be structured as follows:

{
    "instruction":"Translate the following text into English:",
    "input":"Today is a nice day",
    "output":"Today is a nice day!"
}

At the same time, to allow the model to learn a different paradigm from pretraining, specific formats are often set during SFT. For example, the SFT format of LLaMA is:

### Instruction:\n{{content}}\n\n### Response:\n

where content is the specific user instruction, meaning that for each user instruction, it will be embedded into the content part of the above. The user instruction here not only refers to the "instruction" in the above example, but also the concatenation of the instruction and input, i.e., a complete instruction that the model can execute. For example, for the above example, the input that LLaMA receives should be:

Instruction:\nTranslate the following text into English: Today is a nice day\n\n### Response:\n

The output it needs to fit is:

Instruction:\nTranslate the following text into English: Today is a nice day\n\n### Response:\nToday is a nice day!

Note that since instruction tuning is essentially still CLM training for the model, but requires the model to understand and respond to instructions instead of simply predicting the next token, the model's prediction result is not only the output, but should be input + output, although the input part does not participate in the calculation of loss, but the response to the instruction is still implemented by predicting the next token.

However, as LLM capabilities continue to improve, the multi-turn dialogue capability of the model gradually becomes important. Multi-turn dialogue refers to the model's ability to refer to the history of previous conversations to generate responses during each conversation. For example, an LLM without multi-turn dialogue capability may have the following conversation history:

User: Hello, I am a member of the open-source organization Datawhale. Model: Hello, what can I help you with? User: Do you know what Datawhale is? Model: Sorry, I don't know what Datawhale is.

That is, the model cannot record historical information that the user has mentioned or the model has previously answered. If an LLM has multi-turn dialogue capability, its conversation history should be as follows:

User: Hello, I am a member of the open-source organization Datawhale. Model: Hello, what can I help you with? User: Do you know what Datawhale is? Model: Datawhale is an open-source organization.

Whether the model supports multi-turn dialogue has nothing to do with pretraining. In fact, the multi-turn dialogue capability of the model comes entirely from the SFT phase. If we want the model to support multi-turn dialogue, we need to construct the training data into a multi-turn dialogue format during SFT, allowing the model to use previous knowledge to generate answers. Suppose we currently need to construct a multi-turn dialogue:

<prompt_1><completion_1><prompt_2><completion_2><prompt_3><completion_3>

Constructing multi-turn dialogue samples generally has three methods:

1. Directly use the last model response as the output, and the previous history as the input, directly fitting the last response:

   input=<prompt_1><completion_1><prompt_2><completion_2><prompt_3><completion_3>
   output=[MASK][MASK][MASK][MASK][MASK]<completion_3>

2. Construct N rounds of dialogue into N samples:

   input_1 = <prompt_1><completion_1>
   output_1 = [MASK]<completion_1>
   
   input_2 = <prompt_1><completion_1><prompt_2><completion_2>
   output_2 = [MASK][MASK][MASK]<completion_2>
   
   input_3=<prompt_1><completion_1><prompt_2><completion_2><prompt_3><completion_3>
   output_3=[MASK][MASK][MASK][MASK][MASK]<completion_3>

3. Directly require the model to predict the output of each round of dialogue:

   input=<prompt_1><completion_1><prompt_2><completion_2><prompt_3><completion_3>
   output=[MASK]<completion_1>[MASK]<completion_2>[MASK]<completion_3>

It is clear that the first method will lose a lot of intermediate information, the second method causes a lot of redundant calculations, and only the third method is the most reasonable way to construct multi-turn dialogues. We can construct multi-turn dialogue samples in the third way because LLMs are essentially performing CLM tasks, conducting unidirectional attention calculations, so during prediction, it fits from left to right sequentially, and the prediction of the previous round's output does not affect the prediction of the next round's output. Currently, most LLMs use multi-turn dialogue formats for SFT.

4.2.3 RLHF

RLHF, short for Reinforcement Learning from Human Feedback, is a key step in training LLMs using reinforcement learning. Compared to SFT, which had already taken shape in GPT-3, RLHF is considered the most core breakthrough of ChatGPT compared to GPT-3. In fact, from a functional perspective, we can divide the training process of LLMs into two stages: pre-training and alignment. The core role of pre-training is to endow the model with massive knowledge, while alignment refers to making the model consistent with human values, thus outputting the content that humans expect. In this process, SFT aligns the LLM with human instructions, giving it the ability to follow instructions; while RLHF aligns the LLM with human values at a deeper level, making it meet the core standards of safety, usefulness, and harmlessness.

As shown in Figure 4.4, ChatGPT's technical report divides alignment into three stages, and the latter two stages of training RM and PPO training are the steps of RLHF:

Figure 4.4 Three stages of ChatGPT training

The idea of RLHF is to introduce reinforcement learning technology to allow LLMs to provide responses that are more satisfactory to humans through real-time human feedback. Reinforcement learning is another machine learning method distinct from supervised learning, primarily discussing how an agent can maximize the rewards it can obtain in a complex and uncertain environment. Reinforcement learning mainly consists of two parts: the agent and the environment. During the reinforcement learning process, the agent continuously acts and obtains feedback from the environment, adjusting its action strategy based on the feedback. Applied to the alignment of LLMs, it is essentially that for different questions, the LLM continuously generates corresponding responses, and human annotators continuously provide feedback on the LLM's responses, allowing the LLM to learn the responses that humans prefer and like.

RLHF is similar to the process of a student doing homework to improve his problem-solving skills. If we view the LLM as a capable student, Pretrain is teaching him all the basic knowledge, SFT is teaching him how to read the questions and solve them, and RLHF is like real practice. The LLM will continuously solve exercises based on the basic knowledge learned from Pretrain and the problem-solving skills learned from SFT, and then the teacher (human) will correct the LLM's exercises to let the LLM reflect on the wrong problem-solving methods and continuously reinforce the correct ones.

As shown in the figure above, RLHF consists of two steps: training RM and PPO training.

RM, Reward Model, is a reward model used to fit human preferences and provide feedback to LLMs. In the training of reinforcement learning, for each response of the LLM, RM will give a score, which reflects the degree to which the generated response conforms to human preferences. Then the LLM will optimize its training based on the score from the RM according to the principle of reinforcement learning. Therefore, RM is essentially a text classification model that outputs a scalar reward for a text, similar to the hidden layer output in text classification tasks. In specific implementation, RM is often the traditional LLM architecture (or BERT architecture) plus a classification layer, which is completely consistent with the LLM architecture used for text classification, except that it uses the hidden layer output instead of the final classification output.

However, when training RM, we often do not directly use the text and its corresponding scalar reward to train RM. Because to align with human preferences, the preference data used for RM training is usually manually annotated. However, since annotators may also have differences in values, numerical scalar rewards often amplify these differences, leading to inconsistent rewards for the same level of responses during training, making it difficult for the model to fit the correct scalar rewards. Therefore, we often rank the different responses for the same completion and convert the ranking into rewards.

For example, our training data often looks like:

{
    "prompt":"If you plan to steal from a store, do you think morning or evening is better?",
    "chosen":"It is illegal, I cannot provide advice",
    "rejected":"Considering the difference between the population trafficking and surveillance cameras at night is a good thing. Nighttime may be more conducive to avoiding surveillance cameras, but the store employees at night will be more likely to see you. On the other hand, since the daytime is usually less busy, it is easier to avoid being noticed by others. Regardless, it is important to closely monitor everything happening in the store, not to be careless. "
}

Where the prompt is the user's question, the chosen is the response that should be aligned with human preferences, and the rejected is the response that does not conform to human preferences. In training, the prompt will be concatenated with the chosen and rejected respectively to form chosen_example and rejected_example, which are then fed into the model through forward propagation to output a scalar reward. Then the model calculates the loss by maximizing the scalar difference between chosen_example and rejected_example and performs backpropagation to complete the training.

It is worth noting that the model used for RM training is often different in size from the final LLM. For example, OpenAI used a 175B LLM and a 6B RM. Also, the model used for RM can be a LM that has been SFT, or a RM trained from scratch based on preference data. Which is better remains undetermined.

After completing the training of RM, PPO algorithm can be used for reinforcement learning training. PPO, Proximal Policy Optimization, is a classic RL algorithm. In fact, other reinforcement learning algorithms can also be used during reinforcement learning training, but PPO algorithm is currently the most suitable algorithm for RLHF due to its maturity and low cost.

During the specific PPO training process, there are four models. As shown in Figure 4.5, there are two LLMs and two RMs. The two LLMs are the actor model and the ref model, which are not updated during parameter updates, both initialized from the LLM after SFT. The two RMs are the critic model and the reward model, which are not updated during parameter updates, both initialized from the RM trained in the previous step.

Figure 4.5 PPO Training Process

As shown in the figure above, the reinforcement learning training process using the PPO algorithm is as follows:

1. Initialize two models from the LLM after SFT as Actor Model and Ref Model; initialize two models from the trained RM as Reward Model and Critic Model;
2. Input a Prompt, and the Actor Model and Ref Model generate responses respectively;
3. Calculate KL divergence for Actor Response and Ref Response: $r_{KL} = -\theta_{KL}D_{KL}(\pi_{PPO}(y|x)||\pi_{base}(y|x))$ where $\pi_{PPO}(y|x)$ is the output of the Actor Model, and $\pi_{base}(y|x)$ is the output of the Ref Model, $\theta_{KL}D_{KL}$ is the method to calculate KL divergence;
4. Input the Actor Response into the Reward Model and Critic Model for scoring, where the Reward Model outputs the scalar reward of the response, and the Critic Model also outputs the cumulative reward (i.e., the cumulative reward from position i to the end);
5. Input the calculated KL divergence and the scores of the two models into the reward function to calculate the reward: $loss = -(kl_{ctl} \cdot r_{KL} + \gamma \cdot V_{t+1} - V_{t}) \log P(A_t|V_t)$, where $kl_{ctl}$ is the weight parameter controlling the impact of KL divergence on the result, $\gamma$ is the weight parameter controlling the impact of the score of the next time (i.e., the sample) on the result, $V_t$ is the score output of the Critic Model, and $A_t$ is the score output of the Reward Model;
6. Based on the rewards calculated by the reward function, update the parameters of the Actor Model and the Critic Model. Note that the parameter update methods of the Actor Model and the Critic Model are different, and here we will not elaborate further. Readers interested can study the relevant theories of reinforcement learning in depth.

In the above process, because four models are used, the memory usage is several times higher than SFT. For example, if both RM and LLM are 7B in size, the PPO process would require approximately 240G (four 80G A100 cards, each occupying 60G) of memory for model loading. Then, why do we need four models? The Actor Model and Critic Model are easy to understand, and the reason we also need to keep the original parameters unchanged Ref Model and Reward Model is to limit the model's updates from deviating too much from the original model, thus losing the capabilities bestowed by Pretrain and SFT.

Of course, such a large resource consumption and complex training process make RLHF a very high-barrier stage. Some scholars have proposed DPO (Direct Preference Optimization, direct preference optimization) from the perspective of supervised learning, which can be a low-barrier alternative to RLHF. The core idea of DPO is to transform the reinforcement learning problem of RLHF into a supervised learning problem to directly learn human preferences. DPO demonstrates that the constrained reward maximization problem can be optimized through single-phase policy training by using the mapping between the reward function and the optimal policy. That is, by learning the optimization objective proposed by DPO, human preferences can be directly learned without the need to train RM and perform reinforcement learning. Due to the use of supervised learning for training, DPO only needs two LLMs to complete the training, and the training process is much simpler than PPO, making it a simpler and more user-friendly alternative to RLHF. Why the optimization objective proposed by DPO can directly learn human preferences has been proven through a series of mathematical derivations, and interested readers can read further later. Here, we will not elaborate further.

Next, we will sequentially implement how to train an LLM from scratch, including pre-training, SFT, and RLHF.

References

[1] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.

[3] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei. (2020). Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361.

[4] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre. (2022). Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

[5] Qi Wang, Yiyuan Yang, Ji Jiang. (2022). Easy RL: Reinforcement Learning Tutorial . Beijing: Posts & Telecom Press. ISBN: 9787115584700. https://github.com/datawhalechina/easy-rl

[6] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290.

[7] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, Ji-Rong Wen. (2025). A Survey of Large Language Models. arXiv preprint arXiv:2303.18223.