Pre-trained Language Models

3.1 Encoder-only PLM

In the previous chapter, we elaborated on the attention mechanism that brought great changes to the NLP field and the Transformer model built using the attention mechanism. From then on, the milestone transformation of NLP models began. From the explanation of the Transformer in the previous section, we can see that the Transformer structure is mainly composed of the Encoder and Decoder parts, which have different structures and input-output.

Based on the characteristics of the Encoder and Decoder, the pre-training idea of ELMo was introduced, leading to different optimization ideas for the Transformer. For example, Google selected only the Encoder layer, stacked the Encoder layers, and proposed a different pre-training task - Masked Language Model (MLM), creating a representative model for Natural Language Understanding (NLU) tasks - BERT. While OpenAI chose the Decoder layer, used the original language model (LM) task, and by continuously increasing the model parameters and pre-training corpus, created the GPT series model, which is advantageous in Natural Language Generation (NLG) tasks and also the base model of today's large language models (LLMs). Of course, there is another idea that keeps both the Encoder and Decoder, creating a pre-trained Transformer model, such as the T5 model released by Google.

In this chapter, we will introduce the mainstream pre-trained models of the Transformer era in the order of Encoder-Only, Encoder-Decoder, and Decoder-Only, respectively introducing three core model architectures, the pre-training tasks chosen by each mainstream model, and their unique advantages, which are also the foundation of all mainstream LLMs today.

3.1.1 BERT

BERT, full name Bidirectional Encoder Representations from Transformers, is a pre-trained language model released by the Google team in 2018. It was published in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" and achieved the best performance (State Of The Art, SOTA) on seven natural language processing evaluation tasks, including GLUE and MultiNLI, making it a milestone achievement. Since the release of BERT, the mode of pre-training plus fine-tuning has become the mainstream for natural language processing tasks. Not only has BERT itself continued to be updated and improved to enhance model performance, but models such as MacBERT and BART have also been developed based on BERT for optimization. In short, BERT is a phased achievement in natural language processing, marking significant progress in various natural language processing tasks and establishing the dominance of pre-trained models. Until the emergence of LLMs, the dominant position in the NLP field shifted from BERT models. Even in the LLM era, BERT remains an essential part of understanding LLMs and NLP.

(1) Ideological Inheritance

BERT is a pre-trained model that unifies multiple ideas. The core ideas it inherits include:

• Transformer architecture. As we introduced in the previous chapter, in 2017, the paper "Attention is All You Need" proposed a Transformer model that completely uses the attention mechanism and discards the RNN and LSTM structures, bringing a new model architecture. BERT inherits the idea of Transformer, optimizes it on the basis of the Transformer model, stacks the Encoder structure, increases the model parameters, and creates a model architecture with unique talents for NLU tasks;
• Pre-training + fine-tuning paradigm. Also in 2018, the birth of ELMo marked the birth of the pre-training + fine-tuning paradigm. The ELMo model is based on a bidirectional LSTM architecture, pre-trains on training data based on a language model, and then fine-tunes for downstream tasks, showing more superior performance, leading the NLP field towards the research direction of pre-training + fine-tuning. BERT also adopts this paradigm and, by adjusting the model architecture to Transformer and introducing the pre-training task MLM suitable for text understanding that can capture deep bidirectional semantic relationships, brings the pre-training-fine-tuning paradigm to its peak.

Next, we will deeply analyze BERT from three aspects: model architecture, pre-training tasks, and downstream task fine-tuning, to analyze BERT's core ideas and advantages, helping you understand why BERT can achieve much better performance than previous models, and thus gain a deeper understanding of how LLMs can overcome BERT to open up a new era.

(2) Model Architecture —— Encoder Only

The model architecture of BERT is composed of the Encoder part of the Transformer stacked together, and its main structure is shown in Figure 3.1:

Figure 3.1 BERT Model Structure

BERT is a pre-trained model designed for NLU tasks, and its input is usually a text sequence, while the output is usually a Label, such as the positive and negative Labels for sentiment classification. However, just as the Transformer is a Seq2Seq model, the BERT, which is composed of stacked Encoders, is essentially also a Seq2Seq model, but without adding a specific task Decoder. Therefore, to adapt to various NLU tasks, a classification head prediction_heads is added at the top of the model to convert the multi-dimensional hidden states into the classification dimension (e.g., if there are two categories, the prediction_heads outputs a two-dimensional vector).

The entire model is composed of Embedding, Encoder, and prediction_heads:

Figure 3.2 BERT Model Schematic Structure

The input text sequence is first converted into input_ids by the tokenizer (tokenization), then into specific dimensional hidden_states through the Embedding layer, and then goes through the Encoder block. The Encoder block consists of N stacked Encoder Layers, and BERT has two sizes of models, namely the base version (12 Encoder Layers, 768 hidden layer dimensions, total parameter count 110M), and the large version (24 Encoder Layers, 1024 hidden layer dimensions, total parameter count 340M). After encoding by the Encoder, the top-level hidden_states are finally passed through the prediction_heads to obtain the final category probability, and after Softmax calculation, the model's predicted category can be calculated.

BERT uses WordPiece as the tokenization method. WordPiece is a subword segmentation algorithm based on statistics, whose core is to split words into subwords (e.g., "playing" -> ["play", "##ing"]). The merging operation is based on maximizing the likelihood of the language model. For languages like Chinese that are not space-separated, single Chinese characters are usually treated as atomic token units.

prediction_heads is actually a linear layer plus an activation function. Generally speaking, the output dimension of the last linear layer is equal to the number of task categories, as shown in Figure 3.3:

Figure 3.3 prediction_heads Structure

Each Encoder Layer is similar to the Encoder Layer structure in the Transformer, as shown in Figure 3.4:

Figure 3.4 Encoder Layer Structure

As shown in Figure 3.5, the hidden_states mapped through the Embedding layer enter the core attention mechanism, then add the original input through a residual connection, and then go through an Intermediate layer to get the final output. The Intermediate layer is a special term used by BERT, which is actually a linear layer plus an activation function:

Figure 3.5 Intermediate Structure

Note that BERT uses the GELU function as the activation function, whose full name is Gaussian Error Linear Unit activation function, which was first widely noticed since BERT. The calculation of GELU is:

$$GELU(x) = 0.5x(1 + tanh(\sqrt{\frac{2}{\pi}})(x + 0.044715x^3))$$

The core idea of GELU is to introduce the idea of random regularization into the activation function, deciding whether to discard or retain the neuron based on the probability distribution of the input. Regarding the principles and core ideas of GELU, they are not detailed here. Readers interested can learn them on their own.

The attention mechanism used by BERT is almost completely consistent with the self-attention mechanism in the Transformer's Encoder, but BERT integrates relative position encoding into the attention mechanism, treating relative position encoding as a trainable weight parameter, as shown in Figure 3.6:

Figure 3.6 BERT Attention Mechanism Structure

As shown in the figure, the difference between BERT's attention calculation process and the Transformer is that after calculating the attention scores, the Position Embedding layer is first used to integrate the relative position information. This Position Embedding layer is actually a linear matrix. By using trainable parameters to fit the relative position, it can fit more rich relative position information compared to the absolute position encoding used by the Transformer, but this also increases a lot of model parameters, and it is completely unable to handle inputs longer than the model's training length (for example, the maximum context length for BERT is 512 tokens).

Note: The original BERT (i.e., the one proposed in the paper) uses the same absolute position encoding as the Transformer. Subsequent improvements (including various variants of BERT) use the above relative position encoding. To help readers understand the model structure design more comprehensively, the improved version of BERT is selected here.

It can be seen that BERT's model architecture is built on the Transformer's Encoder, which is why it is said that BERT inherits the idea of the Transformer.

(3) Pre-training Tasks —— MLM + NSP

Compared to the model architecture that basically inherits the Transformer, BERT's greater innovation lies in the two new pre-training tasks it proposed - MLM and NSP (Next Sentence Prediction, next sentence prediction). The core advantage of the pre-training-fine-tuning paradigm is that by separating pre-training and fine-tuning, a model that has completed one pre-training can be applied to almost all downstream tasks simply by fine-tuning, as long as the cost of fine-tuning is low, even if the pre-training cost is several times or even tens of times higher than before, the model still has greater application value. Therefore, it is possible to further expand the model parameters and the amount of pre-training data, using massive pre-training corpus to let the model fit potential semantics and underlying knowledge, thereby allowing the model to gain powerful language understanding and generation capabilities through long-term, large-scale pre-training.

Therefore, the core requirement of pre-training data is to have a huge scale of data (hundreds of millions of tokens). There is no doubt that manually annotated fully supervised data is difficult to reach this scale. Therefore, pre-training data must be obtained from unsupervised corpora. This is why traditional pre-training tasks were all LM. LM uses the method of predicting the next word from the previous word, which can directly be applied to any text, for any text, we only need to mask the next word and input the previous word to the model to require it to predict, thus achieving LM training, so all texts on the Internet can be used for pre-training.

However, a major defect of the LM pre-training task is that it directly fits the left-to-right semantic relationship, but ignores the bidirectional semantic relationship. Although the Transformer uses position encoding to represent the position information of the text sequence, this is still essentially different from directly fitting the bidirectional semantic relationship. For example, BiLSTM (bidirectional LSTM model) often performs better than the LSTM model in semantic representation because BiLSTM fits the bidirectional semantic relationship through bidirectional LSTM. So is there a pre-training task that can both use massive unsupervised corpus and train the model to fit the ability of bidirectional semantic relationships?

Based on this idea, Jacob et al. proposed MLM, which is the masked language model as a new pre-training task. Compared to simulating human writing, MLM simulates "fill in the blank". The idea of MLM is simple: in a text sequence, randomly mask some tokens, then input all the unmasked tokens to the model, and require the model to predict the masked tokens. For example, the input and output can be:

Input: I <MASK> you because you are <MASK>
Output: <MASK> - love; <MASK> - wonderful

Since the model can use the context before and after the masked token to understand the semantics and predict the masked token, through this task, the model can fit the bidirectional semantics, and thus better achieve text understanding. Similarly, the MLM task does not require any manual annotation of the text, just needs to randomly mask the text, so it can also use all the text corpus on the Internet for pre-training. For example, BERT's pre-training used 3300 million words of corpus.

However, MLM also has its inherent defects. The LM task simulates the natural creation process of humans, and its training and downstream tasks are completely consistent, meaning that during training, the model predicts the next word based on the previous word, and during downstream task fine-tuning and inference, it is also the same. However, MLM is different. During downstream task fine-tuning and inference, there is no <MASK> that we manually added, we directly get the corresponding hidden state from the original text and then enter the classifier or other components. The inconsistency between pre-training and fine-tuning greatly affects the model's performance in downstream task fine-tuning. To address this issue, the authors improved the strategy of MLM.

During specific MLM training, 15% of the tokens in the training corpus are randomly selected for masking. However, these 15% of tokens are not all masked as <MASK>, but 80% of the time they are masked, 10% of the time they are replaced with any token, and 10% of the time they remain unchanged. The 10% that remains unchanged is to eliminate the inconsistency between pre-training and fine-tuning, and the 10% random replacement is mainly to force the model to maintain learning of the context information. Because if all the tokens are masked, the model only needs to process the masked positions, thus only learning to predict the token and losing the learning of the context. By introducing some random tokens, the model cannot determine which token to predict, thus forcing it to maintain the context representation distribution of each token, thus having the ability to represent the features of the sentence. And since the probability of random tokens is very low, it will not affect the model's actual language understanding ability.

In addition to MLM, BERT also proposed another pre-training task - NSP, i.e., next sentence prediction. The core idea of NSP is to target sentence-level NLU tasks, such as question-answer matching and natural language inference. Question-answer matching refers to inputting a question and several answers, requiring the model to find the correct answer to the question; natural language inference refers to inputting a premise and an inference, determining whether the inference is consistent with the premise. Such tasks require the model to fit the relationship between sentences at the sentence level, rather than the semantic relationship at the token level. Therefore, BERT proposed the NSP task to train the model to fit the semantic relationship at the sentence level.

The core idea of the NSP task is to require the model to judge whether the two sentences in a sentence pair are continuous context. For example, the input and output can be:

Input: Sentence A: I love you. Sentence B: Because you are wonderful. Output: 1 (continuous context)

Input: Sentence A: I love you. Sentence B: Because today's dinner is so nice. Output: 0 (not continuous context)

By requiring the model to judge the relationship between sentence pairs, it forces the model to fit the relationship between sentences, thus adapting to sentence-level NLU tasks. Similarly, since the positive samples of NSP can be randomly selected from unsupervised corpora, and the negative samples can be randomly selected by scrambling the sentences (as long as they are not originally continuous sentences), it can have almost unlimited training data.

During specific pre-training, BERT used 800M BooksCorpus and 2500M English Wikipedia corpora, with 90% of the data trained with a context length of 128, and the remaining 10% of the data trained with a context length of 512, totaling approximately 3.3B tokens. Its training hyperparameters are also worth noting, BERT's training corpus was 13GB in size, and it was trained for 1M steps (40 epochs) on a batch size of 256. In comparison, LLMs generally only train for one epoch and use a much larger batch size than 256.

It can be seen that compared to traditional non-pretrained models, the amount of training data has increased exponentially. Of course, more massive training data requires greater computational power. The base version of BERT and the large version were trained using 16 TPUs and 64 TPUs respectively, taking 4 days to complete.

(4) Downstream Task Fine-tuning

As a milestone achievement in the NLP field, one of the major significances of BERT is to formally establish the two-stage idea of pre-training and fine-tuning, i.e., to obtain general text understanding and generation capabilities through pre-training on massive unsupervised corpora, and then fine-tune on the corresponding downstream tasks. The key point of this idea is whether the powerful capabilities obtained through pre-training can be quickly transferred to the corresponding downstream tasks through low-cost fine-tuning.

To this end, BERT designed a more general input and output layer to adapt to multi-task transfer learning. For each input text sequence, BERT adds a special token <CLS> at the beginning. During subsequent encoding, this token represents the state of the entire sentence, i.e., the sentence-level semantic representation. During NSP pre-training, this feature vector of the token is used as the input to the final classifier.

After pre-training, for each downstream task, only a certain amount of fully supervised manually annotated data is needed to fine-tune the pre-trained BERT on that task. Fine-tuning is essentially the same as updating model parameters during training, except that it is trained on a specific task, with less training data, and a smaller batch size, with a smaller update amplitude. For most downstream tasks, the output of BERT can be directly used. For example, for a text classification task, the classification head in the model structure can be directly modified. For sequence labeling tasks, the hidden layer vectors of multiple layers of BERT can be integrated and the final labeling result can be output. For text generation tasks, the output of the encoder can also be directly decoded to obtain the final generated result. Therefore, BERT can be efficiently applied to various NLP tasks.

Since its release, BERT achieved SOTA results on 11 NLP tracks, becoming the undisputed leader in NLU. Subsequent models that achieved better performance on NLU tasks were all improved based on BERT. Until the LLM era, BERT could still achieve the best results on many NLU tasks with abundant annotated data. In fact, for certain specific tasks with rich training data and high throughput requirements, BERT is more practical than LLM.

3.1.2 RoBERTa

As a landmark work in the NLP era, BERT achieved SOTA results on multiple rankings and drove the entire NLP field towards pre-trained models. Based on BERT, a number of Encoder-Only pre-trained models with similar or identical model structures emerged, optimizing in terms of training data, pre-training tasks, and training parameters to achieve more powerful pre-trained models with better performance on downstream tasks. One of them is RoBERTa, also released by Facebook.

As mentioned earlier, one of the core advantages of the pre-training and fine-tuning paradigm is that it can use massive amounts of unsupervised data for pre-training. In traditional deep learning paradigms, for each task, we need to train a model from scratch, so we cannot use too large model parameters, otherwise it would require extremely large amounts of supervised data to allow the model to fit well, which is costly. However, in the pre-training and fine-tuning paradigm, we can use as much training data as possible during the pre-training stage, and only need one pre-trained model, and then fine-tune it on each downstream task with a small amount of supervised data. BERT used 13GB (3.3B tokens) of data for pre-training, which is an extremely large data scale compared to traditional NLP.

But is 13GB of pre-training data enough for BERT to fit sufficiently? If we use more pre-training corpus, can we further enhance the model's performance? Moreover, are the pre-training tasks and training hyperparameters chosen by BERT optimal? RoBERTa came into being.

(1) Optimization One: Removing the NSP Pre-training Task

RoBERTa's model architecture is completely consistent with BERT, which is to say, it uses the BERT-large (24 Encoder Layers, 1024 hidden layer dimensions, total parameter count 340M) model parameters. In terms of pre-training tasks, some scholars questioned whether the NSP task could improve model performance, as it was too simple, and adding it to the pre-training did not bring obvious benefits to downstream task fine-tuning, and even brought negative effects. RoBERTa set up four experimental groups:

1. MLM + NSP constructed from paragraphs: BERT's original pre-training task, the input is a pair of paragraphs, each paragraph includes multiple sentences to construct the NSP task;
2. MLM + NSP constructed from document pairs: one input constructs a pair of sentences, by increasing the batch size to match the token equivalence of the original input;
3. MLM across documents: remove the NSP task, an input is a complete sentence sampled from one or more documents, to make the input reach the maximum length (512), it may include multiple documents;
4. Single-document MLM: remove the NSP task, and limit an input to sample from a single document, also by increasing the batch size to match the token equivalence of the original input

The experimental results showed that the latter two groups significantly outperformed the former two groups, and the single-document MLM group performed best when fine-tuned on downstream tasks. Therefore, RoBERTa removed the NSP in pre-training and only used the MLM task.

At the same time, RoBERTa also made improvements to the MLM task itself. In BERT, the masking operation was completed during the data processing stage, so the same sample's masked <MASK> was consistent during later pre-training. Since BERT trained for 40 epochs, to make the training data more extensive, BERT performed four random masks, i.e., every 10 epochs, the training data for the model was exactly the same. RoBERTa moved the masking operation to the training stage, i.e., dynamic masking strategy, so that the masked positions in each epoch's training data were inconsistent. In the experiment, dynamic masking had only a slight advantage over static masking, but due to the efficiency and ease of implementation of dynamic masking, subsequent MLM tasks basically used dynamic masking.

(2) Optimization Two: Larger Scale Pre-training Data and Pre-training Steps

RoBERTa used a larger amount of unsupervised corpus for pre-training, in addition to the BookCorpus and English Wikipedia used by BERT, it also used CC-NEWS (English part of the news domain of the CommonCrawl dataset), OPENWEBTEXT (English web pages), and STORIES (story style subset of the CommonCrawl dataset), totaling 160GB of data, ten times larger than BERT.

At the same time, RoBERTa believed that a larger batch size could not only increase the optimization speed but also improve the performance of the task. Therefore, the experiment was conducted at an 8K batch size (compared to BERT's batch size of 256) and trained for 31K steps, which means that when the total number of training tokens was the same as BERT (3.3B), the model performance was better, thus proving the significance of a large batch size. On this basis, RoBERTa trained for 500K steps (approximately 66 epochs). At the same time, RoBERTa no longer used the strategy of training most of the time at 256 length and completing the training at 512 length as BERT did, but instead trained entirely at 512 length.

Of course, larger pre-training data, longer sequence lengths, and more training epochs require more computing resources during the pre-training phase. Training a RoBERTa, Meta used 1024 V100 (32GB memory) GPUs for one day.

(3) Optimization Three: Larger BPE Vocabulary

Unlike BERT's WordPiece algorithm, RoBERTa uses BPE as the tokenization strategy. BPE, or Byte Pair Encoding, is a tokenization method that uses subword pairs as the unit of tokenization. For example, the sentence "Hello World" might be split into "Hel, lo, Wor, ld" four subword pairs. For Chinese, which uses characters as the basic unit, it is generally split according to byte encoding. For example, in UTF-8 encoding, "我" is encoded as "E68891", which in BPE might be split into "E68" and "891" two byte pairs.

Generally speaking, the larger the BPE vocabulary, the better the encoding effect. Of course, since the embedding layer maps tokens from the vocabulary space to the hidden space (that is, the shape of the embedding is (vocab_size, hidden_size), the larger the vocabulary also brings an increase in model parameters.

The original BERT BPE vocabulary size was 30K, and RoBERTa chose a 50K vocabulary size to optimize the model's encoding capability.

Through the above three optimizations, RoBERTa successfully refreshed the SOTA of multiple downstream tasks based on the BERT architecture and once became the most popular pre-trained model in the BERT series. At the same time, RoBERTa's success also proved the importance of larger pre-training data and larger pre-training steps, which is one of the foundations of the emergence of LLMs.

3.1.3 ALBERT

Building on BERT, RoBERTa further explored the role of larger-scale pre-training. ALBERT, which is also optimized based on the BERT architecture, explores whether it is possible to reduce the model parameters while maintaining the model's capabilities. Through optimization of the model structure and improvement of the NSP pre-training task, ALBERT successfully achieved superior capabilities with smaller parameters. Although some of the improvement ideas proposed by ALBERT were not widely adopted in subsequent research, its method of reducing model parameters and the new pre-training task SOP still provided important references for the NLP field.

(1) Optimization One: Decoupling the Embedding Parameters

BERT and other pre-trained models have far more parameters than traditional neural networks. As mentioned earlier, BERT-large has 24 Encoder Layers, a hidden layer dimension of 1024, and a total parameter count of 340M. Among these, the parameter matrix of the Embedding layer has a dimension of $V*H$, where V is the vocabulary size of 30K and H is the hidden layer size of 1024, meaning that the Embedding layer parameters reached 30M. This setting also brings a bigger problem, that when Google tried to build a wider model (i.e., a larger hidden layer dimension), it found that the increase in the hidden layer dimension would cause a huge increase in the Embedding layer parameters. If the hidden layer dimension is increased to 2048, the Embedding layer parameters would expand to 61M, which is a huge increase in model computation costs.

From another perspective, the output vector of the Embedding layer is our dense vector representation of the text token. From the successful experience of Word2Vec, such word vectors do not need to be very high-dimensional. Word2Vec only used 100 dimensions and achieved good results. Therefore, the output of the Embedding layer may not need to be consistent with the hidden layer size.

Therefore, ALBERT decoupled the parameter matrix of the Embedding layer, allowing the output dimension of the Embedding layer to be independent of the hidden layer dimension. That is, after the Embedding layer, a linear matrix is added for dimension transformation. ALBERT sets the output of the Embedding layer to 128, so a $128*1024$ linear matrix is added after the Embedding layer to transform the output of the Embedding layer back to the hidden layer size. In other words, the parameters of the Embedding layer are reduced from $V*H$ to $V*E + E*H$, and when E is much smaller than H, this method can significantly optimize the parameters of the Embedding layer.

(2) Optimization Two: Parameter Sharing Across Layers

By analyzing the parameters of BERT, ALBERT found that the parameters of each Encoder layer are highly consistent. Due to the 24 Encoder layers bringing a huge number of model parameters, ALBERT proposed that each Encoder layer could share model parameters to reduce the number of model parameters.

In specific implementation, it is actually that ALBERT only initializes one Encoder layer. During the calculation process, it still performs 24 calculations, but each calculation is done through this one Encoder layer. Therefore, although it is a model with 24 Encoder calculations, there is only one Encoder parameter, which greatly reduces the number of model parameters. In this way, it is possible to greatly expand the hidden layer dimension to achieve a wider model with fewer parameters. ALBERT proves through experiments that compared to 334M BERT, the same 24 Encoder layers but with a hidden layer dimension of 2048 ALBERT (xlarge version) has only 59M parameters, but its performance is even better than BERT.

However, although the above optimization greatly reduces the model parameters and improves the model performance, it also has obvious shortcomings. Although ALBERT's parameters are much smaller than BERT, the training efficiency is only slightly better than BERT, because in the model setup, although the weights are shared among layers, the calculation still requires 24 Encoder layer calculations, meaning that the training and inference speed is still slower than BERT. This is also an important reason why ALBERT ultimately failed to replace BERT.

(3) Optimization Three: Proposing the SOP Pre-training Task

Similar to RoBERTa, ALBERT also believes that the NSP task is too simple and does not bring significant improvements to the model's performance during pre-training. However, unlike RoBERTa, which chooses to directly remove NSP, ALBERT chooses to improve NSP and increase its difficulty to optimize the model's pre-training.

In the traditional NSP task, the positive examples are sentence pairs composed of two consecutive sentences, and the negative examples are sentence pairs extracted from any two documents. The model can easily judge the positive and negative examples and cannot well learn deep semantics. The improvement proposed by the SOP task is that the positive examples are still composed of two consecutive sentences, but the negative examples are the reverse of these two. That is, the model not only needs to fit the relationship between the two sentences but also needs to learn the order relationship, which greatly increases the difficulty of pre-training. For example, compared to the NSP task example mentioned earlier, the SOP task example is as follows:

Input: Sentence A: I love you. Sentence B: Because you are wonderful. Output: 1 (positive sample)

Input: Sentence A: Because you are wonderful. Sentence B: I love you. Output: 0 (negative sample)

ALBERT proves through experiments that the SOP pre-training task significantly improves the model's performance. The model trained with MLM + SOP performs better than the model trained with only MLM, which in turn performs better than the model trained with MLM + NSP.

Through the above three optimizations, ALBERT successfully achieved stronger performance with smaller parameters. Although the training and inference efficiency caused by its architecture limited the further development of the model, the idea of building a wider model still provides reference value for many more powerful models.

As the king of NLP in the pre-training era, BERT and BERT series models have played an extremely important role in multiple NLP tasks. In addition to the aforementioned RoBERTa and ALBERT, there are many other emerging models that optimize BERT from other higher angles, including ERNIE that further improves the pre-training task, DistilBERT that is a small model through distillation, and XLM that focuses on multilingual tasks, and this article will not elaborate further. Models based on the Encoder-Only architecture, such as BERT, are not the only variations of the Transformer. Next, we will introduce another mainstream architecture of the Transformer, the Encoder-Decoder architecture represented by T5.

3.2 Encoder-Decoder PLM

In the previous section, we learned about the Encoder-Only structure model, mainly introducing BERT's model architecture, pre-training tasks, and downstream task fine-tuning. BERT is a Transformer-based Encoder-Only model that learns bidirectional semantic relationships in text through pre-training tasks MLM and NSP, thereby achieving excellent performance in downstream tasks. However, BERT also has some problems, such as the inconsistency between the MLM task and downstream task fine-tuning, and the inability to handle inputs longer than the model's training length. To solve these problems, researchers proposed Encoder-Decoder models, solving these issues by introducing the Decoder part, and also bringing new ideas and methods to the NLP field.

In this section, we will learn about the Encoder-Decoder structure model, mainly introducing T5's model architecture and pre-training tasks, as well as T5's first proposal of the unified thinking of NLP.

3.2.1 T5

T5 (Text-To-Text Transfer Transformer) is a pre-trained language model proposed by Google, which simplifies the model design and task processing by unifying all NLP tasks into text-to-text conversion problems. T5 is based on the Transformer architecture, containing encoder and decoder parts, using self-attention and multi-head attention to capture global dependencies, using relative position encoding to handle position information in long sequences, and including feed-forward neural networks in each layer to further process features.

The unified thinking of T5 unifies different NLP tasks such as text classification, question answering, translation, etc., into input text to output text conversion, which simplifies model design, parameter sharing, and training processes, improving the model's generalization ability and efficiency. Through this unified processing method, T5 not only reduces the task-specific model debugging work, but also can use the same data processing and training framework, greatly enhancing the performance and convenience of multi-task learning. Next, we will introduce the T5 model from three aspects: model structure, pre-training tasks, and the unified thinking.

(1) Model Structure: Encoder-Decoder

BERT adopts an Encoder-Only structure, containing only the encoder part; while GPT adopts a Decoder-Only structure, containing only the decoder part. T5 adopts an Encoder-Decoder structure, where the encoder and decoder are both based on the Transformer architecture design. The encoder is used to process the input text, and the decoder is used to generate the output text. The encoder and decoder interact with each other through the attention mechanism to achieve the conversion from input text to output text. Its main structure is shown in Figure 3.7:

Figure 3.7 T5 Model Detailed Structure

As shown in Figure 3.8, from the overall perspective, the T5 model structure includes the Tokenizer part and the Transformer part. The Tokenizer part is mainly responsible for converting the input text into the input format acceptable by the model, including tokenization, encoding, and other operations. The Transformer part is divided into EncoderLayers and DecoderLayers, which are composed of a series of small Blocks. Each Block contains multi-head attention mechanisms, feed-forward neural networks, and Norm layers. The design of the Block makes the model more flexible, like LEGO, and can adjust the number and layers of Blocks according to the complexity of the task and the size of the dataset.

Figure 3.8 T5 Model Overall Structure

The Encoder and Decoder parts of the T5 model are both based on the Transformer architecture design, mainly including Self-Attention and feed-forward neural networks. Self-Attention is used to capture global dependencies in the input sequence, and the feed-forward neural network is used to process the non-linear transformation of features.

Unlike the Encoder, the Decoder also contains the Encoder-Decoder Attention structure, which is used to capture the dependencies between the input and output sequences. These two types of Attention structures are almost completely consistent, with differences only in the position encoding and Mask mechanism. As shown in Figure 3.9, the structures of the Encoder and Decoder are as follows:

Figure 3.9 Encoder and Decoder

The Self-Attention mechanism of T5 is the same as that of BERT, both are designed based on the Self-Attention mechanism. Self-Attention is a global dependency modeling method that captures the global dependencies in the input sequence by calculating the similarity between Query, Key, and Value. Encoder-Decoder Attention differs only in position encoding and Mask mechanism, mainly to distinguish between input and output sequences. As shown in Figure 3.10, the Self-Attention structure is as follows:

Figure 3.10 Self-Attention Structure

Unlike the original Transformer model, the LayerNorm of the T5 model uses RMSNorm, which normalizes the activation values of each hidden layer by calculating the root mean square (Root Mean Square) of each neuron. RMSNorm has a simpler parameter setting compared to Layer Normalization, with only one learnable parameter, which can better adapt to different tasks and datasets. The RMSNorm function can be expressed mathematically as:

$$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{n}\sum_{i=1}^{n}x_i^2 + \epsilon}} \cdot \gamma$$

Where:

• $x_i$ is the $i$-th element of the input vector
• $\gamma$ is the learnable scaling parameter
• $n$ is the dimensionality of the input vector
• $\epsilon$ is a small constant for numerical stability (to avoid division by zero)

This normalization helps stabilize the learning process by ensuring that the scale of the weights does not become too large or too small, which is especially useful in deep learning models with many layers.

(2) Pre-training Tasks

The pre-training tasks of the T5 model are a key component, enabling the model to learn rich language representations that can be transferred to various downstream tasks during the fine-tuning process. The training dataset used is a large-scale text dataset containing various types of text data, such as Wikipedia, news, books, etc. After careful processing, a 750GB dataset called C4 was generated and has been open-sourced in TensorflowData.

We can briefly summarize the pre-training tasks of T5, which mainly include the following parts:

• Pre-training task: The pre-training task of the T5 model is MLM, also known as the BERT-style objective. Specifically, 15% of the tokens in the input text are randomly masked, and the model is required to predict these masked tokens. This process does not require labels and can be performed on a large amount of unlabeled text.
• Input format: During pre-training, T5 converts the input text into a "text-to-text" format. For a given text sequence, some tokens are randomly selected for masking and replaced with special placeholders (tokens). Then, the masked token sequence is used as the output target of the model.
• Pre-training dataset: T5 uses its own large-scale dataset "Colossal Clean Crawled Corpus" (C4), which extracts a large amount of clean English text from Common Crawl. The C4 dataset has undergone some cleaning, removing meaningless text, duplicate text, etc.
• Multi-task pre-training: T5 also attempted to mix multiple tasks for pre-training, not just a single MLM task. This helps the model learn more general language representations.
• Pre-training to fine-tuning transition: After pre-training, the T5 model is fine-tuned on downstream tasks. During fine-tuning, the model is trained on task-specific datasets and adjusts the decoding strategy according to the task.

Through large-scale pre-training, the T5 model can learn rich language knowledge and obtain strong language representation capabilities, achieving excellent performance on multiple NLP tasks. Pre-training is one of the key factors for the success of T5.

(3) Unified Thinking

A core concept of the T5 model is the "unified thinking," that is, all NLP tasks can be unified as text-to-text tasks, which has a profound impact on the field of natural language processing. The design philosophy is to convert all different types of NLP tasks (such as text classification, translation, text generation, question answering, etc.) into a unified format: input and output are both pure text.

For example:

• For a text classification task, the input can be "classify: This is a great product," and the output is "positive";
• For a translation task, the input can be "translate English to French: How are you?", and the output is "Comment ça va?".

T5 pre-trains on a large amount of text data and then fine-tunes on specific tasks. This process is similar to models like BERT and GPT, but T5 unifies the pre-training and fine-tuning stages into a text-to-text format, making it more adaptable to various tasks.

We can understand T5's unified thinking more intuitively through Figure 3.11:

Figure 3.11 T5's Unified Thinking

For different NLP tasks, a task description prefix is added before each input to clearly specify the type of the current task. This not only helps the model learn common features between different tasks during the pre-training phase but also facilitates rapid adaptation to specific tasks during the fine-tuning phase. For example, the task prefix can be "summarize: " for summary tasks, or "translate English to German: " for translation tasks.

T5's unified thinking simplifies the task processing workflow by unifying all NLP tasks into a text-to-text format, enhancing the model's generality and adaptability. This idea not only promotes the development of natural language processing technology but also provides a more convenient and efficient solution for practical applications.

3.3 Decoder-Only PLM

In the previous two sections, we explained the two model architectures derived from the Transformer - the Encoder-Only model represented by BERT and the Encoder-Decoder model represented by T5. Naturally, it can be imagined that apart from these two architectures, there is another model architecture - Decoder-Only, which is composed solely of stacked Decoders.

In fact, the Decoder-Only architecture is the basic architecture of the current hot LLMs, and all LLMs are basically Decoder-Only models (excluding non-Transformer architectures like RWKV and Mamba). The model that sparked the LLM craze, ChatGPT, is the culmination of the Decoder-Only series model, the GPT series model. Currently, the open-source LLM basic architecture, the LLaMA model, is also an optimized development of the GPT model architecture. Therefore, in this section, we will not only thoroughly analyze the principles, architecture, and characteristics of the representative model of Decoder-Only, GPT, but also delve into the current mainstream open-source LLMs, analyze their structures and characteristics, combine the previous analysis of other Transformer series models, and help everyone deeply understand how LLMs, which are expected to be the path to AGI, have evolved step by step from traditional PLMs.

First, let's learn about the representative model that opens the door to the LLM world - GPT, released by OpenAI.

3.3.1 GPT

GPT, which stands for Generative Pre-Training Language Model, is a pre-trained language model released by the OpenAI team in 2018. Although the academic community generally recognizes BERT as the representative of the pre-trained language model era, the model that first explicitly proposed the pre-training and fine-tuning idea is actually GPT. GPT proposed the concept of general pre-training, which is to pre-train on massive unsupervised corpora, and then fine-tune on each specific task, thereby achieving significant benefits for these tasks. Although it initially did not achieve the sensational results due to slightly inferior performance to BERT released shortly after, and did not make the Decoder-Only architecture used by GPT become the mainstream in academia, the OpenAI team firmly chose to continue expanding the pre-training data and increasing the model parameters, continuously optimizing the GPT architecture, eventually achieving the foundation of the LLM era with GPT-3 released in 2020, and the ChatGPT based on GPT-3 successfully opened the door to a new era, becoming the strongest competitor and currently the biggest winner in the LLM era.

In this section, we will take GPT as an example, and analyze GPT and the representative Decoder-Only models from three aspects: model architecture, pre-training tasks, and the development history of the GPT series models, and further introduce the current mainstream LLM architecture - LLaMA.

(1) Model Architecture —— Decoder Only

Figure 3.12 GPT Model Structure

As shown in Figure 3.12, the overall structure of GPT is somewhat similar to BERT, but instead of using the Encoder, it selects the Decoder to stack the model structure. Since the Decoder-Only structure is naturally suitable for text generation tasks, compared to the BERT, which is more tailored for NLU tasks, the model design of GPT and T5 is more suitable for NLG tasks and Seq2Seq tasks. Similarly, for a natural language text input, it is first tokenized and converted into the corresponding dictionary sequence numbers of input_ids.

The input input_ids are first processed through the Embedding layer, and then through Positional Embedding for position encoding. Unlike BERT, which chose a trainable fully connected layer as position encoding, GPT retained the classic Sinusoidal position encoding of the Transformer, which uses trigonometric functions for absolute position encoding. Here, we will not elaborate further, and interested readers can refer to the detailed analysis of the Transformer model in Chapter 2.

After encoding into hidden_states through the Embedding layer and Positional Embedding layer, it can enter the decoder (Decoder), the first generation of GPT model and the original Transformer model are similar, choosing 12 decoder layers, but in the internal of the decoder layer, compared to the original Transformer decoder layer's dual attention layer design, GPT's decoder layer is more like an encoder layer. Since there is no encoder encoding input, the decoder layer retains only one masked attention layer, and moves the LayerNorm layer from after the attention layer in the Transformer to before the attention layer. After the hidden_states enters the decoder layer, it first undergoes LayerNorm, then performs masked attention calculation, then goes through a residual connection and another LayerNorm into the MLP to get the final output.

Since there is no encoder encoding result, the masked attention in the decoder layer is also self-attention calculation. That is, for an input hidden_states, it generates query, key, and value through three parameter matrices, rather than the encoder output as key and value as in the Transformer's decoder. The subsequent attention calculation process is similar to BERT, except that after obtaining the attention weights, a mask matrix is used to mask the attention weights of future tokens, thus limiting each token to only pay attention to the attention of previous tokens, thereby achieving masked self-attention calculation.

Another structural difference is that GPT's MLP layer does not choose a linear matrix for feature extraction, but instead chooses two one-dimensional convolution kernels to extract, but in terms of effect, these two are not much different. After N decoder layers, the hidden_states are finally mapped to the vocabulary dimension through a linear matrix, which can be transformed into natural language tokens, thus generating the target sequence.

(2) Pre-training Task —— CLM

The Decoder-Only model structure is often more suitable for text generation tasks, so Decoder-Only models often choose the most traditional and direct pre-training task - causal language model, Casual Language Model, abbreviated as CLM.

CLM can be seen as a direct extension of the N-gram language model. The N-gram language model predicts the next token based on the previous N tokens, while CLM predicts the next token based on all the previous tokens of a natural language sequence, and repeatedly performs this process to achieve the generation of the target text sequence. In other words, CLM is a classic completion form. For example, the input and output of CLM can be:

input: Today's weather output: Today's weather is

input: Today's weather is output: Today's weather is very

Therefore, for an input target sequence length of 256, expecting an output sequence length of 256, the model will perform 256 calculations, each time using the first 256 tokens, 257 tokens (input + predicted first token)... until finally generating a sequence of length 512, with the first 256 tokens being the input and the latter 256 tokens being the desired model output.

As mentioned earlier, BERT could achieve major breakthroughs with the pre-training + fine-tuning paradigm because it chose the MLM and NSP, which could be directly trained on massive unsupervised corpora. Clearly, CLM is a more direct pre-training task, which is inherently aligned with the habits of human writing natural language texts and matches the downstream tasks directly, making it more direct than the MLM task, and can be directly applied to any natural language text. Therefore, CLM can also use massive natural language corpora for large-scale pre-training.

(3) Development of the GPT Series Models

Since the release of GPT-1, OpenAI has always believed in the model structure of Decoder-Only and the idea that "size is everything", continuously expanding the pre-training corpus, model size, and making some small optimizations and corrections to the model to constantly explore more powerful pre-trained models. From the suppressed GPT-1 to the not sufficiently noticed GPT-2, to the one that triggered emergent abilities and brought the era of large models, GPT-3, and finally the cross-era ChatGPT, OpenAI has proven the correctness of its ideas through decades of efforts.

The following table summarizes the changes in the model structure and pre-training corpus size from GPT-1 to GPT-3:

Model	Decoder Layer	Hidden_size	Attention Head Number	Attention Dimension	Total Parameters	Pre-training Corpus
GPT-1	12	3072	12	768	0.12B	5GB
GPT-2	48	6400	25	1600	1.5B	40GB
GPT-3	96	49152	96	12288	175B	570GB

GPT-1 is the pioneer of the GPT series and the first pre-trained model using the Decoder-Only structure. However, the model size and pre-training data of GPT-1 are relatively small, inheriting the traditional Transformer model structure, using 12 Decoder Blocks and a hidden layer dimension of 768, with a total parameter count of only 117 million (0.12B), pre-trained on a 5GB BooksCorpus dataset. It can be seen that the parameter scale and pre-training scale of GPT-1 are roughly comparable to BERT-base, but its performance is somewhat worse than BERT-base, which is also the reason why the GPT series model could not become the representative of the pre-trained language model era.

GPT-2 is the product of OpenAI's further exploration of the multi-task learning capabilities of the pre-trained language model based on GPT-1. The model structure of GPT-2 is roughly similar to GPT-1, but the model parameter scale is expanded, and the Post-Norm is changed to Pre-Norm (i.e., performing LayerNorm calculation first, then entering the attention layer calculation). The core reason for these changes is that with the increase in the number of model layers and the size of the model, the risk of gradient disappearance and explosion is also increasing. To make the model gradients more stable, the above structures were optimized.

The core improvement of GPT-2 is to significantly increase the pre-training corpus and model size. The number of Decoder Blocks in GPT-2 reached 48 (note that GPT-2 released four specifications of models, and here we only refer to the largest specification of GPT-2), the hidden layer dimension reached 1600, and the total model parameter count reached 1.5 billion (1.5B), and it was pre-trained on a 40GB WebText dataset. Whether in terms of model structure or pre-training scale, it exceeded the first generation by an order of magnitude.

Another major breakthrough of GPT-2 is to focus primarily on zero-shot (zero-sample learning), that is, not fine-tuning the model directly to solve the task. For example, in the traditional pre-training and fine-tuning paradigm, to solve a problem, we usually need to collect hundreds or thousands of training samples, and fine-tune the pre-trained language model on these training samples to solve the problem. Zero-shot, however, emphasizes not using any training samples, directly solving the problem by describing the problem to the pre-trained language model. The idea of zero-shot is naturally more advanced and more efficient than the pre-training and fine-tuning paradigm, but in the GPT-2 era, the model capability was not sufficient to support good zero-shot performance. In the era of large models, zero-shot and its extension, few-shot (few-sample learning), gradually became the mainstream.

GPT-3 is a further demonstration of OpenAI's core idea of "powerful and robust", and is also the pioneering work of LLMs. Building on GPT-2, OpenAI further increased the model size and pre-training data volume, with a total parameter count of 175B, which is undoubtedly a "large language model." In terms of model structure, there was basically no major improvement, but due to the huge model size, sparse attention mechanisms were used instead of traditional attention mechanisms. In terms of pre-training data, it sampled from large corpora such as CC, WebText, and Wikipedia, totaling 45T of data, which was cleaned to 570GB. According to estimates, GPT-3 needs to be trained on a distributed training cluster of 1024 A100 (80GB memory) GPUs for one month.

The reason why GPT-3 is considered the pioneering work of LLMs is not only because its massive size highlights the emergence of capabilities, but also because it proposed the important idea of few-shot. Few-shot is an improvement on zero-shot, and researchers found that even with a 175B-sized GPT-3, it is still quite difficult to achieve good performance on zero-shot. Few-shot is a compromise between zero-shot, aiming to provide the model with a few examples to teach it to complete the task. Few-shot generally adds 3~5 examples to the prompt (i.e., the input of the model) to help the model understand. For example, for a sentiment classification task:

zero-shot: Please determine whether the sentiment of 'This is a great opportunity' is positive or negative. If it is positive, output 1; otherwise, output 0.

few-shot: Please determine whether the sentiment of 'This is a great opportunity' is positive or negative. If it is positive, output 1; otherwise, output 0. You can refer to the following examples to determine: 'Your performance is very good' - 1; 'That's terrible' - 0; 'That's a great idea' - 1.

By providing the model with a few examples, the model can achieve much better performance than zero-shot. Few-shot is also called in-context learning, which means letting the model learn the solution to the problem from the examples in the provided context. GPT-3's strong ability in few-shot has brought important progress to NLP. If for most tasks, it is possible to let the model solve them by manually constructing 3~5 examples, the efficiency will be much higher than the traditional pre-training and fine-tuning paradigm, meaning that the further application of NLP becomes possible - and this is precisely the core advantage of LLMs.

Based on the GPT series models, through the three-stage training of pre-training, instruction fine-tuning, and human feedback reinforcement learning, OpenAI released the cross-era ChatGPT, triggering the trend of large models. It is also on the basis of GPT-3 and ChatGPT that the release of LLaMA, ChatGLM, and other models further revealed the endless potential of LLMs. In the next section, we will deeply analyze the universal architecture of current LLMs - LLaMA.

3.3.2 LLaMA

LLaMA model is a series of large pre-trained language models developed by Meta (formerly Facebook). From LLaMA-1 to LLaMA-3, the LLaMA series models demonstrate the evolution of large-scale pre-trained language models and their significant potential in practical applications.

(1) Model Architecture —— Decoder Only

Like the GPT series models, LLaMA models are also pre-trained language models based on the Decoder-Only architecture. The overall structure of the LLaMA model is similar to the GPT series models, but differs in model scale and pre-training dataset. Figure 3.13 is the architecture diagram of the LLaMA model:

Figure 3.13 LLaMA-3 Model Structure

Like GPT, the processing flow of LLaMA also starts with encoding the input text through the tokenizer into a series of input_ids. These input_ids are the data format that the model can understand and process. Next, these input_ids are converted through the embedding layer, where each input_id is mapped to a vector in a high-dimensional space, i.e., word vectors. At the same time, the positional information of the input text is encoded through the positional embedding layer to ensure that the model can understand the contextual information of the word order.

Thus, after combining the input_ids with the embedding layer and the positional embedding layer, the hidden_states are formed. The hidden_states contain the semantic and positional information of the input text, and serve as the basis for the model's subsequent processing. The hidden_states are then input into the model's decoder layer.

In the decoder layer, the hidden_states undergo a series of processing, which are composed of multiple decoder blocks. Each decoder block is the core component of the model, responsible for in-depth analysis and transformation of the hidden_states. In each decoder block, first is a masked self-attention layer. In this layer, the model calculates the query, key, and value vectors separately. These vectors are obtained by linear transformation of the hidden_states, and they are the basis for calculating attention weights. Then, the softmax function is used to calculate the attention score, which reflects the strength of the relationship between different positions. Through the attention score, the model can determine how much attention should be given to the hidden_states of different positions when generating the current word. Then, the value vector is multiplied by the attention score to get the weighted value, which is the result of the attention.

After completing the masked self-attention layer, the hidden_states enter the MLP layer. In this multi-layer perceptron layer, the model further extracts features from the hidden_states through two fully connected layers. The first fully connected layer maps the hidden_states to an intermediate dimension, and then applies an activation function for non-linear transformation, increasing the model's non-linear capability. The second fully connected layer maps the features back to the original hidden_states dimension.

Finally, after processing through multiple decoder blocks, the hidden_states are mapped through a linear layer to the final output, which has the same dimension as the vocabulary. Thus, the model can generate the probability distribution of the target sequence based on the hidden_states, and then generate the final output sequence through sampling or greedy decoding. This process demonstrates the strong sequence generation capability of the LLaMA model.

(2) Development History of LLaMA Models

LLaMA-1 Series:

• Meta released LLaMA-1 in February 2023, including versions with 7B, 13B, 30B, and 65B parameters.
• These models were pre-trained on over 1T tokens of corpus, with the largest 65B parameter model trained for nearly 21 days on 2,048 A100 80G GPUs.
• LLaMA-1 quickly became one of the most popular large models in the open-source community due to its open-source nature and excellent performance.

LLaMA-2 Series:

• In July 2023, Meta released LLaMA-2, including versions with 7B, 13B, 34B, and 70B parameters, with the exception of the 34B model, others have been open-sourced.
• LLaMA-2 expanded the pre-training corpus to 2T tokens and doubled the context length from 2,048 to 4,096.
• Introduced technologies such as grouped query attention mechanism (Grouped-Query Attention, GQA).

LLaMA-3 Series:

• In April 2024, Meta released LLaMA-3, including versions with 8B and 70B parameters, and also revealed that the 400B LLaMA-3 is still being trained.
• LLaMA-3 supports 8K long text and uses a more efficient tokenizer with a vocabulary size of 128K.
• Used over 15T tokens of pre-training corpus, which is more than seven times that of LLaMA-2.

LLaMA models are renowned for their technological innovations, multiple parameter versions, large-scale pre-training, and efficient architectural design. The model supports parameter counts ranging from 700 million to hundreds of billions, adapting to different application needs. LLaMA-1 gained popularity quickly in the community due to its open-source nature and excellent performance, while LLaMA-2 and LLaMA-3 further enhanced the model's performance and application range by introducing grouped query attention mechanism and supporting longer text input. In particular, LLaMA-3 achieved significant progress in multilingual and multitask processing by adopting a 128K vocabulary tokenizer and a 15T token training dataset. Meta's continuous attention to model safety and community support indicates that LLaMA will continue to be a driving force in AI technology development, promoting technological applications and innovations worldwide.

3.3.3 GLM

The GLM series of models are one of the mainstream Chinese LLMs developed by Zhipu, including ChatGLM1, 2, 3, and GLM-4 series models, covering scenarios such as instruction understanding and code generation, and achieving SOTA performance on multiple Chinese evaluation sets.

ChatGLM-6B is the founding model of the GLM series, and is also the earliest open-source Chinese LLM in China in 2023, as well as the first LLM to propose a unique model architecture different from GPT and LLaMA. Throughout the development of Chinese LLMs, GLM has a unique and significant technical significance. This section will briefly describe the development of the GLM series and introduce its unique technical ideas different from GPT and LLaMA series models.

(1) Model Architecture - Slight Modifications Compared to GPT

GLM was initially a general language model base developed by the Computer Science Department of Tsinghua University. Its core idea is to add the MLM idea to the traditional CLM pre-training task, thus building a unified model that performs well on both NLG and NLU tasks.

In terms of the overall model structure, GLM is roughly similar to GPT, both being Decoder-Only structures, with three minor differences:

1. Using Post Norm instead of Pre Norm. Post Norm refers to performing residual calculation first and then LayerNorm calculation; while models like GPT and LLaMA use Pre Norm, which means performing LayerNorm calculation first and then residual calculation. Compared to this, Post Norm has a stronger effect on parameter regularization due to normalization after the residual, thus improving the model's robustness. Pre Norm, on the other hand, due to some parameters directly added after, does not require regularization for these parameters, which can prevent the model from exploding or vanishing gradients. Therefore, for larger models, it is generally believed that Pre Norm performs better. However, the GLM paper proposed that using Post Norm can avoid numerical errors in LLMs (although mainstream LLMs still use Pre Norm);

2. Using a single linear layer to realize the final token prediction instead of using an MLP. This structure is simpler and more robust, reducing the number of parameters in the final output and placing more parameters in the model itself;

3. Changing the activation function from ReLU to GeLUs. ReLU is a traditional activation function, whose core calculation logic is to remove propagation less than 0 and retain propagation greater than 0; GeLUs core is to perform a nonlinear mapping on the propagation close to 0, ensuring the nonlinear output after the activation function, which has a certain continuity.

(2) Pre-training Task - GLM

The core innovation of GLM mainly lies in its proposed GLM (General Language Model, General Language Model) task, which is also the origin of the GLM name. GLM is a pre-training method that combines the idea of auto-encoding and auto-regression. The idea of auto-encoding is essentially the MLM task learning approach, which involves randomly deleting continuous tokens in the input text and requiring the model to learn the deleted tokens; the idea of auto-regression is essentially the traditional CLM task learning approach, which requires the model to reconstruct continuous tokens in order.

GLM achieves the combination of MLM and CLM ideas by optimizing an auto-regressive blank filling task. The core idea is that for an input sequence, it is masked similarly to MLM, but the masked tokens are not individual tokens like MLM, but a series of tokens each time; during the learning process, the model needs to predict the masked tokens using the context of the masked part, and within the masked part, it needs to predict the masked tokens in the CLM manner. For example, the input and output could be:

Input: I <MASK> because you <MASK>
Output: <MASK> - love you; <MASK> - are a wonderful person

By combining the ideas of MLM and CLM, it is suitable for generation tasks that generate token by token, and forces the model to learn the implicit relationships in the input text from both directions to adapt to understanding tasks. The GLM model produced by the GLM pre-training task shows certain advantages over BERT series models of the same scale:

Figure 3.14 alt text

However, the greater advantages of the GLM pre-training task are more evident in the pre-training model era. After entering the LLM era, for ultra-large-scale pre-training, CLM shows a significant advantage over MLM. By increasing the model size and expanding the pre-training scale, the generation model pre-trained with CLM can also have a better understanding capability than the MLM-trained understanding model. Therefore, the ChatGLM series models only used the GLM pre-training idea in the first generation model, and from ChatGLM2 onwards, returned to the traditional CLM modeling. Although from the overall development path of LLMs, the GLM pre-training task seems to be a failed attempt, the idea of integrating CLM and MLM through delicate design and producing the first native LLM in Chinese is still of great reference value.

(3) Development of the GLM Family

Based on the early pre-trained model of the GLM model (i.e., the original GLM architecture and pre-training task), referring to the technical ideas of ChatGPT for SFT and RLHF, Zhizhu released the first Chinese open-source LLM ChatGLM-6B in March 2023, becoming the starting point for many Chinese LLM researchers. ChatGLM-6B was pre-trained on 1T of corpus and supported a context length of 2K.

In June 2023, Zhizhu opened source ChatGLM2-6B. Compared to the first generation, ChatGLM2 extended the context length to 32K, achieving a significant breakthrough in model performance through a larger pre-training scale. However, in ChatGLM2, the model architecture basically returned to the LLaMA architecture, introducing the MQA attention mechanism, and the pre-training task also returned to the classic CLM, abandoning the failed attempt of GLM.

ChatGLM3-6B was released in October 2023, achieving the SOTA performance in semantics, mathematics, reasoning, code, and knowledge compared to the second generation. However, the official technical report stated that the model architecture of ChatGLM3 did not change compared to the second generation, and the main optimization sources were more diverse training datasets, more sufficient training steps, and more optimized training strategies. Another important improvement of ChatGLM3 was its support for function calls and code interpreters, allowing developers to directly use the open-source ChatGLM3 to implement Agent development, which has broader application value.

In January 2024, Zhizhu released the GLM-4 series model supporting 128K context, including various types of GLM-4 models, and evaluated its performance on English benchmarks reaching the level of GPT-4. However, Zhizhu did not directly open-source GLM-4, but opened-source its lightweight version GLM-4-9B model, which was pre-trained on a multilingual corpus of 1T tokens, with a context length of 8K, and used the same pipeline and data for post-training as GLM-4. With less training computation, it surpassed Llama-3-8B and supported all the tools of GLM-4.

Figure 3.15 shows the performance evolution of the GLM series models on benchmark sets:

Figure 3.15 alt text

References

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.

[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.

[3] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942.

[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. (2023). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint arXiv:1910.10683.

[5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140), 1–67.

[6] Alec Radford, Karthik Narasimhan. (2018). Improving Language Understanding by Generative Pre-Training. Retrieved from https://api.semanticscholar.org/CorpusID:49313245

[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei. (2020). Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165.

[8] Zhang Fan, Chen An Dong's article "A ten-thousand-word article takes you through the Llama open-source family: from Llama-1 to Llama-3", source: https://mp.weixin.qq.com/s/5_VnzP3JmOB0D5geV5HRFg

[9] Team GLM, A, and the model.13.111.1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111131011111111111111111111111111113131