Systematic Evaluation of Instruction-Tuned Large Language Models on Open Datasets

This October 2023 paper explores the impact of instruction-tuning on large language models using various open datasets.

The authors aim to systematically evaluate the performance of these models across a range of tasks and provide a comprehensive comparison with state-of-the-art proprietary models like ChatGPT and GPT-4.

The researchers trained a series of instruction-tuned models, ranging from 6.7B to 65B parameters, on 12 different instruction datasets.

We would argue that this paper does not take into account significant progress made on highly curated datasets, as referred to in the Alpagasus paper and the LIMA paper.

How Far Can Camels Go? Exploring the State of Instruction Tuning...arXiv.org

Key findings

1. Different instruction-tuning datasets can enhance specific skills, but no single dataset or combination provides the best performance across all evaluations.

2. Larger or pretrained-for-longer base models consistently outperform smaller ones after instruction tuning.

3. Even the largest model (65B) finetuned on a mix of instruction datasets fails to outperform ChatGPT, although it significantly outperforms similar smaller models.

4. Model and human preference-based evaluations do not always reflect differences in model capabilities exposed by benchmark-based evaluations, highlighting the need for comprehensive evaluation.

Instruction tuning datasets

Below are some of the datasets tested.

To describe the table below:

• Dataset: Dataset source

• Sourced from: Where the data came from (human only or combination of model plus human)

• Instances: The number of separate instructions in the dataset

• N rounds: The average number of conversation turns in each dataset. A conversation turn consists of a user prompt and the corresponding assistant response.

• L prompt: The average length (number of tokens) of the prompts in each dataset.

• L completion: The average length (number of tokens) of the completions or responses generated by the assistant in each dataset.

These definitions provide a clear understanding of the terms used in the table and their relevance to the instruction datasets investigated in the research paper.

Datasets	Sourced from	Instances	rounds	prompt	completion
CoT	NLP datasets + Human-written CoTs	100,000	1.0	266.0	53.2
FlanV2	NLP datasets + Human-written Instructions	100,000	1.0	355.7	31.2
Dolly	Human-written from scratch	15,011	1.0	118.1	91.3
OpenAssistant	Human-written from scratch	34,795	1.6	34.8	212.5
Self-instruct	Generated w/ vanilla GPT3	82,439	1.0	41.5	29.3
Unnatural Instructions	Generated w/ Davinci-002	68,478	1.0	107.8	23.6
Alpaca	Generated w/ Davinci-003	52,002	1.0	27.8	64.6
Code-Alpaca	Generated w/ Davinci-003	20,022	1.0	35.6	67.8
GPT4-Alpaca	Generated w/ Davinci-003 + GPT4	52,002	1.0	28.0	161.8
Baize	Generated w/ ChatGPT	210,311	3.1	17.6	52.8
ShareGPT3	User prompts + outputs from various models	168,864	3.2	71.0	357.8

Instruction Tuning Format

Instruction tuning is a technique used to finetune pretrained language models to better understand and respond to human requests expressed in natural language.

An instruction tuning dataset consists of a collection of input-output pairs, where the input is a user prompt or instruction, and the output is the desired response or completion. The goal is to train the model to generate appropriate responses given the input prompts.

The format typically follows a chat-style schema, where the interactions between the user and the language model (assistant) are encoded together using special tokens.

The format consists of:

1. <|user|> token placed before user utterances

2. <|assistant|> token placed before target assistant responses

3. </s> end-of-text marker placed at the end of each assistant output

To prepare the data for training, special tokens are used to demarcate the boundaries between the user prompts and the assistant responses.

In this case, <|user|> is used to indicate the start of a user prompt, and <|assistant|> is used to indicate the start of the assistant's response.

The entire sequence, including the user prompts and assistant responses, is encoded together.

During training, the loss is computed only on the tokens after <|assistant|> and before the next <|user|> token.

This is achieved using teacher-forcing with loss masking, where the tokens belonging to the input sequence(s) are masked.

Here's an example of the format in a code block:

<|user|> Explain the fault-tolerance of the reaction control system on the Space Shuttle.
<|assistant|> The reaction control system (RCS) on the Space Shuttle was designed to be fault-tolerant, meaning it was able to continue functioning even if one or more of its components failed. The RCS consisted of two sets of ... </s>

<|user|> Did the RCS have any on-orbit failures?
<|assistant|> There were several instances where the reaction control system (RCS) on the Space Shuttle experienced failures or malfunctions during on-orbit missions. These ... </s>

In this example, the user prompts are marked with <|user|> tokens, and the assistant responses are marked with <|assistant|> tokens. The </s> token is used to indicate the end of the assistant's output for each round.

During training, we compute loss only on tokens after <|assistant|> and before the next <|user|> token.

More formally, we consider an instruction dataset as consisting of $N$ tuples, each with $i$ turns of conversation:

\{(x^1_j, y^1_j, x^2_j, y^2_j, \dots, x^i_j, y^i_j)\}^N_{j=1}$

where $x_i$ is a user prompt and $y_i$ the desired output.

For most instances, $i = 1$ and we train the model to output $y_j$ given $x_j$ - meaning the model is trained to generate the output $y$ given the input $x$.

However, in the case of conversation datasets, there can be multiple turns to train the model to predict $y^i_j$ given some conversation history $x^1_j, y^1_j, x^2_j, \dots, x^i_j$.

Given $X$ as the tokens belonging to the input, and $Y$ as the target tokens, the loss function is:

$$L = -\sum_j \log p_\theta(t_j \mid t_{<j}) \times \begin{cases} 1 & \text{if } t_j \in Y \\ 0 & \text{otherwise} \end{cases}$$

where $tⱼ$ is the $jth$ input token (belonging to input $X$ or target $Y$.

The loss function used in this training process is the negative log-likelihood loss. This ensures that the loss is computed only on the tokens that are part of the assistant's response.

This is achieved by using teacher forcing with loss masking.

Teacher forcing means that the model is provided with the ground truth output at each step during training, and it learns to predict the next token based on the previous tokens. Loss masking ensures that the loss is computed only on the relevant tokens (i.e., the assistant's response) and not on the input tokens.

The training process

Now, let's consider how this training process interacts with the Transformer architecture.

The Transformer architecture consists of an encoder and a decoder, but in this case, a decoder-only model is used.

During training, the input sequence (user prompts and conversation history) is passed through the Transformer layers.

The self-attention mechanism in the Transformer allows the model to attend to different parts of the input sequence and capture the relevant information for generating the output.

At each decoding step, the model takes the previously generated tokens as input and attends to the entire input sequence to predict the next token. The loss is computed based on the predicted probability distribution over the vocabulary and the ground truth token at each step.

The Transformer's multi-head attention mechanism enables the model to learn different aspects of the input-output relationship, allowing it to capture the nuances of the task. The feedforward layers in the Transformer help in processing and transforming the learned representations.

Through the process of fine-tuning, the pretrained LLM adapts its parameters to the specific task of instruction following. The model learns to understand the patterns and relationships between the user prompts and the desired responses, enabling it to generate appropriate outputs for new, unseen prompts.

By training on a diverse set of instruction-following examples, the model becomes more versatile and can handle a wide range of tasks and conversations. The quality and diversity of the training dataset play a crucial role in the model's performance and generalization ability.

Decoder Only?

In the context of language modelling and sequence-to-sequence tasks, a decoder-only model refers to a specific architecture where only the decoder component of the Transformer is used, without an explicit encoder.

In a typical sequence-to-sequence model, the Transformer architecture consists of two main components: the encoder and the decoder. The encoder takes the input sequence and generates a set of hidden representations that capture the relevant information from the input. The decoder then takes these hidden representations and generates the output sequence step by step.

However, in a decoder-only model, the entire input sequence (both the user prompts and the conversation history) is concatenated and fed directly into the decoder. The decoder attends to the input sequence and generates the output sequence in an autoregressive manner, meaning it predicts the next token based on the previously generated tokens.

The main difference between a decoder-only model and a standard encoder-decoder model is that the decoder-only model does not have a separate encoder to process the input sequence. Instead, the decoder itself attends to the input sequence and learns to generate the output based on the entire context.

Which is the best dataset?

The analysis of the instruction tuning datasets and base models reveals several key findings:

No single best dataset

There is no single instruction tuning dataset that performs best across all tasks.

Different datasets enable different capabilities in the model, with notable examples being CoT for mathematical reasoning in GSM and Code-Alpaca for Codex-Eval.

Combining datasets is beneficial

Models trained on combined datasets generally achieve the best overall performance on benchmark tasks. While they may not be the best for individual tasks, they have the highest average performance across all tasks.

Base model quality is crucial

The choice of the base model significantly impacts downstream performance.

LLAMA models outperform OPT and Pythia models of comparable size when trained on the same data mixture, likely due to LLAMA being pretrained on more tokens.

The addition of LLAMA-2 further confirms this finding, showing that improvements can come from upgrading the base model alone. This is no surprise.

Conclusion

In conclusion, this research provides a comprehensive evaluation of various publicly available resources for instruction tuning and compares their performance to state-of-the-art proprietary models like ChatGPT and GPT-4.

The findings emphasise the importance of using strong base models, combining diverse datasets, and conducting thorough evaluations across a wide range of tasks and metrics.

The study highlights that no single instruction tuning dataset excels across all tasks, but rather different datasets enable different capabilities in the model.

Combining datasets generally results in the best overall performance on benchmark tasks, although it may lead to slight performance drops compared to the best performance on specific tasks.

The quality of the base model plays a crucial role in downstream performance, with LLAMA models outperforming other models of comparable size when trained on the same data mixture.

However, despite the progress made, the strongest open models still fall short of matching the performance of proprietary models like ChatGPT and GPT-4. This gap underscores the need for continued development of robust base models and more diverse, comprehensive datasets.