# Systematic Evaluation of Instruction-Tuned Large Language Models on Open Datasets

This <mark style="color:blue;">**October 2023**</mark> paper explores the impact of instruction-tuning on large language models using various open datasets.&#x20;

The authors aim to systematically evaluate the performance of these models across a range of tasks and provide a comprehensive comparison with state-of-the-art proprietary models like ChatGPT and GPT-4.

The researchers trained a series of instruction-tuned models, ranging from 6.7B to 65B parameters, on 12 different instruction datasets.&#x20;

We would argue that this paper does not take into account significant progress made on highly curated datasets, as referred to in the [<mark style="color:blue;">**Alpagasus paper**</mark>](https://training.continuumlabs.ai/data/datasets/instruction-fine-tuning-alpagasus) and the [<mark style="color:blue;">**LIMA paper**</mark>](https://training.continuumlabs.ai/data/datasets/less-is-more-for-alignment).

{% embed url="<https://arxiv.org/abs/2306.04751>" %}

### <mark style="color:purple;">Key findings</mark>

1. Different instruction-tuning datasets can enhance specific skills, but <mark style="color:yellow;">no single dataset or combination provides the best performance across all evaluations</mark>.
2. Larger or pretrained-for-longer base models consistently outperform smaller ones after instruction tuning.
3. Even the largest model (65B) finetuned on a mix of instruction datasets fails to outperform ChatGPT, although it significantly outperforms similar smaller models.
4. Model and human preference-based evaluations do not always reflect differences in model capabilities exposed by benchmark-based evaluations, highlighting the need for comprehensive evaluation.

### <mark style="color:purple;">Instruction tuning datasets</mark>

Below are some of the datasets tested.

To describe the table below:

* <mark style="color:blue;">**Dataset:**</mark> Dataset source
* <mark style="color:blue;">**Sourced from:**</mark> Where the data came from (human only or combination of model plus human)
* <mark style="color:blue;">**Instances:**</mark> The number of separate instructions in the dataset
* <mark style="color:blue;">**N rounds:**</mark> The average number of conversation turns in each dataset. A conversation turn consists of a user prompt and the corresponding assistant response.
* <mark style="color:blue;">**L prompt:**</mark> The average length (number of tokens) of the prompts in each dataset.
* <mark style="color:blue;">**L completion:**</mark> The average length (number of tokens) of the completions or responses generated by the assistant in each dataset.

These definitions provide a clear understanding of the terms used in the table and their relevance to the instruction datasets investigated in the research paper.

<table><thead><tr><th width="159">Datasets</th><th width="171">Sourced from</th><th width="108">Instances</th><th width="85">rounds</th><th width="93">prompt</th><th>completion</th></tr></thead><tbody><tr><td>CoT</td><td>NLP datasets + Human-written CoTs</td><td>100,000</td><td>1.0</td><td>266.0</td><td>53.2</td></tr><tr><td>FlanV2</td><td>NLP datasets + Human-written Instructions</td><td>100,000</td><td>1.0</td><td>355.7</td><td>31.2</td></tr><tr><td>Dolly</td><td>Human-written from scratch</td><td>15,011</td><td>1.0</td><td>118.1</td><td>91.3</td></tr><tr><td>OpenAssistant</td><td>Human-written from scratch</td><td>34,795</td><td>1.6</td><td>34.8</td><td>212.5</td></tr><tr><td>Self-instruct</td><td>Generated w/ vanilla GPT3</td><td>82,439</td><td>1.0</td><td>41.5</td><td>29.3</td></tr><tr><td>Unnatural Instructions</td><td>Generated w/ Davinci-002</td><td>68,478</td><td>1.0</td><td>107.8</td><td>23.6</td></tr><tr><td>Alpaca</td><td>Generated w/ Davinci-003</td><td>52,002</td><td>1.0</td><td>27.8</td><td>64.6</td></tr><tr><td>Code-Alpaca</td><td>Generated w/ Davinci-003</td><td>20,022</td><td>1.0</td><td>35.6</td><td>67.8</td></tr><tr><td>GPT4-Alpaca</td><td>Generated w/ Davinci-003 + GPT4</td><td>52,002</td><td>1.0</td><td>28.0</td><td>161.8</td></tr><tr><td>Baize</td><td>Generated w/ ChatGPT</td><td>210,311</td><td>3.1</td><td>17.6</td><td>52.8</td></tr><tr><td>ShareGPT3</td><td>User prompts + outputs from various models</td><td>168,864</td><td>3.2</td><td>71.0</td><td>357.8</td></tr></tbody></table>

### <mark style="color:purple;">Instruction Tuning Format</mark>

Instruction tuning is a technique used to finetune pretrained language models to better understand and respond to human requests expressed in natural language.&#x20;

An instruction tuning dataset consists of a collection of input-output pairs, where the input is a user prompt or instruction, and the output is the desired response or completion. The goal is to train the model to generate appropriate responses given the input prompts.

The format typically follows a chat-style schema, where the interactions between the user and the language model (assistant) are *<mark style="color:yellow;">**encoded together**</mark>* using special tokens.

The format consists of:

1. <mark style="color:purple;"><|user|></mark> token placed before user utterances
2. <mark style="color:purple;"><|assistant|></mark> token placed before target assistant responses
3. <mark style="color:purple;">\</s></mark> end-of-text marker placed at the end of each assistant output

To prepare the data for training, *<mark style="color:yellow;">**special tokens**</mark>* are used to demarcate the boundaries between the user prompts and the assistant responses.

In this case, <mark style="color:purple;">**<|user|>**</mark> is used to indicate the start of a user prompt, and <mark style="color:purple;">**<|assistant|>**</mark> is used to indicate the start of the assistant's response. &#x20;

The entire sequence, including the user prompts and assistant responses, is encoded together.

During training, the <mark style="color:yellow;">**loss is computed only on the tokens after**</mark>**&#x20;**<mark style="color:purple;">**<|assistant|>**</mark>**&#x20;**<mark style="color:yellow;">**and before the next**</mark>**&#x20;**<mark style="color:purple;">**<|user|>**</mark>**&#x20;**<mark style="color:yellow;">**token.**</mark>&#x20;

This is achieved using *<mark style="color:yellow;">**teacher-forcing with loss masking,**</mark>* where the tokens belonging to the input sequence(s) are masked.

Here's an example of the format in a code block:

{% code overflow="wrap" %}

```bash
<|user|> Explain the fault-tolerance of the reaction control system on the Space Shuttle.
<|assistant|> The reaction control system (RCS) on the Space Shuttle was designed to be fault-tolerant, meaning it was able to continue functioning even if one or more of its components failed. The RCS consisted of two sets of ... </s>

<|user|> Did the RCS have any on-orbit failures?
<|assistant|> There were several instances where the reaction control system (RCS) on the Space Shuttle experienced failures or malfunctions during on-orbit missions. These ... </s>
```

{% endcode %}

In this example, the user prompts are marked with <mark style="color:purple;">**<|user|>**</mark> tokens, and the assistant responses are marked with <mark style="color:purple;">**<|assistant|>**</mark> tokens. The <mark style="color:purple;">**\</s>**</mark> token is used to indicate the end of the assistant's output for each round.

During training, we compute loss only on tokens after <mark style="color:purple;">**`<|assistant|>`**</mark> and before the next <mark style="color:purple;">**`<|user|>`**</mark> token.&#x20;

More formally, we consider an instruction dataset as consisting of $$N$$ tuples, each with $$i$$ *<mark style="color:yellow;">**turns of conversation:**</mark>*

$$
{(x^1\_j, y^1\_j, x^2\_j, y^2\_j, \dots, x^i\_j, y^i\_j)}^N\_{j=1}$
$$

&#x20;where $$x\_i$$ is a *<mark style="color:yellow;">**user prompt**</mark>* and $$y\_i$$ the *<mark style="color:yellow;">**desired output.**</mark>*&#x20;

For most instances, $$i = 1$$ and we train the model to output $$y\_j$$ given $$x\_j$$ -  meaning the model is trained to generate the output $$y$$ given the input $$x$$.

However, in the case of conversation datasets, there can be multiple turns to train the model to predict $$y^i\_j$$ given some conversation history $$x^1\_j, y^1\_j, x^2\_j, \dots, x^i\_j$$.&#x20;

Given $$X$$ as the tokens belonging to the input, and $$Y$$ as the target tokens, the *<mark style="color:yellow;">**loss function**</mark>* is:

$$
L = -\sum\_j \log p\_\theta(t\_j \mid t\_{\<j}) \times \begin{cases} 1 & \text{if } t\_j \in Y \ 0 & \text{otherwise} \end{cases}
$$

where $$tⱼ$$ is the $$jth$$ input token (belonging to input $$X$$ or target $$Y$$.

The loss function used in this training process is the <mark style="color:yellow;">**negative log-likelihood loss**</mark>. This ensures that the *<mark style="color:yellow;">**loss is computed only on the tokens that are part of the assistant's response**</mark>*.

This is achieved by using <mark style="color:blue;">**teacher forcing with loss masking**</mark>.&#x20;

Teacher forcing means that the model is provided with the *<mark style="color:yellow;">**ground truth output at each step during training,**</mark>* and it learns to predict the next token based on the previous tokens.  Loss masking ensures that the loss is computed only on the relevant tokens (i.e., the assistant's response) and not on the input tokens.

### <mark style="color:purple;">The training process</mark>

Now, let's consider how this training process interacts with the Transformer architecture.&#x20;

The Transformer architecture consists of an encoder and a decoder, but in this case, a decoder-only model is used.

During training, the input sequence (user prompts and conversation history) is passed through the Transformer layers.&#x20;

The <mark style="color:blue;">**self-attention mechanism**</mark> in the Transformer allows the model to attend to different parts of the input sequence and capture the relevant information for generating the output.

At each <mark style="color:blue;">**decoding step**</mark>, the model takes the previously generated tokens as input and attends to the entire input sequence to predict the next token.  The loss is computed based on the predicted probability distribution over the vocabulary and the ground truth token at each step.

The Transformer's <mark style="color:blue;">**multi-head attention mechanism**</mark> enables the model to learn different aspects of the input-output relationship, allowing it to capture the nuances of the task. The feedforward layers in the Transformer help in processing and transforming the learned representations.

Through the <mark style="color:blue;">**process of fine-tuning**</mark>, the pretrained LLM *<mark style="color:yellow;">**adapts its parameters**</mark>* to the specific task of instruction following. The model learns to understand the patterns and relationships between the user prompts and the desired responses, enabling it to generate appropriate outputs for new, unseen prompts.

By training on a diverse set of instruction-following examples, the model becomes more versatile and can handle a wide range of tasks and conversations. The quality and diversity of the training dataset play a crucial role in the model's performance and generalization ability.

### <mark style="color:purple;">Decoder Only?</mark>

In the context of language modelling and sequence-to-sequence tasks, a decoder-only model refers to a specific architecture where only the decoder component of the Transformer is used, without an explicit encoder.

In a typical sequence-to-sequence model, the Transformer architecture consists of two main components: the encoder and the decoder. The encoder takes the input sequence and generates a set of hidden representations that capture the relevant information from the input. The decoder then takes these hidden representations and generates the output sequence step by step.

However, in a <mark style="color:blue;">**decoder-only model**</mark>, the entire input sequence (both the user prompts and the conversation history) is concatenated and fed directly into the decoder. The decoder attends to the input sequence and generates the output sequence in an autoregressive manner, meaning it predicts the next token based on the previously generated tokens.

The *<mark style="color:yellow;">**main difference between a decoder-only model and a standard encoder-decoder model**</mark>* is that the decoder-only model does not have a separate encoder to process the input sequence. Instead, the decoder itself attends to the input sequence and learns to generate the output based on the entire context.

### <mark style="color:purple;">Which is the best dataset?</mark>

The analysis of the instruction tuning datasets and base models reveals several key findings:

#### <mark style="color:green;">No single best dataset</mark>

There is no single instruction tuning dataset that performs best across all tasks.&#x20;

Different datasets enable different capabilities in the model, with notable examples being CoT for mathematical reasoning in GSM and Code-Alpaca for Codex-Eval.

#### <mark style="color:green;">Combining datasets is beneficial</mark>

Models trained on combined datasets generally achieve the best overall performance on benchmark tasks. While they may not be the best for individual tasks, they have the highest average performance across all tasks.

#### <mark style="color:green;">Base model quality is crucial</mark>

The choice of the base model significantly impacts downstream performance.&#x20;

LLAMA models outperform OPT and Pythia models of comparable size when trained on the same data mixture, likely due to LLAMA being pretrained on more tokens. &#x20;

The addition of LLAMA-2 further confirms this finding, showing that improvements can come from upgrading the base model alone.  This is no surprise.

### <mark style="color:purple;">Conclusion</mark>

In conclusion, this research provides a comprehensive evaluation of various publicly available resources for instruction tuning and compares their performance to state-of-the-art proprietary models like ChatGPT and GPT-4.&#x20;

The findings emphasise the importance of using strong base models, combining diverse datasets, and conducting thorough evaluations across a wide range of tasks and metrics.

The study highlights that no single instruction tuning dataset excels across all tasks, but rather different datasets enable different capabilities in the model. &#x20;

Combining datasets generally results in the best overall performance on benchmark tasks, although it may lead to slight performance drops compared to the best performance on specific tasks.

The quality of the base model plays a crucial role in downstream performance, with LLAMA models outperforming other models of comparable size when trained on the same data mixture.

However, despite the progress made, the strongest open models still fall short of matching the performance of proprietary models like ChatGPT and GPT-4. This gap underscores the need for continued development of robust base models and more diverse, comprehensive datasets.