Aunified framework for the architecture design of LLM-based autonomous agent
HDFS log sentences converted to log templates
The overall design of Data Interpreter. This framework consists of three stages: dynamic plan graph and management, wherein a plan is generated for data science problems, and the state of each task is managed during execution; tool utilisation and evolution, involving the selection or creation of suitable tools to solve problems, continually evolving these tools; and automated confidence-based verification, which examines and votes on logically sound solutions.
The Path Generation process of KNOWAGENT
What is the ReAct model?
The React model, detailed in a paper at ICLR 2023, introduced a novel approach that combined reasoning and acting through language models to enhance task solving in language reasoning and interactive decision-making contexts. The model, referred to as ReAct, interleaves verbal reasoning traces and task-specific actions, enabling dynamic updates to action plans and integration of external information, such as data from APIs like Wikipedia.| Source | % of Dataset | Details |
|---|---|---|
| Pile-CC | 18.11% | Common Crawl: Collection of website crawls, including web pages, metadata, and text extractions. |
| PubMedCentral | 14.40% | PubMed Central: Subset of PubMed repository for biomedical articles, with open full-text access. |
| Books3† | 12.07% | Dataset of books derived from Bibliotik private tracker, mix of fiction and non-fiction. |
| OpenWebText2 | 10.01% | OpenWebText2: Web scraped dataset inspired by WebText and OpenWebTextCorpus, with content from Reddit. |
| ArXiv | 8.96% | Preprint server for research papers predominantly in Math, Computer Science, and Physics. |
| Github | 7.59% | Large corpus of open-source code repositories, enabling code-related task improvements. |
| FreeLaw | 6.12% | Access to and analytical tools for academic studies in the legal realm from the FreeLaw Project. |
| StackExchange | 5.13% | User-contributed content on a network of question-answer websites covering various subjects. |
| USPTOBackgrounds | 3.65% | Background sections from patents granted by the US Patent and Trademark Office. |
| PubMedAbstracts | 3.07% | Abstracts from publications in PubMed, covering a wide range of biomedical topics. |
| Gutenberg (PG-19)† | 2.17% | Classic Western literature from Project Gutenberg books pre-1919. |
| OpenSubtitles | 1.55% | English dataset of subtitles from movies and TV shows, providing natural dialog |
| Datasets | Sourced from | Instances | rounds | prompt | completion |
|---|---|---|---|---|---|
| CoT | NLP datasets + Human-written CoTs | 100,000 | 1.0 | 266.0 | 53.2 |
| FlanV2 | NLP datasets + Human-written Instructions | 100,000 | 1.0 | 355.7 | 31.2 |
| Dolly | Human-written from scratch | 15,011 | 1.0 | 118.1 | 91.3 |
| OpenAssistant | Human-written from scratch | 34,795 | 1.6 | 34.8 | 212.5 |
| Self-instruct | Generated w/ vanilla GPT3 | 82,439 | 1.0 | 41.5 | 29.3 |
| Unnatural Instructions | Generated w/ Davinci-002 | 68,478 | 1.0 | 107.8 | 23.6 |
| Alpaca | Generated w/ Davinci-003 | 52,002 | 1.0 | 27.8 | 64.6 |
| Code-Alpaca | Generated w/ Davinci-003 | 20,022 | 1.0 | 35.6 | 67.8 |
| GPT4-Alpaca | Generated w/ Davinci-003 + GPT4 | 52,002 | 1.0 | 28.0 | 161.8 |
| Baize | Generated w/ ChatGPT | 210,311 | 3.1 | 17.6 | 52.8 |
| ShareGPT3 | User prompts + outputs from various models | 168,864 | 3.2 | 71.0 | 357.8 |
The illustration of four different training strategies in this paper
Visualisations of 6,000 hand written digits from the MNIST dataset
| Model Name | Average | Multi choice | Reasoning | Coding | Future Capabilities | Grade School Math | Math Problems |
|---|---|---|---|---|---|---|---|
| Claude 3.5 Sonnet | 88.38% | 88.70% | 89.00% | 92.00% | 93.10% | 96.40% | 71.10% |
| Claude 3 Opus | 84.83% | 86.80% | 95.40% | 84.90% | 86.80% | 95.00% | 60.10% |
| Gemini 1.5 Pro | 80.08% | 81.90% | 92.50% | 71.90% | 84.00% | 91.70% | 58.50% |
| Gemini Ultra | 79.52% | 83.70% | 87.80% | 74.40% | 83.60% | 94.40% | 53.20% |
| GPT-4 | 79.45% | 86.40% | 95.30% | 67.00% | 83.10% | 92.00% | 52.90% |
| Llama 3 Instruct - 70B | 79.23% | 82.00% | 87.00% | 81.70% | 81.30% | 93.00% | 50.40% |
| Claude 3 Sonnet | 76.55% | 79.00% | 89.00% | 73.00% | 82.90% | 92.30% | 43.10% |
| Claude 3 Haiku | 73.08% | 75.20% | 85.90% | 75.90% | 73.70% | 88.90% | 38.90% |
| Gemini Pro | 68.28% | 71.80% | 84.70% | 67.70% | 75.00% | 77.90% | 32.60% |
| GPT-3.5 | 65.46% | 70.00% | 85.50% | 48.10% | 66.60% | 57.10% | 34.10% |
| Mixtral 8x7B | 59.79% | 70.60% | 84.40% | 40.20% | 60.76% | 74.40% | 28.40% |
| Llama 3 Instruct - 8B | - | 68.40% | - | 62.00% | 61.00% | 79.60% | 30.00% |
Tiktoken: A Fast and Efficient Tokenizer for OpenAI Models
Tiktoken is a fast and efficient open-source tokenizer developed by OpenAI. It is designed to work seamlessly with OpenAI's language models, providing a reliable and consistent way to tokenize text data. **Key Features** 1. Speed: Tiktoken is known for its exceptional speed, outperforming other comparable open-source tokenizers by a factor of 3 to 6 times. 2. Encoding and Decoding: Tiktoken provides simple methods to encode text into token integers and decode token integers back into text. The encode() method converts a text string into a list of token integers, while the decode() method converts a list of token integers back into a string. 3. Token Counting: Tiktoken makes it easy to count the number of tokens in a text string, which is crucial for determining whether a string is too long for a model to process and estimating the cost of an API call. 4. Multilingual Support: Tiktoken can handle text in various languages, making it suitable for multilingual applications. 5. Byte-Level Decoding: Tiktoken offers a decode\_single\_token\_bytes() method that safely converts a single integer token to the bytes it represents, preventing any loss of information that may occur when decoding individual tokens.| Benchmark | Metric | Llama 3 8B | Llama 3.1 8B | Llama 3 70B | Llama 3.1 70B | Llama 3.1 405B |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | macro_avg/acc | 68.5 | 69.4 | 82.0 | 83.6 | 87.3 |
| MMLU (CoT, 0-shot) | macro_avg/acc | 65.3 | 73.0 | 80.9 | 86.0 | 88.6 |
| MMLU-Pro (CoT, 5-shot) | macro_avg/acc | 45.5 | 48.3 | 63.4 | 66.4 | 73.3 |
| ARC-Challenge (0-shot) | acc | 82.4 | 83.4 | 94.4 | 94.8 | 96.9 |
| HumanEval (0-shot) | pass@1 | 60.4 | 72.6 | 81.7 | 80.5 | 89.0 |
| GSM-8K (CoT, 8-shot) | em_maj1@1 | 80.6 | 84.5 | 93.0 | 95.1 | 96.8 |
| MATH (CoT, 0-shot) | final_em | 29.1 | 51.9 | 51.0 | 68.0 | 73.8 |
| API-Bank (0-shot) | acc | 48.3 | 82.6 | 85.1 | 90.0 | 92.0 |
| Gorilla Benchmark API Bench (0-shot) | acc | 1.7 | 8.2 | 14.7 | 29.7 | 35.3 |
| Multilingual MGSM (CoT, 0-shot) | em | - | 68.9 | - | 86.9 | 91.6 |
Illustration of the Mixture-of-Agents Structure. This example showcases 4 MoA layers with 3 agents in each layer. The agents here can share the same model.
Shows the results of the T5-small and T5-3B models on ListOps, comparing performances when fine-tuned without explanations versus with three the types of explanations.
Three ways to increase in-domain compression in a BPE tokenizer with their respective trade-offs
tokenmaster.py codebase
The `tokenmaster.py` codebase is a Python library for the TokenMonster tokenizer.
It provides an interface to load, modify, and use TokenMonster vocabularies for efficient tokenization and detokenization of text.
**Key components and usage**
1. Loading a vocabulary:
* Use `tokenmonster.load(path)` to load a vocabulary from a file, URL, or pre-built vocabulary name.
* For multiprocessing, use `tokenmonster.load_multiprocess_safe(path)` to load the vocabulary safely.
2. Tokenizing text:
* Use `vocab.tokenize(text)` to tokenize a string or a list of strings into token IDs.
* The method returns a numpy array or a list of numpy arrays containing the token IDs.
3. Decoding tokens:
* Use `vocab.decode(tokens)` to decode a single token ID or a list of token IDs back into a string.
* For decoding token streams sequentially, create a decoder object using `decoder = vocab.decoder()` and use `decoder.decode(tokens)` to decode tokens incrementally.
4. Modifying the vocabulary:
* Use `vocab.modify()` to add or delete tokens, resize the vocabulary, enable/disable the UNK token, or reset token IDs.
* Modifications can also be made using individual methods like `add_token()`, `delete_token()`, `add_special_token()`, etc.
* After modifying the vocabulary, save it using `vocab.save(filename)`.
5. Accessing vocabulary information:
* Use `vocab.get_dictionary()` to retrieve a dictionary of all tokens in the vocabulary.
* Access properties like `vocab.vocab_size`, `vocab.unk_token_id()`, `vocab.capcode()`, `vocab.charset()`, etc., to get information about the vocabulary.
**To build your own tokenizer**
1. Prepare a dataset of text that represents the domain you want to tokenize.
2. Use the `tokenmonster` library to train a new vocabulary on your dataset:
* Create a new vocabulary using `vocab = tokenmonster.new(yaml)`, where `yaml` is a YAML string defining the vocabulary configuration.
* Customise the vocabulary configuration, specifying the desired vocabulary size, optimization mode, and other parameters.
* Save the trained vocabulary using `vocab.save(filename)`.
3. Use the trained vocabulary to tokenize and detokenize text in your application:
* Load the saved vocabulary using `vocab = tokenmonster.load(filename)`.
* Tokenize text using `tokens = vocab.tokenize(text)`.
* Decode tokens back into text using `decoded_text = vocab.decode(tokens)`.
The key inputs for building your own tokenizer are:
* A representative dataset of text for training the vocabulary.
* A YAML configuration file specifying the vocabulary parameters (size, optimization mode, etc.).
By following these steps and leveraging the `tokenmonster` library, you can build a custom tokenizer optimized for your specific domain and use case.
Remember to handle any errors and exceptions appropriately, and refer to the documentation and examples provided in the TokenMonster repository for more detailed guidance on using the library.
Definition: LSTM (Long Short-Term Memory) and MLP (Multilayer Perceptron)
LSTM (Long Short-Term Memory) and MLP (Multilayer Perceptron) are two types of neural network architectures that can be used as the mapping function in P-Tuning to transform the continuous prompt embeddings into the hidden states of the model. **LSTM (Long Short-Term Memory)** * LSTM is a type of recurrent neural network (RNN) architecture designed to handle sequential data and capture long-term dependencies. * It consists of a unique cell state and multiple gating mechanisms (input gate, forget gate, and output gate) that regulate the flow of information in and out of the cell. * The cell state acts as a memory unit, allowing the LSTM to selectively remember or forget information over long sequences. * LSTMs are particularly effective in tasks involving sequential data, such as natural language processing, speech recognition, and time series analysis. * In the context of P-Tuning, an LSTM can be used to process the continuous prompt embeddings and generate hidden states that capture the contextual information and long-term dependencies within the prompts. **MLP (Multilayer Perceptron)** * MLP is a feedforward neural network architecture consisting of multiple layers of interconnected nodes (neurons). * It has an input layer, one or more hidden layers, and an output layer. * Each neuron in an MLP applies a nonlinear activation function to a weighted sum of its inputs, allowing the network to learn complex nonlinear mappings between the input and output. * MLPs are versatile and can be used for a wide range of tasks, including classification, regression, and feature learning. * In the context of P-Tuning, an MLP can be used to transform the continuous prompt embeddings into hidden states by applying a series of linear transformations and nonlinear activations. Both LSTM and MLP can be used as the mapping function in P-Tuning, depending on the specific requirements of the task and the nature of the prompt embeddings. LSTMs are particularly suitable when the prompts have a sequential structure and capturing long-term dependencies is important. MLPs, on the other hand, are simpler and more straightforward, making them a good choice when the prompts do not have a strong sequential nature or when computational efficiency is a priority. The choice between LSTM and MLP as the mapping function in P-Tuning ultimately depends on the characteristics of the task, the complexity of the prompts, and the available computational resources. Experimenting with both architectures and comparing their performance can help determine the most suitable choice for a given application.An example of prompt search for “The capital of Britain is [MASK]”. Given the context (blue zone, “Britain”) and target (red zone, “[MASK]”), the orange zone refer to the prompt. In (a), the prompt generator only receives discrete rewards; on the contrary, in (b) the continuous prompt embeddings and prompt encoder can be optimized in a differentiable way.
What is ensembling?
Prompt ensembling is a technique that involves training multiple sets of soft prompts for the same task using a single frozen pre-trained language model. Each set of prompts can be viewed as a separate "model" that adapts the language model to the specific task. By combining the predictions from these multiple prompt-based models, we can create an ensemble that often outperforms individual prompt-based models and matches the performance of the best single prompt. **The main advantages of prompt ensembling are:** **Improved performance:** Ensembling multiple prompts leads to better task performance compared to the average single prompt and often matches or exceeds the best individual prompt. **Parameter efficiency:** Prompt ensembling allows for the creation of multiple task-specific models while sharing the same core language model parameters. This drastically reduces storage costs compared to traditional model ensembling, where each model in the ensemble is a separate copy of the entire model. **Inference efficiency:** During inference, instead of running multiple forward passes for each model in the ensemble, we can process the input with a single forward pass using a batch size equal to the number of prompts in the ensemble. This makes inference more efficient compared to traditional model ensembling. **Here's an example of how prompt ensembling works** Let's say we have a sentiment analysis task where we need to classify movie reviews as positive or negative. We start by training five different sets of soft prompts (P1, P2, P3, P4, P5) on the same training data using a single frozen pre-trained language model (e.g., T5-XXL). During inference, given a new movie review, we prepend each set of prompts to the input and run a single forward pass with a batch size of five. This gives us five different sentiment predictions, one for each prompt: * P1: Positive * P2: Positive * P3: Negative * P4: Positive * P5: Positive To get the final ensemble prediction, we can use a simple majority voting scheme. In this case, four out of five prompts predict "Positive," so the final ensemble prediction is "Positive." Another example is in question answering tasks, such as SQuAD. We can train multiple sets of prompts on the SQuAD dataset and use them to generate multiple answers for a given question. The ensemble prediction can be obtained by combining the answers generated by each prompt, either by voting or by taking the answer with the highest average confidence score. Prompt ensembling is a technique that leverages the parameter efficiency of prompt tuning to create multiple task-specific models while ***sharing the same core language model***. This allows for improved performance, reduced storage costs, and efficient inference compared to traditional model ensembling.A transformer block modified for prefix tuning
| Feature | Parameter-efficient Fine-tuning | Standard Fine-tuning |
|---|---|---|
| Goal | Improve performance on a specific task with limited data and computation | Improve performance on a specific task with ample data and computation |
| Training Data | Small dataset (fewer examples) | Large dataset (many examples) |
| Training Time | Faster, as only a subset of parameters is updated | Longer, due to updating the entire model |
| Computational Resources | Fewer required | Larger required |
| Model Parameters | Modifies only a small subset | Re-trains the entire model |
| Overfitting | Less prone | More prone |
| Training Performance | Good enough, though not as high as full fine-tuning | Typically better than PEFT |
| Use Cases | Ideal for low-resource settings | Ideal for high-resource settings |
Full Transcript
Low Rank Adaptation, or LoRa, allows you to efficiently customise pre trained neural networks, such as diffusion models or language models. It speeds up training and drastically reduces the size of model checkpoints by training very few parameters compared to the base model, while preserving the performance of full fine tuning. And it has become one of the go to methods for customizing AI models. My name is Edward Hu, and I led the invention of LoRa when I was a researcher at Microsoft. In this video, I'll share the research story behind LoRa, how I understand it, and its technical benefits. My team at Microsoft was tasked with answering, can this GPU 3 stuff actually make money? Somewhat surprising finding was that few shot prompting was not enough to get even the largest models performed well enough for production, especially for tasks like natural language to code because it rarely appears in the training data. Fine tuning through gradient updates was a necessity. However, full fine tuning is prohibitively expensive. A single model checkpoint for the 175,000,000,000 parameter variant is 1 terabyte large, which is hard to store and takes minutes to load when deployed. And that's not gonna work when we need to switch among tasks and users rapidly. We tried many off the shelf, parameter efficient fine tuning methods, but all of them had a compromise for our case. It was with product impact in mind that we invented LoRa. So what is LoRa? I like to see it as a generalization of full fine tuning by asking 2 questions. Question 1: Do we need to fine tune all the parameters? Question 2: For the weight matrices we fine tune, how expressive should the updates be in terms of matrix rank? We can turn these two questions into the 2 axes of a 2 d plane. Full fine tuning is all the way in the upper right corner and the axes, including the origin, correspond to the original model. Any point in this box is a valid LoRa configuration. Let's quickly talk about how we control the expressivity of a matrix update by controlling its rank. A d by d matrix can represent any linear transformation in a d dimensional vector space. However, if we start with a vector in Rd, first transform it to rr, where r is less than d, and finally transform it back to rd, we restrict the kind of linear transformations we can represent. How does stopping by rr achieve that? Imagine an extreme case where r, or rank, equals 1. Whatever the input does boils down to just one number, which can only scale the output. By picking a small r, the kind of linear transformation we can represent is greatly limited, even though the output is still in rd. Now, we only have to store 2 times d times r parameters instead of d squared. This is how LoRa stores matrix updates. Now back to the 2 d plane we talked about. The surprising result of the LoRa paper is that a point near the origin performed just as well as full fine tuning all the way in the corner. Once we see LoRa as a generalization of full fine tuning, we can easily answer some commonly asked questions for using LoRa, such as how to choose the rank R, or when to use full fine tuning. Since full fine tuning is a special case of LoRa, and we know that full fine tuning works, we can start with a point near the origin and work our way back to the corner. At some point, this has to work, and more likely than not, what ended up working will be near the origin. Otherwise, just give up and do full fine tuning. How might that happen? Let's consider a thought experiment. If we take a language model pre trained on English and English only, but we wanna adapt it for some tasks, say in Martian. And let's say that English and Martian have little in common, since we're basically training a model all over again. Parameter efficient fine tuning methods shouldn't work too well, So we might as well do full fine tuning instead. Another question is, can I use LoRa for a certain model of architecture? Say a wave net or a support vector machine. And by the way, nobody really asks about the latter, but as long as the model uses matrix multiplication, we can ask, do we need to fine tune all the parameters and how expressive should the updates be? As long as we can ask these 2 questions, we can use LoRa, which makes it very generally applicable. Indeed, while we invented LoRa for large language models, people later found it to be very effective for diffusion models as well. I wanna point out that one advantage of LoRa is that it's clear what to do next if underperformance. We adapt more parameters and increase the rank. For approaches like prefix tuning, midfit, or adapter, it is not clear what we can do next because there isn't a knob to turn that allow these methods to recover full fine tuning, unlike LoRa. Now, let's dive into the specifics of the benefits of LoRa. The most visible one is a reduction of checkpoint sizes. On GP3, we reduce the checkpoint size from 1 terabyte down to 25 megabytes. This is a direct result of training much fewer parameters 4,700,000 in this case compared to 175,000,000,000. Another important benefit is that ***LoRa doesn't introduce any inference latency.*** You might say, hold on, don't we have these additional low rank matrices on the side while that's truly trained? What happens during inference stats? Since LoRa updates are additive to the original parameters, we can expand the low rank matrices by multiplying out the low rank bottleneck and add the updates to the original parameters. Now, we can perform inference literally the same way as with a base model, and there is no additional latency by definition. When we need to switch tasks, we simply repeat the process but this time subtract the updates. By being careful about numerical precisions, we can recover the original parameters and we repeat to load another LoRa module. This process can be done in parallel for all parameters and is faster than a single forward pass. This is how we can switch models quickly without introducing any additional inference latency. Finally, I want to mention a few engineering ideas enabled by LoRa. The first one is to cache many LoRa modules in RAM during deployment. So model switching simply involves data transfer between RAM and VRAM. Since RAM is usually much larger than VRAM, we can cache thousands of LoRa modules and never worry about reading from the disk again. Another idea is to train multiple LoRa modules in parallel, each on its own task. This is achieved by sharing the same base model and routing different inputs in a single batch through different LoRa modules. This way, we can batch different LoRa jobs together and fully utilize the GPUs. There are several community implementations of this, which are linked in the video description. The final idea uses the fact that LoRa models are additive. Imagine a pipeline where a pre trained model is gradually specialized, or maybe it's first fine tuned for a particular language, and then a particular domain, and finally a particular task or even a specific user. The adaptive models form a tree, Each non root node can be a Lua module on top of the sum of its ancestors. The model rank can be larger near the root and smaller near the leaves to accommodate different dataset sizes. Model switching here becomes tree traversal and we never have to load the base model more than once. Let me know in the comments if you have any cool ideas on how LoRa can be used or extended or if you just have any questions. I'll see you in the next video.What are low rank matrices?
A **matrix** is a rectangular array of numbers arranged in **rows and columns**. The **rank of a matrix** is the ***maximum number of linearly independent rows or columns*** in the matrix. In other words, it's the dimension of the vector space spanned by the matrix's rows or columns. Consider a **matrix** A: ```yaml A = [1 2 3] [2 4 6] [3 6 9] ``` In this matrix, we can see that the second row is 2 times the first row, and the third row is 3 times the first row. This means that the rows are **linearly dependent**. We can express any row as a linear combination of the other rows. Similarly, the second column is 2 times the first column, and the third column is 3 times the first column. The columns are also **linearly dependent**. In this case, the rank of the matrix is 1. Despite the matrix being 3x3, it only contains one independent piece of information. Now, let's look at the ***concept of lower-rank matrices**.* A matrix is considered to be of lower rank if its **rank is less than the minimum of its number of rows and columns**. In the example above, the matrix has a rank of 1, which is lower than min(3, 3) = 3, so it is a lower-rank matrix. The idea of lower-rank matrices is used in many applications, such as: **Data Compression**: By approximating a matrix with a lower-rank matrix, we can store less data while preserving the most important information. **Recommendation Systems:** User-item matrices in recommendation systems are often of lower rank because user preferences can be described by a smaller number of latent factors. **Image Processing:** Many operations in image processing, such as image denoising and compression, exploit the fact that image matrices are often of lower rank. The **Rank-Nullity Theorem** states that for a linear map (which can be represented by a matrix) between two vector spaces, the dimension of the domain (number of columns) equals the sum of the rank (dimension of the image) and the nullity (dimension of the kernel). This theorem connects the concepts of rank and nullity, showing that they are complementary. In summary, lower-rank matrices are matrices whose rank is less than the maximum possible given their dimensions. They are used in many applications to simplify data, reduce dimensionality, and uncover hidden structures. The rank of a matrix can be determined by the number of linearly independent rows or columns, which are always equal, as stated by the Rank-Nullity Theorem.Gaussian initialisation
In the LoRA method, matrix A is initialised with a random Gaussian distribution, while matrix B is initialised with zeros. The reason for using Gaussian initialisation for matrix A is to ***introduce randomness and break symmetry*** in the initial values of the weights. When the weights of a neural network are initialised to the same value (e.g., all zeros), the network ***may struggle to learn meaningful patterns because all the neurons behave identically***. By initialising the weights with random values drawn from a Gaussian distribution, we ensure that the neurons start with different initial activations, allowing them to learn diverse features during training. The choice of Gaussian initialisation is based on the principle of **"symmetry breaking"** and the idea that the weights should be initialised with **small random values** to facilitate gradient flow and prevent vanishing or exploding gradients. Gaussian initialisation has been shown to work well in practice and is commonly used in deep learning. In the context of LoRA, initialising matrix $$A$$ with a Gaussian distribution ensures that the initial weight update matrix $$∆W$$ (which is the product of $$A$$ and $$B$$) has random, small values. This allows the model to ***gradually adapt the pre-trained weights to the downstream task during fine-tuning***, starting from a point of random perturbation. By initialising matrix $$B$$ with zeros, the initial weight update matrix $$∆W$$ is effectively zero, meaning that the model starts with the original pre-trained weights. As training progresses, the values of $$A$$ and $$B$$ are updated based on the gradients, allowing the model to learn the necessary adaptations for the specific task. The combination of Gaussian initialisation for matrix of $$A$$and zero initialisation for matrix of $$B$$ in LoRA ensures a balanced starting point for fine-tuning, facilitating the learning of task-specific adaptations while leveraging the knowledge captured in the pre-trained weights.Forward Pass and Backpropagation explanation
**Forward Pass** During the forward pass with LoRA, the input is multiplied by both the pre-trained weight matrix $$Wpretrained$$ and the weight update matrix $$∆W$$. The updated output is computed as follows: $$output = input × Wpretrained + input × ∆W ∆W = B × A$$ The pre-trained weight matrix $$Wpretrained$$ remains frozen, while the matrices $$A$$ and $$B$$ are learned during fine-tuning. The output of the self-attention layer is obtained by summing the results of the matrix multiplications. **Backpropagation** During backpropagation, the ***gradients are computed with respect to the input and the trainable parameters***. In LoRA, only the matrices matrices $$A$$ and $$B$$ are updated based on the gradients, while the pre-trained weight matrix $$Wpretrained$$ remains unchanged. The gradients of the loss with respect to $$A$$ and $$B$$ are computed using the chain rule: $$∂loss / ∂A = (∂loss / ∂∆W) × B^T ∂loss / ∂B = (∂loss / ∂∆W)^T × A$$ The optimiser then uses these gradients to update the values of $$A$$ and $$B$$A conceptual diagram of LoRA with an r value equal to 1 and 2. In both examples the decomposed A and B matrices result in the same sized change matrix, but r=2 is able to encode more linearly independent information into the change matrix, due to having more information in the A and B matrices. Source: Daniel Warfield
Backpropagation is a fundamental algorithm used in training neural networks and other differentiable machine learning models. It is a method for efficiently calculating the gradients of the model's parameters with respect to the loss function. The goal of backpropagation is to update the model's parameters in a way that minimizes the difference between the predicted output and the desired output.
Benchmarks?
The performance of models against these benchmarks is measured using various methods and metrics, depending on the specific focus of each benchmark. Here is an overview of how performance is typically measured for each: **MMLU (Massive Multitask Language Understanding)** * **Measurement Method:** Multiple-choice questions. * **Metrics:** Accuracy is the primary metric, calculated as the percentage of correct answers out of the total questions. * **Details:** Performance is evaluated across 57 tasks from different domains, and the overall accuracy provides a comprehensive measure of the model's general knowledge and understanding. **Vicuna Benchmark** * **Measurement Method:** Evaluation of conversational tasks and scenarios. * **Metrics:** Human evaluation scores, BLEU (Bilingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and other dialogue-specific metrics. * **Details:** Human judges often rate the quality of responses based on coherence, relevance, informativeness, and fluency. Automated metrics may also be used to compare generated text against reference responses. **OA (OpenAI) Benchmark** * **Measurement Method:** A variety of tasks designed to test different capabilities. * **Metrics:** Task-specific metrics such as accuracy, F1 score, precision, recall, and others depending on the nature of each task. * **Details:** The benchmark includes diverse tasks, and the performance is measured using the appropriate metric for each task to provide a detailed view of the model's strengths and weaknesses. **BigBench (Beyond the Imitation Game Benchmark):** * **Measurement Method:** A wide range of tasks developed by the research community. * **Metrics:** Varies by task; common metrics include accuracy, F1 score, and others relevant to the specific task. * **Details:** The benchmark covers reasoning, commonsense understanding, and other advanced skills. Performance is evaluated task by task, and an aggregate score may be used to summarize overall performance. **RAFT (Realistic Adversarial Functionality Test)** * **Measurement Method:** Adversarial tasks designed to expose model weaknesses. * **Metrics:** Accuracy, robustness metrics, error rates, and other task-specific metrics. * **Details:** Models are tested with challenging and tricky inputs to assess their robustness and reliability. Performance is measured by how well the model can handle these difficult scenarios. **HELM (Holistic Evaluation of Language Models)** * **Measurement Method:** A comprehensive set of evaluations across various dimensions. * **Metrics:** Accuracy, fairness metrics, robustness metrics, efficiency (e.g., speed, computational resources), and others. * **Details:** The benchmark aims to provide a holistic view of performance, considering multiple aspects beyond just accuracy. Metrics are chosen to reflect the model's performance in terms of fairness, robustness, and efficiency. In summary, each benchmark employs specific methods and metrics tailored to its focus area, providing a nuanced and detailed assessment of language model performance across different tasks and dimensions.FP8 is a natural progression for accelerating deep learning (DL) training beyond the 16-bit formats common in modern processors. DL applications require two 8-bit floating point (FP8) binary interchange formats, both supported by Hopper and Ada GPU architectures: E4M3 and E5M2. These types enable doubling the math throughput as well as reducing bandwidth pressure in half; however, their use requires some care due to their narrower range and lower precision compared to the 16-bit formats. We'll cover three aspects of FP8 for deep learning: