QLORA: Efficient Finetuning of Quantized Language Models
Last updated
Copyright Continuum Labs - 2023
Last updated
QLORA: Efficient Finetuning of Quantized Language Models
The process of fine-tuning large parameter language models to adapt to specific tasks remains a challenge due to the extensive memory requirements.
Quantized Low Rank Adapters (QLORA) is an approach that addresses this issue by enabling efficient fine-tuning of quantized LLMs.
QLORA combines quantization techniques with Low Rank Adapters (LoRA) to significantly reduce the memory footprint during fine-tuning.
The core idea behind QLORA is to perform gradient backpropagation through a 4-bit quantized pretrained language model into the LoRA layers.
This approach allows for fine-tuning large models, such as those with 65B parameters, on a single 48GB GPU, a feat that was previously impractical due to the excessive memory requirements of regular 16-bit fine-tuning.
QLORA introduces several techniques to optimise memory usage without compromising performance:
4-bit Normal Float (NF4)
NF4 is a new data type that is information-theoretically optimal for representing normally distributed weights. It efficiently captures the statistical properties of the weights, enabling accurate quantization while minimising information loss.
Double Quantization
QLORA employs a technique called Double Quantization, which quantizes the quantization constants themselves. This additional level of quantization further reduces the memory footprint, resulting in an average saving of about 0.37 bits per parameter.
Paged Optimisers
To handle memory spikes during the fine-tuning process, QLORA uses Paged Optimisers. These optimisers leverage NVIDIA's unified memory feature to automatically manage memory transfers between the CPU and GPU, preventing out-of-memory errors and ensuring smooth processing even when the GPU memory is exhausted.
State-of-the-Art Performance Using QLORA, the authors introduce the Guanaco family of models, which achieve state-of-the-art performance on the Vicuna benchmark.
The second-best model in the Guanaco family reaches an impressive 97.8% of ChatGPT's performance level.
With less than 24 hours of training on a professional GPU, the largest Guanaco model attains 99.3% of ChatGPT's performance, effectively closing the gap. Moreover, the smallest Guanaco model, with only 7B parameters, outperforms the larger 26GB Alpaca model on the same benchmark while requiring just 5GB of memory.
QLORA's efficiency enables an extensive analysis of instruction fine-tuning and chatbot performance across various scales, architectures, and datasets.
The authors train over 1,000 models, ranging from 80M to 65B parameters, demonstrating QLORA's ability to achieve 16-bit performance and train state-of-the-art chatbots like Guanaco.
The study reveals intriguing insights, such as the importance of data quality over dataset size.
A smaller dataset of 9k samples outperforms a much larger 450k sample dataset in chatbot performance, emphasising the significance of dataset suitability for specific tasks.
The evaluation methodology employed in the paper is innovative, using tournament-style benchmarking where models compete against each other, with winners determined by either GPT-4 or human annotators.
While GPT-4 and human evaluations largely agree, instances of strong disagreement highlight the uncertainties in model-based evaluation. The qualitative analysis of the Guanaco models provides valuable insights into their successes and failures, complementing the quantitative benchmarks.
Quantization is the process of converting a higher-bit datatype to a lower-bit representation.
In QLORA, quantization is performed by rescaling the input data into the target data type's range through normalization. To handle outliers, the input tensor is divided into blocks, and each block is independently quantized with its own quantization constant.
LoRA is a parameter-efficient fine-tuning method that reduces memory requirements by using a small set of trainable parameters called adapters.
These adapters augment the linear projection of the model through an additional factorised projection. LoRA's minimal memory footprint allows for the use of more adapters to enhance performance without significantly increasing the total memory usage.
Gradient checkpointing is a technique used to reduce the memory consumption during the training of deep neural networks.
Instead of storing all intermediate activations, gradient checkpointing stores only a subset of them, called checkpoints. During backpropagation, the missing gradients between checkpoints are recomputed on-the-fly, enabling the training of deeper models within the same memory constraints.
NF4 quantization is an information-theoretically optimal data type for representing normally distributed weights.
It builds upon Quantile Quantization, aiming to assign an equal number of values to each quantization bin.
The process involves estimating the quantiles of the input tensor, transforming the weights to a fixed distribution, and normalizing them to the range [-1, 1]. Fast quantile approximation algorithms, such as SRAM quantiles, are employed to handle the computational cost of exact quantile estimation.
Double Quantization is a technique used to reduce the memory overhead associated with 4-bit quantization.
It involves quantizing the quantization constants themselves.
First, the quantization constants are quantized (cFP32→cFP8), and then a second quantization step yields quantized constants (cFP8→cFP8) and a second level of quantization constants (cFP32).
By using 8-bit Floats with a block size of 256 for the second quantization, the memory footprint per parameter is reduced from 0.5 bits to 0.127 bits, resulting in a significant reduction of 0.373 bits per parameter.
Paged Optimisers leverage NVIDIA's unified memory feature to facilitate automatic page-to-page transfers between the CPU and GPU.
This feature acts like regular memory paging between CPU RAM and the disk, allowing for error-free GPU processing even when the GPU runs out of memory.
Paged memory is allocated for the optimiser states and automatically transferred to CPU RAM when the GPU memory is exhausted, and then paged back into GPU memory when needed.
The authors compare QLORA with standard full-model fine-tuning to assess whether it can match the performance.
Three architectures are considered for the comparison, including various datasets and models of different sizes. The importance of using LoRA on all transformer layers is emphasised, particularly for LLaMA 7B models, to achieve 16-bit performance.
While paged optimisers are crucial for certain tasks, the authors acknowledge the need for future work to characterise potential slowdowns that might occur due to the paging process.
QLORA represents a significant advancement in the efficient fine-tuning of large language models. By combining quantization techniques, Low Rank Adapters, and innovative memory management strategies.
QLORA enables the training of state-of-the-art chatbots and achieves performance comparable to ChatGPT while significantly reducing memory requirements.
The comprehensive analysis and evaluation conducted in the paper provide valuable insights into the factors influencing model performance, such as data quality and the importance of applying LoRA to all transformer layers.
The introduction of novel techniques like 4-bit Normal Float (NF4) quantization, Double Quantization, and Paged Optimizers demonstrates the potential for further optimization in the fine-tuning process.
These innovations not only reduce memory usage but also maintain high performance, making QLORA a promising approach for researchers and practitioners working with large language models.
