> For the complete documentation index, see [llms.txt](https://training.continuumlabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://training.continuumlabs.ai/training.md). # Training - [The Fine Tuning Process](https://training.continuumlabs.ai/training/the-fine-tuning-process.md): Fine tuning deep learning models is completely different to fine tuning machine learning models - [Why fine tune?](https://training.continuumlabs.ai/training/the-fine-tuning-process/why-fine-tune.md) - [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://training.continuumlabs.ai/training/the-fine-tuning-process/why-fine-tune/does-fine-tuning-llms-on-new-knowledge-encourage-hallucinations.md) - [Explanations in Fine Tuning](https://training.continuumlabs.ai/training/the-fine-tuning-process/why-fine-tune/explanations-in-fine-tuning.md) - [Tokenization](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization.md) - [Tokenization Is More Than Compression](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenization-is-more-than-compression.md): Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, Chris Tanner - [Tokenization - SentencePiece](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenization-sentencepiece.md): The Unsupervised Text Tokenizer for Neural Networks - [Tokenization explore](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenization-explore.md) - [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenizer-choice-for-llm-training-negligible-or-crucial.md) - [Getting the most out of your tokenizer for pre-training and domain adaptation](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/getting-the-most-out-of-your-tokenizer-for-pre-training-and-domain-adaptation.md) - [TokenMonster](https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenmonster.md) - [Parameter Efficient Fine Tuning](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning.md) - [P-Tuning](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/p-tuning.md): The highly cited "GPT Understands Too" paper first submitted March 2021, introducing P-Tuning - [The Power of Scale for Parameter-Efficient Prompt Tuning](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/p-tuning/the-power-of-scale-for-parameter-efficient-prompt-tuning.md) - [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/prefix-tuning-optimizing-continuous-prompts-for-generation.md) - [Harnessing the Power of PEFT: A Smarter Approach to Fine-tuning Pre-trained Models](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/harnessing-the-power-of-peft-a-smarter-approach-to-fine-tuning-pre-trained-models.md): Parameter-Efficient Fine-Tuning (PEFT) is a technique used to fine tune neural language models - [What is Low-Rank Adaptation (LoRA) - explained by the inventor](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/what-is-low-rank-adaptation-lora-explained-by-the-inventor.md): Edward Hu - [Low Rank Adaptation (Lora)](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/low-rank-adaptation-lora.md) - [Practical Tips for Fine-tuning LMs Using LoRA (Low-Rank Adaptation)](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/practical-tips-for-fine-tuning-lms-using-lora-low-rank-adaptation.md) - [QLORA: Efficient Finetuning of Quantized LLMs](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/qlora-efficient-finetuning-of-quantized-llms.md) - [Bits and Bytes](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/bits-and-bytes.md): Tim Dettmers (PhD candidate, University of Washington) presents "8-bit Methods for Efficient Deep Learning" in this Cohere For AI Technical Talk. - [The Magic behind Qlora](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/the-magic-behind-qlora.md) - [Practical Guide to LoRA: Tips and Tricks for Effective Model Adaptation](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/practical-guide-to-lora-tips-and-tricks-for-effective-model-adaptation.md): A range of practical tips and questions around using Lora - [The quantization constant](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/the-quantization-constant.md) - [QLORA: Efficient Finetuning of Quantized Language Models](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/qlora-efficient-finetuning-of-quantized-language-models.md) - [QLORA and Fine-Tuning of Quantized Language Models (LMs)](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/qlora-and-fine-tuning-of-quantized-language-models-lms.md) - [ReLoRA: High-Rank Training Through Low-Rank Updates](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/relora-high-rank-training-through-low-rank-updates.md) - [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/slora-federated-parameter-efficient-fine-tuning-of-language-models.md): Leveraging Lora - [GaLora: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://training.continuumlabs.ai/training/the-fine-tuning-process/parameter-efficient-fine-tuning/galora-memory-efficient-llm-training-by-gradient-low-rank-projection.md) - [Hyperparameters](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters.md): Art and science - [Batch Size](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/batch-size.md): Choosing the right batch size is critical - [Padding Tokens](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/padding-tokens.md) - [Mixed precision training](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/mixed-precision-training.md) - [FP8 Formats for Deep Learning](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/fp8-formats-for-deep-learning.md) - [Floating Point Numbers](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/floating-point-numbers.md) - [Batch Size and Model loss](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/batch-size-and-model-loss.md) - [Batch Normalisation](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/batch-normalisation.md) - [Rethinking Learning Rate Tuning in the Era of Language Models](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/rethinking-learning-rate-tuning-in-the-era-of-language-models.md): One of the most important hyperparameters - [Sample Packing](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/sample-packing.md) - [Gradient accumulation](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/gradient-accumulation.md) - [A process for choosing the learning rate](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/a-process-for-choosing-the-learning-rate.md) - [Learning Rate Scheduler](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/learning-rate-scheduler.md): Key Considerations with Learning Rate Scheduling in Neural Network Training - [Checkpoints](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/checkpoints.md) - [A Survey on Efficient Training of Transformers](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/a-survey-on-efficient-training-of-transformers.md) - [Sequence Length Warmup](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/sequence-length-warmup.md) - [Understanding Training vs. Evaluation Data Splits](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/understanding-training-vs.-evaluation-data-splits.md) - [Cross-entropy loss](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/cross-entropy-loss.md) - [Weight Decay](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/weight-decay.md) - [Optimiser](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/optimiser.md) - [Caching](https://training.continuumlabs.ai/training/the-fine-tuning-process/hyperparameters/caching.md) - [Training Processes](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes.md) - [Extending the context window](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/extending-the-context-window.md) - [PyTorch Fully Sharded Data Parallel (FSDP)](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/pytorch-fully-sharded-data-parallel-fsdp.md) - [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/train-short-test-long-attention-with-linear-biases-enables-input-length-extrapolation.md): Ofir Press, Noah A. Smith, Mike Lewis - [YaRN: Efficient Context Window Extension of Large Language Models](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/yarn-efficient-context-window-extension-of-large-language-models.md): Nous Research, EleutherAI, University of Geneva - [Sliding Window Attention](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/sliding-window-attention.md): Iz Beltagy, Matthew E. Peters, and Arman Cohan - [LongRoPE](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/longrope.md) - [Reinforcement Learning](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/reinforcement-learning.md) - [An introduction to reinforcement learning](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/an-introduction-to-reinforcement-learning.md) - [Reinforcement Learning from Human Feedback (RLHF)](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/reinforcement-learning-from-human-feedback-rlhf.md): Most often useful when creating domain specific models - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://training.continuumlabs.ai/training/the-fine-tuning-process/training-processes/direct-preference-optimization-your-language-model-is-secretly-a-reward-model.md)