Tokenizer Choice For LLM Training: Negligible or Crucial?
PreviousTokenization exploreNextGetting the most out of your tokenizer for pre-training and domain adaptation
Last updated
Was this helpful?
Continuum Knowledge
Last updated
Was this helpful?
This March 2024 paper investigates the impact of tokenizer choice on the performance of Large Language Models (LLMs), particularly in the context of mono- and multilingual models.
The authors argue that while various factors such as dataset composition, model architecture, and pretraining objectives have been extensively studied, the influence of tokenizers remains underexplored.
The paper provides an overview of various tokenization approaches, including word, subword, and character tokenization.
It also discusses the usage of tokenizers in encoder and decoder models, highlighting the lack of extensive studies on the extrinsic tokenizer performance in a monolingual and multilingual setting with a focus on decoder-only models.
Intrinsic evaluation assesses the performance of a tokenizer based on its inherent properties and the characteristics of its output, without considering its impact on downstream tasks or models. In the context of the study, the intrinsic evaluation focused on three main metrics:
Fertility
Fertility is a measure of the average number of tokens required to represent a word or document after tokenization.
It is calculated by dividing the total number of tokens in a tokenized dataset by the total number of words in the original dataset.
A lower fertility score indicates that the tokenizer is more efficient in representing the text with fewer tokens.
High fertility can lead to increased sequence lengths, which can impact the model's ability to learn long-range dependencies and increase computational costs during training and inference.
Parity
Parity assesses how fairly a tokenizer treats equivalent sentences across different languages.
It measures the consistency of the tokenizer in representing similar semantic content across languages.
In the study, parity was calculated using the FLORES-200 parallel corpus, which contains the same sentences human-translated into 200 languages. The closer the parity scores are to 1, the more consistent the tokenizer is in handling different languages.
Vocabulary Overlap
Vocabulary overlap measures the similarity between the vocabularies of different tokenizers.
It is calculated by comparing the number of shared tokens between two tokenizers' vocabularies.
A high vocabulary overlap indicates that the tokenizers generate similar subword units and may have similar performance characteristics.
Extrinsic evaluation assesses the performance of a tokenizer based on its impact on downstream tasks or models.
In the study, the extrinsic evaluation focused on two main aspects:
The authors trained language models using different tokenizers and evaluated their performance on a wide range of downstream tasks, such as natural language inference, question answering, reading comprehension, and text classification.
The models' performance on these tasks was used to assess the effectiveness of the tokenizers in enabling the models to learn meaningful representations and solve real-world problems.
The study evaluated the computational costs associated with each tokenizer when used in a specific language model.
The computational costs were measured in terms of the average number of floating-point operations (FLOPs) required to process a single word during training.
This metric takes into account the tokenizer's fertility and the model's architecture, such as the number of layers, hidden size, and vocabulary size.
Lower computational costs indicate that the tokenizer is more efficient in terms of resource utilization during training.
The distinction between intrinsic and extrinsic evaluation is important because a tokenizer's intrinsic performance metrics, such as fertility and parity, may not always directly correlate with its impact on downstream tasks.
A tokenizer with good intrinsic performance may not necessarily lead to better downstream performance, and vice versa.
Therefore, considering both intrinsic and extrinsic evaluation metrics provides a more comprehensive understanding of a tokenizer's effectiveness and its suitability for specific applications.
By conducting both intrinsic and extrinsic evaluations, the study aimed to provide insights into the relationship between tokenizer characteristics and their impact on language model performance, helping researchers and practitioners make informed decisions when selecting or designing tokenizers for their specific use cases.
Comprehensive study: The authors trained 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations to assess their impact on downstream performance.
Significant impact: The study reveals that tokenizer choice can significantly affect the model's downstream performance and training costs.
Questionable evaluation metrics: Common tokenizer evaluation metrics like fertility and parity are not always predictive of the model's downstream performance, making them unreliable proxies.
Multilingual tokenizers: Multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three compared to English tokenizers.
English-centric tokenizers: Applying English-centric tokenizers to multilingual LLMs results in severe downstream performance degradation and additional training costs of up to 68% due to an inefficient tokenization vocabulary.
The authors created two datasets (monolingual English and multilingual) with 70B words each, ensuring that the mixture proportions of data domains were consistent between the tokenizer and model training datasets to avoid domain shift.
They trained 24 different tokenizers using BPE and Unigram algorithms, varying the language composition, vocabulary size, and tokenizer library (Huggingface and SentencePiece).
For each of the 24 trained tokenizers, they trained a 2.6B transformer-based decoder-only model on up to 52B tokens, following a specific scaling law. They also trained monolingual and multilingual baseline models using the pre-trained GPT-2 tokenizer.
The authors conducted both intrinsic and extrinsic evaluations of the tokenizers.
The intrinsic evaluation assessed the tokenizers' output based on fertility, parity, and vocabulary overlap, while the extrinsic evaluation measured the impact of tokenizers on the model's downstream performance and computational costs.
Evaluate and compare different tokenizer algorithms, configurations, and libraries for their specific use case, considering both intrinsic and extrinsic performance metrics.
Design and train custom tokenizers tailored to the target languages and domains of their LLMs, ensuring optimal downstream performance and computational efficiency.
Use the provided computational cost formula to estimate the impact of tokenizer choice and model architecture on training costs, and optimize their setups accordingly.
By applying these insights and approaches, practitioners can develop more efficient and effective LLMs, particularly in multilingual settings, ultimately leading to better performance on downstream tasks and more accessible language technology for a wider range of users.
If you are working with multilingual data, it is crucial to use a multilingual tokenizer.
Using a monolingual tokenizer on multilingual data leads to significantly higher fertility and parity scores, indicating inefficient tokenization.
Multilingual tokenizers perform only slightly worse on English documents compared to monolingual English tokenizers, making them a better choice for multilingual settings.
The optimal vocabulary size depends on the language(s) you are working with.
For monolingual English settings, smaller vocabulary sizes (around 33k-50k) tend to perform better.
However, for multilingual settings, larger vocabulary sizes (up to 100k) generally yield better downstream performance, especially for non-Germanic languages. Consider the trade-off between performance and computational costs when choosing the vocabulary size.
The choice of tokenizer library (e.g., Huggingface vs. SentencePiece) can impact the downstream performance.
In the study, SentencePiece's BPE implementation generally outperformed Huggingface's BPE in both monolingual and multilingual settings.
Consider the differences in pre- and post-processing steps between libraries when selecting a tokenizer.
The choice between BPE and Unigram algorithms may depend on the target language(s).
In the study, Germanic languages (German and English) benefited more from BPE, while Romanic languages (French and Spanish) benefited more from Unigram.
Consider the linguistic properties of your target language(s) when choosing the tokenizer algorithm.
Larger vocabulary sizes generally increase computational costs, even if they lead to lower fertility scores.
When choosing a tokenizer, consider the trade-off between downstream performance and computational costs, especially if you need to process a fixed set of documents during training.
Using pre-trained tokenizers (e.g., GPT-2) for multilingual models may lead to suboptimal performance. It is better to train a custom tokenizer tailored to your specific language(s) and domain.
The study provides valuable insights into the impact of tokenizer choice on the downstream performance of language models, particularly in monolingual and multilingual settings.
The findings highlight the importance of training tokenizers with a balanced share across languages to achieve comparable low fertility and parity scores, which has significant implications for computational costs and the model's ability to learn long-range dependencies.
The study demonstrates that the tokenizer choice can significantly impact the model's downstream performance, with the BPE algorithm performing well in both mono- and multilingual settings.
For English, a vocabulary size of 33k is sufficient, while multilingual models based on the five considered languages require up to three times larger vocabulary sizes. Additionally, the SentencePiece library outperforms the Huggingface tokenizer library.
Interestingly, the study finds no clear correlation between intrinsic and extrinsic tokenizer performance, suggesting that the correlation is rather task-specific. A small fertility value might be a necessary condition for good downstream performance but not a sufficient one.
