TokenMonster

TokenMonster is an ungreedy subword tokenizer and vocabulary generator designed to improve the efficiency and performance of language models.

It selects an optimal vocabulary for a given dataset, resulting in up to 37.5% fewer tokens required to represent text compared to other modern tokenizing methods. This allows for faster inference, training, and longer text generation.

GitHub - alasdairforsythe/tokenmonster: Ungreedy subword tokenizer and vocabulary trainer for Python, Go & JavascriptGitHub

Key technical features include

• Ungreedy tokenization algorithm that follows up to 6 parallel branches

• Supports 5 optimisation modes: unfiltered, clean, balanced, consistent, strict

• Uses capcode marker tokens to encode uppercasing and forward delete

• Identifies words, subwords, common phrases, and figures of speech

• Achieves up to 7 characters per token depending on vocabulary size and optimization mode

• Provides 422 pretrained vocabularies and tools to train custom vocabularies

• Implementations available in Go, Python, and JavaScript

Using TokenMonster in Practice

1. Choose a suitable pretrained vocabulary based on your dataset (e.g., code, English, fiction), desired vocabulary size, and optimization mode. Alternatively, train a custom vocabulary using the provided tools.

2. Install the TokenMonster library in your preferred language (Go, Python, or JavaScript).

3. Load the selected vocabulary:

import tokenmonster
vocab = tokenmonster.load("englishcode-32000-consistent-v1")

Tokenize your text using the loaded vocabulary:

tokens = vocab.tokenize("This is a test.")

Integrate the tokenized text into your language model training or inference pipeline to benefit from the optimized vocabulary and improved efficiency.

By using TokenMonster, you can potentially reduce the vocabulary size of your language model by 50-75% while maintaining or improving performance.

This frees up resources that can be used to make the model smarter and faster.

The ungreedy tokenization algorithm and carefully selected vocabularies enable more efficient usage of embeddings and simpler grammar for the model to learn.

An explanation

tokenmaster.py codebase

The tokenmaster.py codebase is a Python library for the TokenMonster tokenizer.

It provides an interface to load, modify, and use TokenMonster vocabularies for efficient tokenization and detokenization of text.

Key components and usage

1. Loading a vocabulary:

   • Use tokenmonster.load(path) to load a vocabulary from a file, URL, or pre-built vocabulary name.

   • For multiprocessing, use tokenmonster.load_multiprocess_safe(path) to load the vocabulary safely.

2. Tokenizing text:

   • Use vocab.tokenize(text) to tokenize a string or a list of strings into token IDs.

   • The method returns a numpy array or a list of numpy arrays containing the token IDs.

3. Decoding tokens:

   • Use vocab.decode(tokens) to decode a single token ID or a list of token IDs back into a string.

   • For decoding token streams sequentially, create a decoder object using decoder = vocab.decoder() and use decoder.decode(tokens) to decode tokens incrementally.

4. Modifying the vocabulary:

   • Use vocab.modify() to add or delete tokens, resize the vocabulary, enable/disable the UNK token, or reset token IDs.

   • Modifications can also be made using individual methods like add_token(), delete_token(), add_special_token(), etc.

   • After modifying the vocabulary, save it using vocab.save(filename).

5. Accessing vocabulary information:

   • Use vocab.get_dictionary() to retrieve a dictionary of all tokens in the vocabulary.

   • Access properties like vocab.vocab_size, vocab.unk_token_id(), vocab.capcode(), vocab.charset(), etc., to get information about the vocabulary.

To build your own tokenizer

1. Prepare a dataset of text that represents the domain you want to tokenize.

2. Use the tokenmonster library to train a new vocabulary on your dataset:

   • Create a new vocabulary using vocab = tokenmonster.new(yaml), where yaml is a YAML string defining the vocabulary configuration.

   • Customise the vocabulary configuration, specifying the desired vocabulary size, optimization mode, and other parameters.

   • Save the trained vocabulary using vocab.save(filename).

3. Use the trained vocabulary to tokenize and detokenize text in your application:

   • Load the saved vocabulary using vocab = tokenmonster.load(filename).

   • Tokenize text using tokens = vocab.tokenize(text).

   • Decode tokens back into text using decoded_text = vocab.decode(tokens).

The key inputs for building your own tokenizer are:

• A representative dataset of text for training the vocabulary.

• A YAML configuration file specifying the vocabulary parameters (size, optimization mode, etc.).

By following these steps and leveraging the tokenmonster library, you can build a custom tokenizer optimized for your specific domain and use case.

Remember to handle any errors and exceptions appropriately, and refer to the documentation and examples provided in the TokenMonster repository for more detailed guidance on using the library.

Case Study: Training a TokenMonster Tokenizer for Medical Text

Objective: Build a domain-specific tokenizer for medical text to improve the efficiency and accuracy of a fine-tuned language model for medical question answering.

Process

Data Collection

• Gather a large corpus of medical text, such as medical research papers, clinical notes, and medical textbooks.

• Ensure the dataset is representative of the medical domain and covers various medical specialties and terminology.

Data Preprocessing

• Clean the dataset by removing any irrelevant information, such as headers, footers, or metadata.

• Normalize the text by handling special characters, converting to lowercase, and addressing any domain-specific formatting.

Vocabulary Training

• Prepare a YAML configuration file specifying the desired vocabulary size (e.g., 32,000 tokens), optimization mode (e.g., "consistent"), and any additional settings.

• Create a new TokenMonster vocabulary using the preprocessed medical text dataset:

import tokenmonster
yaml_config = """
vocab_size: 32000
optimization_mode: consistent
"""
vocab = tokenmonster.new(yaml_config)

Train the vocabulary on the medical text dataset

medical_text = load_medical_dataset()
vocab.tokenize(medical_text)

Save the trained vocabulary

vocab.save("medical_tokenizer.vocab")

Fine-tuning the Language Model

Load the trained TokenMonster vocabulary

medical_tokenizer = tokenmonster.load("medical_tokenizer.vocab")

Tokenize the medical text dataset using the trained tokenizer:

tokenized_medical_text = medical_tokenizer.tokenize(medical_text)

• Fine-tune a pre-trained language model (e.g., BERT, RoBERTa) on the tokenized medical text dataset.

• During fine-tuning, use the TokenMonster vocabulary to tokenize the input text and convert the model's output tokens back to text.

Inference and Evaluation

• Load the fine-tuned medical language model.

• For inference, tokenize the input medical questions using the TokenMonster tokenizer:

equestion = "What are the symptoms of pneumonia?"
tokenized_question = medical_tokenizer.tokenize(question)

• Feed the tokenized question to the fine-tuned model and obtain the predicted answer tokens.

• Decode the predicted answer tokens back to text using the TokenMonster tokenizer:

predicted_answer_tokens = model.predict(tokenized_question)
predicted_answer = medical_tokenizer.decode(predicted_answer_tokens)

• Evaluate the model's performance using appropriate metrics for medical question answering, such as accuracy, F1 score, or BLEU score.

By incorporating TokenMonster into your workflow, you can create domain-specific tokenizers that capture the unique vocabulary and patterns of your target domain. This can lead to improved efficiency and accuracy in fine-tuning language models for specialized tasks, such as medical question answering, legal document analysis, or scientific text generation.

Remember to evaluate the performance of your fine-tuned model and iterate on the tokenizer training and model fine-tuning process to achieve the best results for your specific use case.