# TokenMonster TokenMonster is an ungreedy subword tokenizer and vocabulary generator designed to improve the efficiency and performance of language models. It selects an optimal vocabulary for a given dataset, resulting in up to 37.5% fewer tokens required to represent text compared to other modern tokenizing methods. This allows for faster inference, training, and longer text generation. {% embed url="" %} ### Key technical features include * Ungreedy tokenization algorithm that follows up to 6 parallel branches * Supports 5 optimisation modes: unfiltered, clean, balanced, consistent, strict * Uses capcode marker tokens to encode uppercasing and forward delete * Identifies words, subwords, common phrases, and figures of speech * Achieves up to 7 characters per token depending on vocabulary size and optimization mode * Provides 422 pretrained vocabularies and tools to train custom vocabularies * Implementations available in Go, Python, and JavaScript ### Using TokenMonster in Practice 1. Choose a suitable pretrained vocabulary based on your dataset (e.g., code, English, fiction), desired vocabulary size, and optimization mode. Alternatively, train a custom vocabulary using the provided tools. 2. Install the TokenMonster library in your preferred language (Go, Python, or JavaScript). 3. Load the selected vocabulary: ```python import tokenmonster vocab = tokenmonster.load("englishcode-32000-consistent-v1") ``` Tokenize your text using the loaded vocabulary: ```python tokens = vocab.tokenize("This is a test.") ``` Integrate the tokenized text into your language model training or inference pipeline to benefit from the optimized vocabulary and improved efficiency. By using TokenMonster, you can potentially reduce the vocabulary size of your language model by 50-75% while maintaining or improving performance. This frees up resources that can be used to make the model smarter and faster. The ungreedy tokenization algorithm and carefully selected vocabularies enable more efficient usage of embeddings and simpler grammar for the model to learn. An explanation
tokenmaster.py codebase The `tokenmaster.py` codebase is a Python library for the TokenMonster tokenizer. It provides an interface to load, modify, and use TokenMonster vocabularies for efficient tokenization and detokenization of text. **Key components and usage** 1. Loading a vocabulary: * Use `tokenmonster.load(path)` to load a vocabulary from a file, URL, or pre-built vocabulary name. * For multiprocessing, use `tokenmonster.load_multiprocess_safe(path)` to load the vocabulary safely. 2. Tokenizing text: * Use `vocab.tokenize(text)` to tokenize a string or a list of strings into token IDs. * The method returns a numpy array or a list of numpy arrays containing the token IDs. 3. Decoding tokens: * Use `vocab.decode(tokens)` to decode a single token ID or a list of token IDs back into a string. * For decoding token streams sequentially, create a decoder object using `decoder = vocab.decoder()` and use `decoder.decode(tokens)` to decode tokens incrementally. 4. Modifying the vocabulary: * Use `vocab.modify()` to add or delete tokens, resize the vocabulary, enable/disable the UNK token, or reset token IDs. * Modifications can also be made using individual methods like `add_token()`, `delete_token()`, `add_special_token()`, etc. * After modifying the vocabulary, save it using `vocab.save(filename)`. 5. Accessing vocabulary information: * Use `vocab.get_dictionary()` to retrieve a dictionary of all tokens in the vocabulary. * Access properties like `vocab.vocab_size`, `vocab.unk_token_id()`, `vocab.capcode()`, `vocab.charset()`, etc., to get information about the vocabulary. **To build your own tokenizer** 1. Prepare a dataset of text that represents the domain you want to tokenize. 2. Use the `tokenmonster` library to train a new vocabulary on your dataset: * Create a new vocabulary using `vocab = tokenmonster.new(yaml)`, where `yaml` is a YAML string defining the vocabulary configuration. * Customise the vocabulary configuration, specifying the desired vocabulary size, optimization mode, and other parameters. * Save the trained vocabulary using `vocab.save(filename)`. 3. Use the trained vocabulary to tokenize and detokenize text in your application: * Load the saved vocabulary using `vocab = tokenmonster.load(filename)`. * Tokenize text using `tokens = vocab.tokenize(text)`. * Decode tokens back into text using `decoded_text = vocab.decode(tokens)`. The key inputs for building your own tokenizer are: * A representative dataset of text for training the vocabulary. * A YAML configuration file specifying the vocabulary parameters (size, optimization mode, etc.). By following these steps and leveraging the `tokenmonster` library, you can build a custom tokenizer optimized for your specific domain and use case. Remember to handle any errors and exceptions appropriately, and refer to the documentation and examples provided in the TokenMonster repository for more detailed guidance on using the library.
### Case Study: Training a TokenMonster Tokenizer for Medical Text Objective: Build a domain-specific tokenizer for medical text to improve the efficiency and accuracy of a fine-tuned language model for medical question answering. ### Process #### Data Collection * Gather a large corpus of medical text, such as medical research papers, clinical notes, and medical textbooks. * Ensure the dataset is representative of the medical domain and covers various medical specialties and terminology. #### Data Preprocessing * Clean the dataset by removing any irrelevant information, such as headers, footers, or metadata. * Normalize the text by handling special characters, converting to lowercase, and addressing any domain-specific formatting. #### Vocabulary Training * Prepare a YAML configuration file specifying the desired vocabulary size (e.g., 32,000 tokens), optimization mode (e.g., "consistent"), and any additional settings. * Create a new TokenMonster vocabulary using the preprocessed medical text dataset: ```python import tokenmonster yaml_config = """ vocab_size: 32000 optimization_mode: consistent """ vocab = tokenmonster.new(yaml_config) ``` #### Train the vocabulary on the medical text dataset ```python medical_text = load_medical_dataset() vocab.tokenize(medical_text) ``` #### Save the trained vocabulary ```python vocab.save("medical_tokenizer.vocab") ``` ### Fine-tuning the Language Model #### Load the trained TokenMonster vocabulary ```python medical_tokenizer = tokenmonster.load("medical_tokenizer.vocab") ``` #### Tokenize the medical text dataset using the trained tokenizer: ```python tokenized_medical_text = medical_tokenizer.tokenize(medical_text) ``` * Fine-tune a pre-trained language model (e.g., BERT, RoBERTa) on the tokenized medical text dataset. * During fine-tuning, use the TokenMonster vocabulary to tokenize the input text and convert the model's output tokens back to text. ### Inference and Evaluation * Load the fine-tuned medical language model. * For inference, tokenize the input medical questions using the TokenMonster tokenizer: ```python equestion = "What are the symptoms of pneumonia?" tokenized_question = medical_tokenizer.tokenize(question) ``` * Feed the tokenized question to the fine-tuned model and obtain the predicted answer tokens. * Decode the predicted answer tokens back to text using the TokenMonster tokenizer: ```python predicted_answer_tokens = model.predict(tokenized_question) predicted_answer = medical_tokenizer.decode(predicted_answer_tokens) ``` * Evaluate the model's performance using appropriate metrics for medical question answering, such as accuracy, F1 score, or BLEU score. By incorporating TokenMonster into your workflow, you can create domain-specific tokenizers that capture the unique vocabulary and patterns of your target domain. This can lead to improved efficiency and accuracy in fine-tuning language models for specialized tasks, such as medical question answering, legal document analysis, or scientific text generation. Remember to evaluate the performance of your fine-tuned model and iterate on the tokenizer training and model fine-tuning process to achieve the best results for your specific use case. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://training.continuumlabs.ai/training/the-fine-tuning-process/tokenization/tokenmonster.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.