Padding Tokens

What are padding tokens?

Padding tokens are special tokens added to sequences in a batch to make them all have the same length.

In natural language processing (NLP) and machine learning tasks, input data often consists of variable-length sequences, such as sentences or phrases. However, most deep learning models, like recurrent neural networks (RNNs) and transformers, require fixed-length input sequences to process data efficiently.

Padding tokens help address this issue by filling the shorter sequences with non-informative tokens until they match the length of the longest sequence in the batch. These padding tokens usually have no semantic meaning and are ignored by the model during training and inference.1

By padding the shorter sequences with non-informative tokens, the input data can be represented as a fixed-size tensor that the model can process efficiently. This fixed-size tensor is usually a matrix or a higher-dimensional tensor with consistent dimensions across all samples in the batch.

Moreover, using padding tokens helps maintain the structure and readability of the input data. Since padding tokens have no semantic meaning and are ignored during processing, they don't interfere with the model's understanding of the actual content.

In summary, padding is done to facilitate efficient parallel processing of input samples in deep learning models by ensuring that input tensors have consistent dimensions, rather than to satisfy any specific matrix symmetry requirements.

Paddings across tokenization techniques

Therefore, when dealing with sequences, whether they are tokenized at the character level, word level, subword level, or using Byte Pair Encoding (BPE), it is often necessary to ensure that the sequences fed into a model are of consistent length.

Here's how paddings relate to the different levels of tokenization:

Character Level

Sequences of characters can vary considerably in length. For instance, the word "apple" has 5 characters, while "characterization" has 15.

In tasks like character-level sequence-to-sequence modeling (e.g., for machine translation or spelling correction), sequences might be tokenized into characters, and then padding is used to ensure input and output sequences have consistent lengths.

Word Level

In models like RNNs, LSTM, or GRU that operate at the word level, sentences or documents are tokenized into words. Since sentences can vary greatly in length, padding is used to ensure all sequences in a batch have the same length.

For instance, "I love apples." might be tokenized as ["I", "love", "apples", "."], but to fit this in a batch with longer sentences, padding tokens might be added.

Subword Level

Sometimes words are broken down into smaller units, which are not as small as characters but not as long as full words. This is useful for languages or texts with many compound words or morphological variations.

After tokenization, sequences of subwords might still vary in length, requiring padding for consistent model input.

BPE (Byte Pair Encoding)

BPE is a type of subword tokenization. It starts with character-level tokenization and then iteratively merges the most frequent pairs of characters/subwords until a certain vocabulary size is reached.

Even after BPE tokenization, sentences will result in sequences of varying lengths, hence the need for padding.

How is Padding Typically Done?

A special token (often "<PAD>" or just 0 when dealing with integer representations of tokens) is introduced.

Sequences shorter than a specified maximum length are filled with this padding token up to that length.

In deep learning frameworks like TensorFlow and PyTorch, this is usually done using built-in functions (pad_sequences in TensorFlow's Keras API, or pad_packed_sequence in PyTorch).

It's important to note that during model training or evaluation, these padding tokens need to be masked or ignored to ensure they don't influence the model's computations and outputs.

Note: While padding ensures consistency in sequence lengths, it's essential to handle it properly. For many deep learning models, especially transformers, attention masks are used in tandem with padding to ensure the model doesn't "pay attention" to padding tokens.

Here's a simple example to illustrate the concept of padding tokens:

Original sequences (variable length):

["I", "love", "NLP"]

["This", "is", "a", "great", "example"]

After adding padding tokens (fixed length):

["I", "love", "NLP", "<PAD>", "<PAD>"]

["This", "is", "a", "great", "example"]

In this example, the padding token is represented by "<PAD>". It's added to the first sequence to match the length of the second sequence, which is the longest in the batch.

When using padding tokens, it's crucial to also use attention masks or other mechanisms to inform the model which tokens are actual content and which ones are padding. This ensures that the model doesn't process the padding tokens and that they don't contribute to the model's predictions or loss calculation.

Padding is done primarily to facilitate efficient batch processing in deep learning models.

While the reason is not specifically related to the requirement of matrices being symmetrical, it has to do with the need for input tensors to have consistent dimensions when being processed in parallel.