# StreamingLLM The **September 2023** paper "Efficient Streaming Language Models with Attention Sinks" introduces StreamingLLM, a framework that enables Large Language Models (LLMs) trained with a finite attention window to generalise to infinite sequence lengths without fine-tuning. The main challenge in applying LLMs to infinite input streams is the quadratic memory and computational complexity of the attention mechanism, which limits the model's ability to handle longer sequences than it was trained on. {% embed url="" %} Efficient Streaming Language Models with Attention Sinks {% endembed %} {% embed url="" %} Github repository - over 6,000 stars and counting {% endembed %} ### Technical terms * **Key and Value states (KV):** In Transformer-based LLMs, the Key and Value states are cached for all previous tokens during the decoding stage. * **Attention window:** The maximum sequence length that the model is trained on, constraining the model's ability to generalise to longer sequences. * **Quadratic attention:** The computational complexity of the attention mechanism, which scales quadratically with the sequence length. * **Softmax operation:** A function that normalizes the attention scores, ensuring they sum up to one for all contextual tokens. * **Autoregressive language modeling:** A type of language modeling where the model predicts the next token based on the previous tokens in the sequence. ### The authors discuss two existing approaches #### **Window attention** The window attention is a technique that maintains a fixed-size sliding window on the Key-Value (KV) states of the most recent tokens. While this approach ensures constant memory usage and decoding speed, the model's performance collapses once the sequence length exceeds the cache size, and the initial tokens are evicted. #### Sliding window with re-computation Rebuilds the KV states of recent tokens for each generated token, offering strong performance but significantly slower due to the computation of quadratic attention within its window. ### The Concept of Attention Sinks The authors discover an interesting phenomenon called "attention sink," where a surprisingly ***large amount of attention score is allocated to the initial tokens***, irrespective of their relevance to the language modeling task. They attribute this to the **softmax operation**. The **softmax function** prevents all attended tokens from having zero values, requiring the model to aggregate information from other tokens even if the current embedding has sufficient self-contained information for prediction. Consequently, the model tends to **dump unnecessary attention values to specific tokens**, which are typically the initial tokens due to their visibility to all subsequent tokens in autoregressive language modeling. Based on these insights, the authors propose StreamingLLM, which keeps the attention sink tokens' KV (just 4 initial tokens suffice) together with the sliding window's KV to anchor the attention computation and stabilise the model's performance.

The diagram above compares StreamingLLM with existing methods for handling long input sequences in language models. The language model is pre-trained on texts of length $$L$$ and is tasked with predicting the $$Tth$$ token, where $$T$$ is much greater than $$L$$. The comparison includes: **(a) Dense Attention:** It has a time complexity of $$O(T^2)$$ and an increasing cache size. The model's performance decreases when the text length exceeds the pre-training text length. **(b) Window Attention:** It caches the Key and Value (KV) states of the most recent $$L$$ tokens. While efficient in inference, the performance declines sharply once the starting tokens' keys and values are evicted from the cache. **(c) Sliding Window with Re-computation:** It rebuilds the KV states from the $$L$$ most recent tokens for each new token. Although it performs well on long texts, its $$O(TL^2)$$ complexity, stemming from quadratic attention in context re-computation, makes it considerably slow. **(d) StreamingLLM:** It keeps the attention sink (several initial tokens) for stable attention computation, combined with the recent tokens. It is efficient and offers stable performance on extended texts. StreamingLLM addresses the limitations of existing methods by introducing an attention sink, which consists of ***several initial tokens that stabilise the attention computation***. By combining the attention sink with the most recent tokens, StreamingLLM achieves efficient and stable performance on long input sequences, outperforming dense attention, window attention, and sliding window with re-computation approaches. ### Key Advantages of StreamingLLM * **Extended Context Without Re-Training**: StreamingLLM allows models to handle text sequences of virtually unlimited length without the need for model retraining or modification. * **Efficient and High-Quality Inference**: It addresses the challenges of previous methods, offering a solution that is fast, maintains high quality, and requires low memory. * **Model Compatibility**: StreamingLLM is compatible with various LLMs like Llama-2, Falcon, and Pythia, enabling them to model up to 4 million tokens effectively. ### Implementation and Future Potential * **Publicly Accessible Code**: The code for StreamingLLM is available on GitHub, offering compatibility with several LLMs and integration with Hugging Face transformers libraries. * **Enhanced Language Modeling Applications**: With StreamingLLM, LLMs can be applied to tasks requiring processing of much longer text sequences, such as prolonged chat sessions or comprehensive document analysis, without compromising on performance or incurring prohibitive costs. StreamingLLM presents an innovative approach to extend the context window of Large Language Models (LLMs) like Transformers, but it's not without potential challenges or drawbacks. ### Here are problems that might arise with StreamingLLM ### Dependency on Initial Tokens (Attention Sinks) * **Reliance on Specific Tokens**: StreamingLLM relies heavily on maintaining the initial tokens (attention sinks) in the model's KV (Key-Value) cache. This reliance could be problematic if the initial tokens are not sufficiently representative or relevant to the ongoing context. * **Potential for Irrelevant Context Preservation**: If the initial tokens are not closely related to the current topic of discussion or text, their preservation may not contribute meaningfully to the model's understanding and could even introduce noise or irrelevant context. ### Handling of Evolving Contexts in Long Conversations * **Contextual Relevance Over Time**: In prolonged conversations or text sequences, the relevance of initial tokens might diminish as the subject evolves. StreamingLLM’s mechanism might struggle to adapt to these changes, potentially leading to less accurate or relevant outputs. * **Complexity in Dynamic Conversations**: The model might face challenges in dynamically evolving conversations where new information significantly changes the context or where the conversation shifts to entirely different topics. ### Computational Efficiency and Throughput * **Trade-Offs in Efficiency**: While StreamingLLM aims to be computationally efficient, the process of maintaining a rolling KV cache and managing the attention sinks could still introduce computational overhead, especially in very long sequences. * **Throughput Concerns**: The need to constantly update and manage the KV cache for attention sinks might impact the throughput of the model, affecting its real-time responsiveness in applications like interactive chatbots or live document editing. ### Model Generalisation and Training * **Pre-Training Constraints**: StreamingLLM’s approach necessitates certain considerations during the pre-training phase, like the inclusion of a global trainable attention sink token. This requirement could impose constraints on the general pre-training process of LLMs. * **Potential Impact on Model Flexibility**: The specific design choices and architecture adjustments required for StreamingLLM might impact the model's flexibility and generalization capabilities across different types of tasks and datasets. ### Quality and Consistency of Outputs * **Quality Maintenance in Extended Contexts**: There’s a potential challenge in maintaining the quality and consistency of the model’s outputs as the context window extends significantly. Ensuring that the model remains coherent and contextually accurate over long text sequences is crucial. * **Balancing Context and Relevance**: StreamingLLM must balance the retention of old context (through attention sinks) with the incorporation of new information. Achieving this balance without losing relevance or coherence can be challenging, especially in complex or nuanced text sequences. While StreamingLLM offers a promising solution to the context window limitation of Transformers, these potential challenges highlight the complexity and nuances involved in implementing such a system effectively.