Lessons Learned on LLM RAG Solutions

Injecting data via embedding model into your vector database for future retrieval

In Retrieval Augmented Generation, an embedded knowledge base is created in a vector database.

This knowledge base is queried with a user's prompt to retrieve semantically relevant content.

This content is then injected into the context window of a large language model, allowing it to provide responses based on both its training and the retrieved, relevant real-world data and specified domain knowledge.

Retrieval is not just about finding the most similar text but ensuring it is contextually relevant. The retrieval step is seen as the most crucial part of a RAG application, as incorrect or outdated information retrieved can lead to inaccurate responses from the model.

Summary of process

Creating a knowledge base for RAG involves choosing the domain knowledge carefully, structuring the data and then chunking the content into contextually relevant pieces. The chunk are then passed through an embedding model.

The resulting embeddings are then stored in a vector database.

Source Material Diversity

The source material for RAG isn't limited to text documents. It could include a variety of formats like PDFs, Word documents, PowerPoint slides, etc. This diversity adds complexity to the process of converting these materials into a suitable format for the model.

Document Parsing Challenges

One of the initial and significant challenges in RAG projects is parsing documents in a way that retains crucial document structure. This parsing process can be more complex and critical than the actual embedding and retrieval process.

Techniques for Parsing Documents

Different types of documents may require specific parsing libraries and techniques. The goal is to convert these documents into a data structure that preserves meaning while being suitable for embedding generation and machine learning operations.

Workflow for Handling Diverse Data

Expect to spend significant time setting up libraries, writing heuristics for parsing different document types, and structuring data appropriately. This groundwork is essential for ensuring the data fed into the model is in the right format and maintains its intended meaning.

Embedding Generation

The process involves converting chunks of text into numerical vector embeddings, allowing for the identification of text with similar meanings through mathematical techniques. Embeddings are crucial for matching a user's query with the most relevant text from the data source.

Embedding Model Choice and Preparatory Work

Choosing the right embedding model is essential, but it's also about the preparatory work that goes into structuring and processing the data before it's even passed through the embedding model. This preparation can significantly impact the quality of the retrieval process.

Challenges in Scaling RAG Applications

Scaling RAG involves challenges like generating embeddings at scale (which is compute-intensive and typically requires GPUs), data sanitisation, continuous processing of new and updated content, and efficient storage and querying of massive amounts of vector data.

A major challenge in RAG applications is how to effectively chunk and structure data into vector databases.

Documents often have a complex hierarchy, with important context spread across sections and subsections. Ensuring that this structure is maintained in the chunks of text used for generating embeddings is crucial for retaining the context and relevance in responses.

For example, a chunking process could entail using a tokenizer to truncate documents to a manageable size (e.g., 4,000 tokens). This strategy ensures that the data chunks are optimal for creating embeddings without losing essential information or overloading the retrieval system.

Metadata Utility

The metadata associated with vector embeddings enhances the retrieval process, allowing for more precise and contextually relevant results. This metadata can include information about the content source or other relevant details.

Hierarchy Preservation in Data

Preserving the hierarchy and context of data in documents is essential for accurate information retrieval. If the hierarchy is ignored, the application may miss crucial contextual cues leading to incomplete or incorrect responses.

Context in Embedding Models

The size of the text chunk used for generating embeddings is a critical factor. Too small chunks might miss context, and too large ones might be too broad, making the embedding less effective. Finding the right balance in chunk size is essential for effective retrieval.

Overlapping Chunks for Contextual Integrity

To ensure that no important information is missed, chunks can be made to overlap. This method ensures that the wider context is captured, which is important for the integrity of the information being processed.

Performance Trade-Offs in Chunk Size

The chunk size impacts the performance of the lookup process. While smaller chunks can increase the granularity of the search, they can also lead to an impractically large lookup table and may not be as contextually useful.

Summary

To enhance the retrieval process, the process of embedding the knowledge base into the vector database is critical. The process needs to be tailored specifically to the application.

Factors like the type of documents, the context of the queries, and the specific information needs of the application greatly influence how the retrieval process should be structured.

1. Retrieval isn't just about choosing the right embedding model. It may involve using embeddings in conjunction with metadata, rules, or heuristics to improve the retrieval accuracy.

2. Techniques like summarising documents before embedding them or using embeddings on summaries can refine the retrieval process.

3. Techniques like iterative summarisation, which condenses information into a progressively smaller set of sentences, can be applied for better summarization in RAG systems.