Vector Databases

Vector databases are a specialised type of database designed for handling vector embeddings, which are numerical representations of various data objects.

Traditional databases store data in tabular forms and index it by assigning values, focusing on exact matches. Vector databases, in contrast, are designed for embeddings and similarity-based searches.

They offer capabilities like CRUD operations, metadata filtering, and horizontal scaling.

Definition and Functionality of Vector Databases

Storage of Vector Embedding

Vector databases store information as vectors, known as vector embeddings. These embeddings represent data objects numerically and are generated by AI models, including large language models.

Handling of Unstructured and Semi-Structured Data

They are particularly adept at managing massive datasets of unstructured data (like images and text) and semi-structured data (like sensor data).

Capabilities Beyond Vector Search Libraries

Unlike vector search libraries or indexes, vector databases offer a comprehensive data management solution, including metadata storage, scalability, dynamic data handling, backups, and security features.

High-Dimensional Vectors

The data in these databases is organized through high-dimensional vectors, where each dimension represents a specific characteristic of the data object.

Vector Embeddings

Numerical Representation

Vector embeddings are numerical representations of various subjects like words, images, or data pieces.

Distance Measurement for Similarity

The distance between vector embeddings (using various mathematical measures) allows the database to determine the similarity between vectors, aiding in pattern recognition and AI's understanding of relationships.

Working Mechanism

1. Indexing: Vectors are indexed using techniques like hashing, quantization, or graph-based methods, facilitating efficient searches.

2. Querying: Queries are processed by comparing indexed vectors to the query vector using similarity measures like cosine similarity, Euclidean distance, and dot product similarity.

3. Post-processing: After initial querying, there may be additional filtering or re-ranking using different similarity measures, focusing on metadata.

Importance of Vector Databases

Specialisation in Unstructured Data Management

They are crucial for managing unstructured data, providing capabilities like indexing, distance metrics, and similarity searches.

Enabling Advanced AI and ML Applications

Vector databases are fundamental in AI learning applications, enabling efficient processing of complex data types.

How Vector Databases Differ from Vector Indices

While standalone vector indices like Facebook AI Similarity Search (FAISS) improve search and retrieval, they lack comprehensive data management capabilities.

Vector databases, on the other hand, are purpose-built for managing vector embeddings. They offer a suite of features that standalone indices simply don't, including:

• Data Management: Easier and more efficient handling of vector data.

• Metadata Storage and Filtering: Enhanced query capabilities through metadata.

• Scalability: Optimised for growing data volumes and user demands.

• Real-time Updates: Ability to dynamically update data.

• Backups and Collections: Routine operations for data security and efficiency.

• Ecosystem Integration: Seamless compatibility with other AI tools and data processing ecosystems.

• Data Security and Access Control: Essential for protecting sensitive information.

Embeddings in Vector Databases

Embeddings are vectors representing semantic relationships. They place related objects close in the embedding space. They allow for the comparison and retrieval of similar items based on their conceptual similarity.

Preprocessing Embeddings

• Normalisation: Involves scaling data to unit length, which eliminates the impact of vector scale and allows for consistent high-dimensional data handling. It enables using dot product similarity, which is faster than cosine similarity.

• Standardisation: This process shifts and scales data to have zero mean and unit variance, imparting Gaussian distribution properties to vectors and ensuring equal contribution in distance calculations.

• Importance: Normalization is crucial for meaningful dot product results, especially when using various embedding generation tools like OpenAI, PaLM, or Simsce.

Defining a Vector Field

• Vector Type: Vector fields are defined with a VECTOR type, with fixed dimensionality determined by the embedding model.

• Embedding Model: The choice of embedding model is crucial for creating a structured embedding space where related objects are near each other.

Popular Embedding Models

• Various models like 'bge-large-en-v1.5', 'distiluse-base-multilingual-cased-v2', 'glove.6B.300d', etc., are available, each with specific dimensions and sources like Hugging Face or OpenAI.

Vector Search Process

• Involves creating a collection of embeddings, generating an embedding for new content, and conducting a similarity search to find similar existing content.

Best Practices for Vector Search

• Metadata Storage: Store relevant metadata alongside vectors for more contextual searches.

• Model Selection: Choose an embedding model based on your data type (text, images, audio, etc.) and query requirements.

Limitations of Vector Search

• Readability: Vector embeddings are not human-readable.

• Direct Retrieval: Not suited for direct data retrieval from a table.

• Completeness: There is a risk of incomplete or incorrect results due to model limitations.

Common Use Cases

• Retrieval-Augmented Generation (RAG): Enhances LLM accuracy by incorporating relevant content into the LLM’s context window.

• AI Agents: Capable of performing actions like Google searches and storing results and embeddings in a vector database for a persistent memory build-up.Overall Implications

Vector databases revolutionise how we handle and retrieve data in AI and machine learning by focusing on the similarity of content rather than traditional relational data structures.

They are instrumental in enhancing the capabilities of large language models (LLMs), recommendation systems, and other AI-driven applications.

However, their effectiveness heavily relies on the proper selection and preprocessing of embeddings, as well as understanding their limitations in comparison to traditional databases.