# Search

Search engines have traditionally operated through a two-phase process involving <mark style="color:yellow;">retrieval and ranking</mark>.&#x20;

Initially, they relied on lexical retrieval methods, like the inverted index and bag-of-words model, for identifying documents containing specific keywords related to a user's query.&#x20;

These documents were then ranked according to their relevance, often using algorithms such as BM25, based on keyword frequency and document length, among other factors.

The integration of Artificial Intelligence (AI), especially with the advent of Large Language Models (LLMs) like BERT (Bidirectional Encoder Representations from Transformers), has significantly enhanced the capability of search engines.&#x20;

AI has introduced a profound shift in how search engines interpret and process queries, moving from simple keyword matching to understanding the semantic meaning behind user queries. This is achieved through several key innovations:

<mark style="color:green;">**Dense Retrieval**</mark>

AI enables the conversion of text into dense vectors that encapsulate semantic information, allowing for more nuanced matching beyond mere keywords. This is facilitated by bi-encoders that generate semantically rich embeddings for texts, improving the efficiency and relevance of the retrieval phase.

<mark style="color:green;">**Advanced Ranking**</mark>

Cross-encoders, leveraging deep learning, assess the similarity between queries and documents at a deeper level, considering context and the interplay between query and document words. This results in a more precise ranking of documents, ensuring that the most relevant information is presented to the user.

<mark style="color:green;">**Hybrid Search**</mark>

Modern AI-powered search engines employ a hybrid approach that combines traditional lexical search with dense retrieval. This ensures a broad yet accurate initial retrieval of documents, which are then finely ranked using sophisticated AI models.

<mark style="color:green;">**Contextual Understanding**</mark>

BERT and similar models have been instrumental in enhancing search engines' ability to comprehend the context of queries. By pretraining on large datasets with tasks designed to capture linguistic nuances, these models can generate contextually informed embeddings that significantly improve both retrieval and ranking processes.

The emergence of AI-enhanced search engines promises a more intuitive search experience, where the engines not only grasp the literal text of queries but also their underlying intent.&#x20;

This shift towards understanding context and semantics means search results can become more relevant, accurate, and personalised, reducing reliance on exact keyword matches.&#x20;

Consequently, this evolution is set to redefine how users interact with information online, making search more efficient and aligned with user intentions.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://training.continuumlabs.ai/disruption/search.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.