Unsupervised Dense Retrievers

Dense retrievers have shown superior results in domains with large training datasets but struggle in zero-shot scenarios or new applications without specific training data, where unsupervised term-frequency methods still excel.

This work focuses on exploring unsupervised dense retrievers trained through contrastive learning, demonstrating their potential across various retrieval settings, including multilingual retrieval.

Unsupervised Dense Information Retrieval with Contrastive LearningarXiv.org

Unsupervised dense retrievers trained through contrastive learning

Key Points

Traditional vs. Neural Network-Based Retrieval: Classical retrieval methods rely on term-frequency and are limited by the lexical gap. Neural network-based methods, such as dense retrievers, learn beyond lexical similarities but require extensive training data.

Unsupervised Dense Retrievers: The paper investigates the potential of training dense retrievers without supervision using contrastive learning. This approach aims to match the performance of BM25 in scenarios where training data is scarce or non-existent.

Performance on BEIR Benchmark: The unsupervised model trained through contrastive learning outperforms BM25 on a significant portion of the BEIR benchmark, especially in Recall@100 metric.

Few-Shot and Fine-Tuning: When pre-trained via contrastive learning and then fine-tuned with a small number of in-domain examples or on the large MSMARCO dataset, the model shows improvements across the BEIR benchmark.

Multilingual Retrieval: The approach leads to robust unsupervised performance in multilingual settings and strong cross-lingual transfer capabilities, even for languages with limited resources or different scripts.

Conclusion

This work posits contrastive learning as a viable method for training unsupervised dense retrievers, showcasing strong performance across a variety of retrieval tasks and languages.

It addresses the limitations of existing neural and term-frequency-based methods, particularly in zero-shot and multilingual retrieval scenarios, marking a significant step towards more adaptable and universally applicable information retrieval systems.

The method described aims to train a dense retriever without supervision using contrastive learning. Here's a breakdown of the approach.