SQLite is Enough. Lexical, Semantic, and Hybrid Search with scrydb
Timo Breuer
Abstract
This work introduces scrydb, a Python library that enables lexical, semantic, and hybrid search within SQLite. For lexical search, scrydb leverages SQLite's full-text search extension FTS5. Semantic search builds on sqlite-vec, a SQLite extension for vector search. Furthermore, the library allows users to rerank and fuse retrieval results to combine both lexical and semantic approaches, providing a lightweight solution for downstream tasks in information retrieval (IR) or agentic search. We evaluate scrydb on various IR benchmark datasets and demonstrate its effectiveness in text retrieval based on keyword matching, semantic similarity, and rank fusion. In addition, we provide insights into query latency and the trade-off between efficiency and effectiveness. scrydb is available under the MIT license.
Create a lesson
Related papers
SURF: Subtractive Updates for Recommender Forgetting
Filippo Betello, Antonio Purificato, Nicola Tonellotto et al.
Exploring LLMs and RAG for Plausible and Explainable Material Prediction of Vehicle Components
Frederik Wagner, Annerose Eichel, Sabine Schulte im Walde
One-Step Retrieval Framework for Real-Time Sponsored Search Ads Using Hierarchical Text Representations
Tongtong Liu, Renyu Zhang, Jiayu Ding et al.
Quanta: A Self-Contained Python Library for Hybrid Retrieval over Quantised Embeddings, Lexical Indexes, and Knowledge Graphs
Ioannis E. Livieris
Single-Token Expected-Value Scoring for Cold-Start Candidate Ranking
Qihang Wang, Jinwei Tan, Mengyuan Shi et al.
PageRecall: Measuring Page Selection in Literature-Grounded Question Answering
Aaditya Chauhan