diff --git a/README.md b/README.md index e2ae0b7a..92eeab70 100644 --- a/README.md +++ b/README.md @@ -474,7 +474,6 @@ _In-process databases usable directly from Python._ - [duckdb](https://github.com/duckdb/duckdb) - An in-process SQL OLAP database management system; optimized for analytics and fast queries, similar to SQLite but for analytical workloads. - [chdb](https://github.com/chdb-io/chdb) - In-process OLAP SQL engine with the full ClickHouse dialect, zero-copy pandas/Arrow interop, and federation to remote ClickHouse clusters via `remoteSecure()`. - Vector - - [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors. - [lancedb](https://github.com/lancedb/lancedb) - A developer-friendly embedded retrieval database for multimodal AI. - [chromadb](https://github.com/chroma-core/chroma) - An open-source embedding database for building AI applications with embeddings and semantic search. - [zvec](https://github.com/alibaba/zvec) - A lightweight, in-process vector database that embeds directly into applications. @@ -495,10 +494,13 @@ _Libraries for caching data._ _Libraries and software for indexing and performing search queries on data._ -- [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/). -- [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch). -- [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine. -- [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. +- Full-Text Search + - [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/). + - [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch). + - [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine. + - [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django. +- Vector Search + - [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors. ### Serialization diff --git a/website/data/category_intros/database.md b/website/data/category_intros/database.md index 4162af33..408271eb 100644 --- a/website/data/category_intros/database.md +++ b/website/data/category_intros/database.md @@ -5,7 +5,6 @@ How to choose: - Analytical SQL over Parquet, CSV, or DataFrames: DuckDB - Vector search in a prototype or notebook: Chroma - ClickHouse's SQL dialect, or pandas code that runs on ClickHouse: chDB -- Similarity search over vectors you hold in NumPy arrays: Faiss - Vectors stored with their raw images and text, on disk: LanceDB - Vector search embedded in an app or an edge device: Zvec - A small JSON document store for one script: TinyDB @@ -16,8 +15,6 @@ chDB runs ClickHouse inside your process, so you get its SQL dialect without a C Chroma's [in-memory client](https://docs.trychroma.com/docs/run-chroma/clients) is the quick way to try it in a notebook, and `PersistentClient` keeps the data on disk. Add text documents, and Chroma [embeds and indexes them for you](https://docs.trychroma.com/docs/overview/getting-started), so you can query with text too. -Faiss is a similarity search library, not a database. It [keeps no metadata with the vectors](https://arxiv.org/html/2401.08281#S6.SS1), only integer IDs, so you store your documents elsewhere and look them up by ID. Its [tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started) builds an index for the vector dimension, adds a float32 NumPy array, and calls `search()` for the nearest neighbors. `IndexFlatL2` returns exact results. To trade some accuracy for speed or memory, [pick an index by RAM and dataset size](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index). - LanceDB starts as an [embedded library pointed at a local directory](https://docs.lancedb.com/quickstart). Its indexes live on disk, and a table [stores the raw data, metadata, and embeddings together](https://docs.lancedb.com/faq/faq-oss), which suits images and other multimodal data. Its [embedding function registry](https://docs.lancedb.com/embedding) generates vectors as you insert rows, and Python code can then query with text. Zvec runs entirely in-process, with no server or daemon, and its docs pitch it [from prototypes to embedded apps and edge devices](https://zvec.org/en/docs/db/). Its [quickstart](https://zvec.org/en/docs/db/quickstart/) defines a schema of scalar and vector fields first, then combines vector search with filters on those scalar fields. diff --git a/website/data/category_intros/search.md b/website/data/category_intros/search.md index 9a6e3a39..d1c79727 100644 --- a/website/data/category_intros/search.md +++ b/website/data/category_intros/search.md @@ -1,4 +1,4 @@ -The engine comes first, then its official Python search library. Meilisearch keeps a search bar simple, while analytics call for Elasticsearch or OpenSearch. +For document search, pick an engine and its official Python search library. Faiss searches vectors directly in your process. How to choose: @@ -6,6 +6,7 @@ How to choose: - Log analytics and aggregations beyond search: elasticsearch - The same under Apache instead of the AGPL, or on Amazon OpenSearch Service: opensearch-py - Search over Django models, with a backend you can swap later: django-haystack +- Similarity search over vectors you hold in NumPy arrays: Faiss Meilisearch is [an open-source search engine](https://github.com/meilisearch/meilisearch-python), and meilisearch is its Python client. Its docs call it [a perfect choice for a typo-tolerant search bar](https://www.meilisearch.com/docs/resources/comparisons/alternatives), built for instant search aimed at end users. Meilisearch [queues writes and processes them asynchronously](https://www.meilisearch.com/docs/capabilities/indexing/tasks_and_batches/async_operations), so after `add_documents()`, the Python quick start [waits for indexing to complete](https://www.meilisearch.com/docs/getting_started/sdks/python) with `client.wait_for_task(task.task_uid)`. @@ -15,4 +16,6 @@ OpenSearch is [a fork of Elasticsearch, and all of its software is Apache-licens django-haystack gives Django [one API over pluggable search backends](https://django-haystack.readthedocs.io/en/latest/), such as Elasticsearch, so you can switch engines without rewriting your search code. Its FAQ [advises against it](https://django-haystack.readthedocs.io/en/latest/faq.html#when-should-i-not-be-using-haystack) for data that isn't in Django models and for ultra-high volume, since the abstraction costs performance and some engine features. [Create a `SearchIndex` for each model](https://django-haystack.readthedocs.io/en/latest/tutorial.html#creating-searchindexes) you index, in a `search_indexes.py` file in its app. Load your data with [`./manage.py rebuild_index`](https://django-haystack.readthedocs.io/en/latest/tutorial.html#reindex), then keep the index current with a cron job running `update_index`. +Faiss is a similarity search library, not a database. It [keeps no metadata with the vectors](https://arxiv.org/html/2401.08281#S6.SS1), only integer IDs, so you store your documents elsewhere and look them up by ID. Its [tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started) builds an index for the vector dimension, adds a float32 NumPy array, and calls `search()` for the nearest neighbors. `IndexFlatL2` returns exact results. To trade some accuracy for speed or memory, [pick an index by RAM and dataset size](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index). + Keep your data in your database, and send the search engine a copy of what people search for. Meilisearch [wasn't designed to be your main data container](https://www.meilisearch.com/docs/capabilities/indexing/advanced/indexing_best_practices#do-not-use-meilisearch-as-your-main-database), and django-haystack's docs call the database [authoritative and the search index non-authoritative](https://django-haystack.readthedocs.io/en/latest/signal_processors.html).