mirror of
https://github.com/vinta/awesome-python.git
synced 2026-11-08 02:15:12 -06:00
docs: move Faiss into vector search
Faiss indexes vectors and integer IDs rather than storing documents and metadata, so its Database placement contradicted the category definition. Co-Authored-By: Codex <noreply@openai.com>
This commit is contained in:
@@ -474,7 +474,6 @@ _In-process databases usable directly from Python._
|
||||
- [duckdb](https://github.com/duckdb/duckdb) - An in-process SQL OLAP database management system; optimized for analytics and fast queries, similar to SQLite but for analytical workloads.
|
||||
- [chdb](https://github.com/chdb-io/chdb) - In-process OLAP SQL engine with the full ClickHouse dialect, zero-copy pandas/Arrow interop, and federation to remote ClickHouse clusters via `remoteSecure()`.
|
||||
- Vector
|
||||
- [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors.
|
||||
- [lancedb](https://github.com/lancedb/lancedb) - A developer-friendly embedded retrieval database for multimodal AI.
|
||||
- [chromadb](https://github.com/chroma-core/chroma) - An open-source embedding database for building AI applications with embeddings and semantic search.
|
||||
- [zvec](https://github.com/alibaba/zvec) - A lightweight, in-process vector database that embeds directly into applications.
|
||||
@@ -495,10 +494,13 @@ _Libraries for caching data._
|
||||
|
||||
_Libraries and software for indexing and performing search queries on data._
|
||||
|
||||
- [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/).
|
||||
- [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch).
|
||||
- [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine.
|
||||
- [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django.
|
||||
- Full-Text Search
|
||||
- [opensearch-py](https://github.com/opensearch-project/opensearch-py) - The official low-level Python client for [OpenSearch](https://opensearch.org/).
|
||||
- [elasticsearch](https://github.com/elastic/elasticsearch-py) - The official low-level Python client for [Elasticsearch](https://www.elastic.co/elasticsearch).
|
||||
- [meilisearch](https://github.com/meilisearch/meilisearch-python) - The official Python client for the [Meilisearch](https://www.meilisearch.com/) search engine.
|
||||
- [django-haystack](https://github.com/django-haystack/django-haystack) - Modular search for Django.
|
||||
- Vector Search
|
||||
- [faiss-cpu](https://github.com/facebookresearch/faiss) - A library for efficient similarity search and clustering of dense vectors.
|
||||
|
||||
### Serialization
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ How to choose:
|
||||
- Analytical SQL over Parquet, CSV, or DataFrames: DuckDB
|
||||
- Vector search in a prototype or notebook: Chroma
|
||||
- ClickHouse's SQL dialect, or pandas code that runs on ClickHouse: chDB
|
||||
- Similarity search over vectors you hold in NumPy arrays: Faiss
|
||||
- Vectors stored with their raw images and text, on disk: LanceDB
|
||||
- Vector search embedded in an app or an edge device: Zvec
|
||||
- A small JSON document store for one script: TinyDB
|
||||
@@ -16,8 +15,6 @@ chDB runs ClickHouse inside your process, so you get its SQL dialect without a C
|
||||
|
||||
Chroma's [in-memory client](https://docs.trychroma.com/docs/run-chroma/clients) is the quick way to try it in a notebook, and `PersistentClient` keeps the data on disk. Add text documents, and Chroma [embeds and indexes them for you](https://docs.trychroma.com/docs/overview/getting-started), so you can query with text too.
|
||||
|
||||
Faiss is a similarity search library, not a database. It [keeps no metadata with the vectors](https://arxiv.org/html/2401.08281#S6.SS1), only integer IDs, so you store your documents elsewhere and look them up by ID. Its [tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started) builds an index for the vector dimension, adds a float32 NumPy array, and calls `search()` for the nearest neighbors. `IndexFlatL2` returns exact results. To trade some accuracy for speed or memory, [pick an index by RAM and dataset size](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index).
|
||||
|
||||
LanceDB starts as an [embedded library pointed at a local directory](https://docs.lancedb.com/quickstart). Its indexes live on disk, and a table [stores the raw data, metadata, and embeddings together](https://docs.lancedb.com/faq/faq-oss), which suits images and other multimodal data. Its [embedding function registry](https://docs.lancedb.com/embedding) generates vectors as you insert rows, and Python code can then query with text.
|
||||
|
||||
Zvec runs entirely in-process, with no server or daemon, and its docs pitch it [from prototypes to embedded apps and edge devices](https://zvec.org/en/docs/db/). Its [quickstart](https://zvec.org/en/docs/db/quickstart/) defines a schema of scalar and vector fields first, then combines vector search with filters on those scalar fields.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
The engine comes first, then its official Python search library. Meilisearch keeps a search bar simple, while analytics call for Elasticsearch or OpenSearch.
|
||||
For document search, pick an engine and its official Python search library. Faiss searches vectors directly in your process.
|
||||
|
||||
How to choose:
|
||||
|
||||
@@ -6,6 +6,7 @@ How to choose:
|
||||
- Log analytics and aggregations beyond search: elasticsearch
|
||||
- The same under Apache instead of the AGPL, or on Amazon OpenSearch Service: opensearch-py
|
||||
- Search over Django models, with a backend you can swap later: django-haystack
|
||||
- Similarity search over vectors you hold in NumPy arrays: Faiss
|
||||
|
||||
Meilisearch is [an open-source search engine](https://github.com/meilisearch/meilisearch-python), and meilisearch is its Python client. Its docs call it [a perfect choice for a typo-tolerant search bar](https://www.meilisearch.com/docs/resources/comparisons/alternatives), built for instant search aimed at end users. Meilisearch [queues writes and processes them asynchronously](https://www.meilisearch.com/docs/capabilities/indexing/tasks_and_batches/async_operations), so after `add_documents()`, the Python quick start [waits for indexing to complete](https://www.meilisearch.com/docs/getting_started/sdks/python) with `client.wait_for_task(task.task_uid)`.
|
||||
|
||||
@@ -15,4 +16,6 @@ OpenSearch is [a fork of Elasticsearch, and all of its software is Apache-licens
|
||||
|
||||
django-haystack gives Django [one API over pluggable search backends](https://django-haystack.readthedocs.io/en/latest/), such as Elasticsearch, so you can switch engines without rewriting your search code. Its FAQ [advises against it](https://django-haystack.readthedocs.io/en/latest/faq.html#when-should-i-not-be-using-haystack) for data that isn't in Django models and for ultra-high volume, since the abstraction costs performance and some engine features. [Create a `SearchIndex` for each model](https://django-haystack.readthedocs.io/en/latest/tutorial.html#creating-searchindexes) you index, in a `search_indexes.py` file in its app. Load your data with [`./manage.py rebuild_index`](https://django-haystack.readthedocs.io/en/latest/tutorial.html#reindex), then keep the index current with a cron job running `update_index`.
|
||||
|
||||
Faiss is a similarity search library, not a database. It [keeps no metadata with the vectors](https://arxiv.org/html/2401.08281#S6.SS1), only integer IDs, so you store your documents elsewhere and look them up by ID. Its [tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started) builds an index for the vector dimension, adds a float32 NumPy array, and calls `search()` for the nearest neighbors. `IndexFlatL2` returns exact results. To trade some accuracy for speed or memory, [pick an index by RAM and dataset size](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index).
|
||||
|
||||
Keep your data in your database, and send the search engine a copy of what people search for. Meilisearch [wasn't designed to be your main data container](https://www.meilisearch.com/docs/capabilities/indexing/advanced/indexing_best_practices#do-not-use-meilisearch-as-your-main-database), and django-haystack's docs call the database [authoritative and the search index non-authoritative](https://django-haystack.readthedocs.io/en/latest/signal_processors.html).
|
||||
|
||||
Reference in New Issue
Block a user