3.7 KiB
Inside your process, a Python database library runs without a server. DuckDB handles analytical SQL there; to search vectors, start with Chroma.
How to choose:
- Analytical SQL over Parquet, CSV, or DataFrames: DuckDB
- Vector search in a prototype or notebook: Chroma
- ClickHouse's SQL dialect, or pandas code that runs on ClickHouse: chDB
- Similarity search over vectors you hold in NumPy arrays: Faiss
- Vectors stored with their raw images and text, on disk: LanceDB
- Vector search embedded in an app or an edge device: Zvec
- A small JSON document store for one script: TinyDB
Start DuckDB with duckdb.sql(), which runs on an in-memory database kept inside the module. It queries Parquet, CSV, and JSON files by name, and pandas DataFrames, Polars DataFrames, and Arrow tables by variable name, with no import step. When the data should stay, pass a file name to duckdb.connect().
chDB runs ClickHouse inside your process, so you get its SQL dialect without a ClickHouse server. If you come from pandas, start with the DataStore API: change the import, and your pandas code works unchanged. For SQL, use the connection-based API, and query DataFrames with the Python() table engine.
Chroma's in-memory client is the quick way to try it in a notebook, and PersistentClient keeps the data on disk. Add text documents, and Chroma embeds and indexes them for you, so you can query with text too.
Faiss is a similarity search library, not a database. It keeps no metadata with the vectors, only integer IDs, so you store your documents elsewhere and look them up by ID. Its tutorial builds an index for the vector dimension, adds a float32 NumPy array, and calls search() for the nearest neighbors. IndexFlatL2 returns exact results. To trade some accuracy for speed or memory, pick an index by RAM and dataset size.
LanceDB starts as an embedded library pointed at a local directory. Its indexes live on disk, and a table stores the raw data, metadata, and embeddings together, which suits images and other multimodal data. Its embedding function registry generates vectors as you insert rows, and Python code can then query with text.
Zvec runs entirely in-process, with no server or daemon, and its docs pitch it from prototypes to embedded apps and edge devices. Its quickstart defines a schema of scalar and vector fields first, then combines vector search with filters on those scalar fields.
TinyDB is pure Python with no dependencies, and stores Python dicts in a JSON file: open it with TinyDB('db.json') and search with Query(). Its own docs say to pick something else when you need access from several processes or threads, indexes, or ACID guarantees.