awesome-python/website/data/category_intros/database.md

No server to run: a Python database library can live inside your process. Pick DuckDB when you run analytical SQL, LanceDB when you search vectors.

How to choose:

• Analytical SQL over DataFrames and Parquet files: DuckDB
• Vector search with the raw data stored beside its embeddings: LanceDB
• ClickHouse SQL in a notebook, the same queries your ClickHouse server runs: chDB
• A RAG prototype that embeds your text for you: Chroma
• Vector search, full-text search, and filters in one query: Zvec
• Tables whose columns run AI models on every insert: Pixeltable
• Python dicts in a JSON file, for a small app with one process: TinyDB

DuckDB is built for analytical queries, and it runs inside your Python process with no database server to install. duckdb.sql() runs on an in-memory database, while duckdb.connect() with a file name keeps your tables on disk. It queries pandas and Polars DataFrames and Arrow tables directly, and Parquet files by their file name. In a package others import, create your own connection objects instead of calling the module's functions, which share one global database.

chDB is an in-process SQL engine powered by ClickHouse, for ClickHouse SQL without a ClickHouse server. It speaks the full ClickHouse SQL dialect, so a query you write in a notebook runs unchanged on a ClickHouse server later. chdb.query() is stateless. For tables that last across queries, open a Session, and give it a directory name to keep them on disk.

Chroma's embedded mode is for prototyping and experimentation. It gets a RAG prototype running fast, since it embeds and indexes your documents for you with a default model that runs on your machine. Save data to a directory with PersistentClient. For production, its docs prefer a Chroma server that your app connects to as a client.

LanceDB is an embedded retrieval library that runs in your process: point it at a local directory, or at an object storage URI like s3://. It stores the raw data, metadata, and embeddings together, and its indexes live on disk.

Zvec is a vector database that runs entirely in-process, from notebooks and servers to edge devices. Define a schema with scalar fields and vectors, then create a collection from it. One query can combine vector similarity, full-text search, and filters.

Pixeltable is more than a vector store: it's the database, orchestration, and serving in one Python file. Model inference can go in a computed column, which runs on insert and on update. Each new row gets its model outputs without a pipeline you rerun. Put an embedding index on a column, and each insert keeps it current, with no separate vector database.

TinyDB is pure Python with no dependencies, and TinyDB('db.json') gives you a database that stores Python dicts in that JSON file. Its docs call it the wrong database when you need access from several processes or threads, indexes, ACID guarantees, or high performance.

Most of these databases expect one process to write at a time. DuckDB lets one process read and write, or several processes only read. A chDB data directory opens in one process at a time. Zvec shares a collection across processes in read-only mode, and Pixeltable keeps one writer process even with several API workers.