awesome-python/website/data/category_intros/data-validation.md

Validate API input and config with Pydantic, the Python data validation library built on type hints. Use Pandera for dataframes, jsonschema for JSON Schema.

How to choose:

• API input, forms, and config: Pydantic
• Data checked against a JSON Schema document: jsonschema
• pandas, polars, or PySpark dataframes: Pandera

Pydantic builds the schema from your type hints and guarantees the types of the output, not the input: by default, a numeric string passed to an int field comes out as an int. Where a wrong type should raise an error instead, turn on strict mode per field or per model. Validate incoming JSON directly instead of parsing it into a dict first. To load config from environment variables, use Pydantic Settings.

jsonschema is an implementation of the JSON Schema specification, so one schema can work across different systems and platforms. When you validate many instances against one schema, create a validator for your schema's draft once and call its validate method. Use iter_errors() to report every error, not only the first. To enforce format keywords such as dates or emails, hook a format checker into the validator.

Pandera validates dataframe-like objects: define a schema once and use it on pandas, polars, PySpark, and other dataframe libraries. Write the schema as a DataFrameModel class, much like a Pydantic model, and add the check_types() decorator to validate at run time. To check an existing pipeline, put check_input() and check_output() on its functions. To see every failure in one run instead of only the first, validate with lazy=True.

Pick by the shape of your data. Running a dataframe through a Pydantic model row by row might not scale to larger datasets, so use Pandera there; a DataFrameModel can still be a field in a Pydantic model. Pydantic can generate a JSON Schema from any model for tools that read the format. jsonschema works the other way: it validates data against a JSON Schema document you already have.