awesome-python/website/data/category_intros/machine-learning.md

Tabular data goes to scikit-learn, and boosted trees to LightGBM or CatBoost. Both fit in that Python machine learning library's Pipeline.

How to choose:

• Classification, regression, and clustering on tabular data: scikit-learn
• Boosted trees that train fast on large datasets: LightGBM
• Boosted trees on data with many categorical columns: CatBoost
• Bayesian networks and causal models: pgmpy
• Feature engineering on pandas dataframes: Feature-engine
• Boosted trees on a Spark, Dask, or Ray cluster: XGBoost
• Forecasting time series without training a model first: TimesFM

scikit-learn covers supervised and unsupervised learning, plus preprocessing, model selection, and evaluation. Deep learning is out of its scope. To pick a model, follow its Choosing the right estimator chart. Split your data into train and test sets before any preprocessing. Then put the preprocessing and the model in one Pipeline, so cross-validation and search never fit on the data they score. For gradient boosting without another dependency, its HistGradientBoostingClassifier and HistGradientBoostingRegressor handle missing values and categorical data with no preprocessing.

LightGBM aims at faster training and lower memory use. It grows trees leaf-wise, so num_leaves is the main parameter to tune: keep it below 2^(max_depth). To prevent overfitting, raise min_data_in_leaf. Instead of one-hot encoding, mark categorical columns with categorical_feature: it often performs better. With a validation set, use early stopping to find the number of boosting rounds.

CatBoost takes non-numeric features without preprocessing and gives good results with its default parameters. Its docs say not to one-hot encode during preprocessing: list the categorical columns in cat_features instead. Before tuning anything else, rule out underfitting and overfitting: set a large number of iterations, and turn on the overfitting detector and the use-best-model option. The learning rate is set from your data by default.

pgmpy does causal and probabilistic reasoning with graphical models, from learning a graph from data to running inference on the fitted model. scikit-learn leaves graphical models out, so use pgmpy for Bayesian networks. Every discovery algorithm follows one pattern: instantiate, fit, and read the result. Switching algorithms means changing only the class. Pass what you already know about the domain as required or forbidden edges with ExpertKnowledge. For queries, Variable Elimination is the default choice while the model is small enough for exact inference.

Feature-engine is for work where pandas and scikit-learn are your main tools. Each transformer takes the columns it changes in its variables argument, so it applies steps to selected groups of variables. Fit the transformers on the training set and transform both sets, as the quick start does. Put them in a scikit-learn Pipeline, and your whole feature engineering pipeline saves as one object.

XGBoost is built to be efficient, flexible, and portable. The same code runs on distributed environments, and its docs cover training on Dask, Spark, and Ray. Use its scikit-learn interface, like XGBClassifier, so it works with scikit-learn's tools such as cross-validation. For categorical columns, pass a dataframe with the category dtype and set enable_categorical. Its docs suggest you tune with cross-validation, then retrain with the best parameters and early stopping. To keep a model, save it with save_model, since a pickle is a memory snapshot meant only for checkpoints.

TimesFM is a forecasting model that Google Research pretrained on a large time-series corpus. It does well zero-shot on benchmarks from many domains, so you can forecast without training a model first. Install it with the extra for your backend, and load a checkpoint from the Hugging Face Hub. The code is Apache licensed, but the pretrained weights carry their own license, so check the model card before commercial or production use.

Every pick but TimesFM works with scikit-learn. XGBoost, LightGBM, and CatBoost ship scikit-learn estimators, Feature-engine's transformers go in a Pipeline, and pgmpy is scikit-learn compatible where possible. Independent benchmarks find no single winner among the three boosting libraries, so compare them on your own data in the same cross-validation.

Treat a model file you didn't make like code. scikit-learn's docs say to never load a pickle from an untrusted source, and point to skops.io or ONNX instead. XGBoost's security notes say the same about pickles.