Merge branch 'audit/websocket-and-pgmpy'

README.md

@@ -213,7 +213,6 @@ _Libraries for machine learning. Also see [awesome-machine-learning](https://git
 - General
   - [scikit-learn](https://github.com/scikit-learn/scikit-learn) - The most popular Python library for machine learning with extensive documentation and community support.
   - [imbalanced-learn](https://github.com/scikit-learn-contrib/imbalanced-learn) - Resampling techniques for imbalanced datasets, compatible with scikit-learn.
-  - [pgmpy](https://github.com/pgmpy/pgmpy) - A Python library for causal and probabilistic reasoning with graphical models.
   - [feature-engine](https://github.com/feature-engine/feature_engine) - sklearn-compatible API with the widest toolset for feature engineering and selection.
 - Gradient Boosting
   - [xgboost](https://github.com/dmlc/xgboost) - A scalable, portable, and distributed gradient boosting library.
@@ -323,11 +322,12 @@ _ASGI and WSGI compatible web servers._
 
 _Libraries for working with WebSocket._
 
-- General
+- Servers and Clients
   - [websockets](https://github.com/python-websockets/websockets) - A library for building WebSocket servers and clients with a focus on correctness and simplicity.
+  - [autobahn-python](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html).
+- Framework Integrations
   - [channels](https://github.com/django/channels) - Brings WebSocket, long-poll HTTP, and other async support to Django.
   - [flask-socketio](https://github.com/miguelgrinberg/Flask-SocketIO) - Socket.IO integration for Flask applications.
-  - [autobahn-python](https://github.com/crossbario/autobahn-python) - WebSocket & WAMP for Python on Twisted and [asyncio](https://docs.python.org/3/library/asyncio.html).
 - WebRTC
   - [aiortc](https://github.com/aiortc/aiortc) - A WebRTC and ORTC implementation for asyncio, with audio, video, and data channels.
 
@@ -598,6 +598,7 @@ _Libraries for scientific computing. Also see [Python-for-Scientists](https://gi
   - [sympy](https://github.com/sympy/sympy) - A Python library for symbolic mathematics.
 - Statistics
   - [statsmodels](https://github.com/statsmodels/statsmodels) - Statistical modeling and econometrics in Python.
+  - [pgmpy](https://github.com/pgmpy/pgmpy) - A Python library for causal and probabilistic reasoning with graphical models.
 - Biology and Chemistry
   - [rdkit](https://github.com/rdkit/rdkit) - Cheminformatics and machine learning software.
   - [biopython](https://github.com/biopython/biopython) - Biopython is a set of freely available tools for biological computation.

website/data/category_intros/machine-learning.md

@@ -4,7 +4,6 @@ How to choose:
 
 - Classification, regression, and clustering on tabular data: scikit-learn
 - Boosted trees with no extra dependency: scikit-learn
-- Causal graphs and Bayesian networks: pgmpy
 - Feature engineering that keeps your DataFrame's column names: Feature-engine
 - Boosted trees trained across a cluster: XGBoost
 - Boosted trees on large data, fast and light on memory: LightGBM
@@ -14,11 +13,10 @@ How to choose:
 - Forecasting next to classification and other time-series tasks: sktime
 - Forecasts with no model to train: TimesFM
 - Tuning a model's hyperparameters: Optuna
+- Runs, parameters, and metrics of every training job, tracked: MLflow
 
 scikit-learn does [supervised and unsupervised learning](https://scikit-learn.org/stable/getting_started.html), plus the tools around it: preprocessing, model selection, and evaluation. Every estimator learns with `fit`. Put your preprocessing and model in one Pipeline, and run cross-validation and searches on that, since the pipeline [keeps test data out of training](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage). You may not need a separate boosting library: its histogram-based gradient boosting is [inspired by LightGBM](https://scikit-learn.org/stable/modules/ensemble.html) and supports missing values and categorical data natively. Deep learning is another matter: it [doesn't fit scikit-learn's design constraints](https://scikit-learn.org/stable/faq.html#why-is-there-no-support-for-deep-or-reinforcement-learning-will-there-be-such-support-in-the-future), so it's out of scope.
 
-pgmpy does [causal and probabilistic reasoning with graphical models](https://pgmpy.org/). Its [quickstart](https://pgmpy.org/started/quickstart.html) follows the tasks: learn a graph's structure from data, fit the distributions for a graph you know, then query the model for probabilities or causal effects.
-
 Feature-engine is [designed to work with dataframes](https://feature-engine.trainindata.com/en/latest/): a DataFrame goes in, and the same DataFrame comes out, with no column order or name changes. Instead of wrapping transformers in a ColumnTransformer, you tell each one which variables to transform. Its transformers [work just like any scikit-learn transformer](https://feature-engine.trainindata.com/en/latest/quickstart/index.html), so they fit in a Pipeline.
 
 XGBoost is [an optimized distributed gradient boosting library](https://xgboost.readthedocs.io/en/stable/). Its Python package has [a native interface, a scikit-learn interface, and a Dask interface](https://xgboost.readthedocs.io/en/stable/python/python_intro.html).
@@ -37,4 +35,6 @@ TimesFM is [a pretrained foundation model from Google Research](https://github.c
 
 Optuna is [an automatic hyperparameter optimization framework, particularly designed for machine learning](https://optuna.readthedocs.io/en/latest/). Write an `objective` function that takes a `trial`, picks each hyperparameter with methods like `suggest_float`, trains the model, and returns its score. Then [create a study and call `optimize`](https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/001_first.html) with the number of trials. The search space is [plain Python, with conditionals and loops](https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/002_configurations.html), so you can tune an SVM's `C` only in trials that picked an SVM. To stop unpromising trials early, [call `report` after each training step and raise `optuna.TrialPruned()` when `should_prune()` returns true](https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/003_efficient_optimization_algorithms.html#activating-pruners).
 
+MLflow groups [runs and models into experiments](https://mlflow.org/docs/latest/ml/tracking/), one per task. Call [`mlflow.autolog()`](https://mlflow.org/docs/latest/ml/tracking/autolog/) before your training code, and it logs metrics, parameters, and models from libraries like scikit-learn, XGBoost, and PyTorch with no explicit log statements. Log anything else with `mlflow.log_param()` and `mlflow.log_metric()`, then run `mlflow server` to compare runs in its UI.
+
 To evaluate a model on time-series data, test it on observations from after its training data. scikit-learn's [TimeSeriesSplit](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split) does that in cross-validation. For more, see [awesome-machine-learning](https://github.com/josephmisiti/awesome-machine-learning#python).

website/data/category_intros/science.md

@@ -6,7 +6,7 @@ How to choose:
 - Optimization, integration, interpolation, and linear algebra: SciPy
 - Numeric loops that won't vectorize: Numba
 - Exact, symbolic math: SymPy
-- Regression, time series, and hypothesis tests: statsmodels
+- Regression, time series, and hypothesis tests: statsmodels; causal graphs and Bayesian networks: pgmpy
 - Biological sequences and file formats: Biopython; molecules: RDKit
 - Physical units: Pint; astronomy: Astropy; seismology: ObsPy
 - Bayesian models: PyMC
@@ -25,6 +25,8 @@ SymPy does symbolic math: expressions stay [exact, not approximate](https://docs
 
 statsmodels [estimates statistical models, runs hypothesis tests, and explores data](https://www.statsmodels.org/stable/index.html), and [most of its results are verified](https://www.statsmodels.org/stable/about.html#testing) against another statistical package. SciPy's stats docs [send regression, linear models, and time series analysis to statsmodels](https://docs.scipy.org/doc/scipy/reference/stats.html), while `scipy.stats` keeps the probability distributions and statistical tests. Write your model as an [R-style formula](https://www.statsmodels.org/stable/example_formulas.html) on a pandas DataFrame, like `smf.ols("y ~ x", data=df).fit()`, and read the fit with `summary()`.
 
+pgmpy does [causal and probabilistic reasoning with graphical models](https://pgmpy.org/). Its [quickstart](https://pgmpy.org/started/quickstart.html) follows the tasks: learn a graph's structure from data, fit the distributions for a graph you know, then query the model for probabilities or causal effects.
+
 Biopython [parses bioinformatics file formats](https://biopython.org/docs/latest/Tutorial/chapter_introduction.html) like FASTA and GenBank, queries online services like NCBI, and gives you a standard sequence class. Read sequence files with `Bio.SeqIO.parse()`, or [`Bio.SeqIO.read()` when a file holds one record](https://biopython.org/docs/latest/Tutorial/chapter_seqio.html#parsing-or-reading-sequences). When you query NCBI through `Bio.Entrez`, [pass your email](https://biopython.org/docs/latest/Tutorial/chapter_entrez.html#entrez-guidelines) so NCBI can contact you if there's a problem. RDKit is a [cheminformatics toolkit with its core in C++](https://www.rdkit.org/docs/Overview.html), for 2D and 3D molecular operations and descriptors for machine learning. To compare molecules, [create a fingerprint generator](https://www.rdkit.org/docs/GettingStartedInPython.html#fingerprinting-and-molecular-similarity) for the fingerprint type you want, the docs' most consistent way to get fingerprints.
 
 Pint handles [physical quantities](https://pint.readthedocs.io/en/stable/getting/overview.html): a value times a unit, in any numeric type. In a package, [create one `UnitRegistry` in a single place](https://pint.readthedocs.io/en/stable/getting/pint-in-your-projects.html#having-a-shared-registry) and import it everywhere, since quantities from different registries don't mix. Astropy is [a common core package for astronomy](https://www.astropy.org/), with affiliated packages around it. [Import the subpackage you need](https://docs.astropy.org/en/stable/importing_astropy.html#importing-astropy-and-sub-packages), like `from astropy import units as u`, and never import with `*`. ObsPy is [a framework for processing seismological data](https://docs.obspy.org/): `read()` loads [SAC, MiniSEED, and other formats into a Stream](https://docs.obspy.org/tutorial/code_snippets/reading_seismograms.html) of Traces, and its [FDSN client](https://docs.obspy.org/packages/obspy.clients.fdsn.html#basic-fdsn-client-usage) fetches data from data centers.

website/data/category_intros/websocket.md

@@ -7,6 +7,7 @@ How to choose:
 - Plain WebSocket servers and clients: websockets
 - A few real-time features next to a Django project: websockets as a separate server
 - RPC and pub/sub over WAMP, or a Twisted app: Autobahn|Python
+- Real-time audio, video, and data channels over WebRTC: aiortc
 
 Channels [extends Django beyond HTTP](https://channels.readthedocs.io/en/latest/) to handle WebSockets, and it [integrates with Django's auth and sessions](https://channels.readthedocs.io/en/latest/introduction.html). Write [sync consumers by default](https://channels.readthedocs.io/en/latest/topics/consumers.html#basic-layout). Switch to async ones only when async handling helps and every library you call is async-native. Serve everything with Daphne, or [keep HTTP on your WSGI server](https://channels.readthedocs.io/en/latest/deploying.html#http-and-websocket) and send only WebSockets to Daphne.
 
@@ -16,6 +17,8 @@ websockets runs on asyncio by default, which is [ideal for servers with many con
 
 Autobahn|Python implements [both WebSocket and WAMP, on Twisted or asyncio](https://autobahn.readthedocs.io/en/latest/). WAMP adds [RPC and pub/sub over WebSocket](https://github.com/crossbario/autobahn-python), and every WAMP client [needs a WAMP router to talk to](https://autobahn.readthedocs.io/en/latest/wamp/programming.html#wamp-programming-1). Write components with functions and decorators, [the recommended approach](https://autobahn.readthedocs.io/en/latest/wamp/programming.html#creating-components-1).
 
+aiortc implements [WebRTC and ORTC on asyncio](https://aiortc.readthedocs.io/en/latest/), with an API that closely follows the JavaScript one: promises become coroutines. Since it's plain Python, one server can [handle both signaling and data channels](https://aiortc.readthedocs.io/en/latest/#why-should-i-use-aiortc), or run computer vision on incoming video frames.
+
 Always [secure WebSocket connections with TLS](https://websockets.readthedocs.io/en/stable/topics/security.html#encryption) in production. Any site can open a WebSocket to yours, with your users' cookies attached. If you serve private data, [restrict the allowed origins](https://channels.readthedocs.io/en/latest/topics/security.html#websockets): Channels has `AllowedHostsOriginValidator`, websockets has the [`origins` argument](https://websockets.readthedocs.io/en/stable/reference/asyncio/server.html#websockets.asyncio.server.serve), and Flask-SocketIO [allows only the same origin by default](https://flask-socketio.readthedocs.io/en/latest/deployment.html#cross-origin-controls).
 
 Broadcasting across processes needs a message broker such as Redis. Channels' [production channel layer](https://channels.readthedocs.io/en/latest/topics/channel_layers.html#redis-channel-layer) runs on it, Flask-SocketIO [takes it as a message queue](https://flask-socketio.readthedocs.io/en/latest/deployment.html#using-multiple-workers), with sticky sessions at the load balancer, and websockets' docs [suggest it for pub/sub](https://websockets.readthedocs.io/en/stable/faq/server.html#how-do-i-send-a-message-to-a-channel-a-topic-or-some-users).