[PR #130] [PROPOSAL] Plugin Discovery and Namespace Code Structure #568

New Issue

GiteaMirror · 2026-04-19T13:10:04-05:00

GiteaMirror commented

2026-04-19 13:10:04 -05:00

📋 Pull Request Information

Original PR: https://github.com/reconurge/flowsint/pull/130
Author: @gustavorps
Created: 3/15/2026
Status: 🔄 Open

Base: main ← Head: feat/ns-and-plugin

📝 Commits (1)

0a82d24 Plugin discovery; Namespace refactor code structure for improved readability and maintainability

📊 Changes

25 files changed (+3850 additions, -2 deletions)

View changed files

📝 flowsint-api/app/api/routes/events.py (+2 -1)
📝 flowsint-core/src/flowsint_core/core/events.py (+8 -1)
➕ python/.python-version (+1 -0)
➕ python/packages/flowsint-core/README.md (+0 -0)
➕ python/packages/flowsint-core/pyproject.toml (+16 -0)
➕ python/packages/flowsint-core/src/flowsint/core/__init__.py (+1 -0)
➕ python/packages/flowsint-core/src/flowsint/core/__main__.py (+1 -0)
➕ python/packages/flowsint-enrichers/README.md (+0 -0)
➕ python/packages/flowsint-enrichers/pyproject.toml (+21 -0)
➕ python/packages/flowsint-enrichers/src/flowsint/enricher/__init__.py (+17 -0)
➕ python/packages/flowsint-enrichers/src/flowsint/enricher/abc.py (+8 -0)
➕ python/packages/flowsint-enrichers/src/flowsint/enricher/util/importlib.py (+185 -0)
➕ python/packages/flowsint-example-plugin/README.md (+0 -0)
➕ python/packages/flowsint-example-plugin/pyproject.toml (+22 -0)
➕ python/packages/flowsint-example-plugin/src/flowsint/example/__init__.py (+0 -0)
➕ python/packages/flowsint-example-plugin/src/flowsint/example/enricher/__init__.py (+2 -0)
➕ python/packages/flowsint-example-plugin/src/flowsint/example/enricher/mock.py (+38 -0)
➕ python/packages/flowsint-types/README.md (+0 -0)
➕ python/packages/flowsint-types/pyproject.toml (+19 -0)
➕ python/packages/flowsint-types/src/flowsint/types/__init__.py (+1 -0)

...and 5 more files

📄 Description

This pull request introduces a new plugin-based architecture for enrichers in the Flowsint project, leveraging Python's entry points and lazy loading to improve modularity and startup performance. It also adds new workspace package configurations and updates Python version requirements. The most important changes are grouped below:

Plugin architecture and lazy loading

Added a PEP 810-style lazy import utility in flowsint.enricher.util.importlib, enabling modules and entry points to be loaded only when first accessed, which reduces unnecessary imports and speeds up startup.
Updated flowsint.enricher.__init__.py to discover and lazily load plugins via entry points, printing loaded plugins and registry contents for debugging.
Introduced EnricherABC in flowsint.enricher.abc.py, which automatically registers subclasses with the enricher registry, supporting plugin discovery and registration.

Example plugin implementation

Added a new example plugin package flowsint-example-plugin with entry point configuration and a sample Enricher class, demonstrating the new plugin system and type annotations for input/output. [1] [2] [3]

Workspace and core package changes

Added workspace-compatible pyproject.toml files for flowsint-core-next, flowsint-enrichers-next, and flowsint-types-next, specifying Python version requirements and editable sources for modular development. [1] [2] [3]
Added __init__.py files to expose relevant symbols and updated the Python version to 3.13 in .python-version for compatibility with new features. [1] [2] [3]

Redis connection initialization fix

Deferred Redis connection in EventEmitter by adding a connect() method, preventing premature connection during module import and fixing issues when importing enrichers. [1] [2]

These changes lay the foundation for a scalable, plugin-based enrichment system with improved dependency management and startup efficiency.

How to test the PR

% cd python

% uv sync

% source .venv/bin/activate

% python -c 'import flowsint.enricher'
{'flowsint.example.enricher.mock:Enricher': <lazy entry point 'flowsint.example.enricher.mock:Enricher'>}
EntryPoint(name='flowsint.example.enricher.mock:Enricher', value='flowsint.example.enricher.mock:Enricher', group='flowsint.enricher')
<class 'flowsint.example.enricher.mock.Enricher'>
{'my_enricher': <class 'flowsint.example.enricher.mock.Enricher'>}

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/reconurge/flowsint/pull/130 **Author:** [@gustavorps](https://github.com/gustavorps) **Created:** 3/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/ns-and-plugin` --- ### 📝 Commits (1) - [`0a82d24`](https://github.com/reconurge/flowsint/commit/0a82d248efea3108ca58f8fedb90ed08d9050bb4) Plugin discovery; Namespace refactor code structure for improved readability and maintainability ### 📊 Changes **25 files changed** (+3850 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `flowsint-api/app/api/routes/events.py` (+2 -1) 📝 `flowsint-core/src/flowsint_core/core/events.py` (+8 -1) ➕ `python/.python-version` (+1 -0) ➕ `python/packages/flowsint-core/README.md` (+0 -0) ➕ `python/packages/flowsint-core/pyproject.toml` (+16 -0) ➕ `python/packages/flowsint-core/src/flowsint/core/__init__.py` (+1 -0) ➕ `python/packages/flowsint-core/src/flowsint/core/__main__.py` (+1 -0) ➕ `python/packages/flowsint-enrichers/README.md` (+0 -0) ➕ `python/packages/flowsint-enrichers/pyproject.toml` (+21 -0) ➕ `python/packages/flowsint-enrichers/src/flowsint/enricher/__init__.py` (+17 -0) ➕ `python/packages/flowsint-enrichers/src/flowsint/enricher/abc.py` (+8 -0) ➕ `python/packages/flowsint-enrichers/src/flowsint/enricher/util/importlib.py` (+185 -0) ➕ `python/packages/flowsint-example-plugin/README.md` (+0 -0) ➕ `python/packages/flowsint-example-plugin/pyproject.toml` (+22 -0) ➕ `python/packages/flowsint-example-plugin/src/flowsint/example/__init__.py` (+0 -0) ➕ `python/packages/flowsint-example-plugin/src/flowsint/example/enricher/__init__.py` (+2 -0) ➕ `python/packages/flowsint-example-plugin/src/flowsint/example/enricher/mock.py` (+38 -0) ➕ `python/packages/flowsint-types/README.md` (+0 -0) ➕ `python/packages/flowsint-types/pyproject.toml` (+19 -0) ➕ `python/packages/flowsint-types/src/flowsint/types/__init__.py` (+1 -0) _...and 5 more files_ </details> ### 📄 Description This pull request introduces a new plugin-based architecture for enrichers in the Flowsint project, leveraging Python's entry points and lazy loading to improve modularity and startup performance. It also adds new workspace package configurations and updates Python version requirements. The most important changes are grouped below: ### Plugin architecture and lazy loading * Added a PEP 810-style lazy import utility in `flowsint.enricher.util.importlib`, enabling modules and entry points to be loaded only when first accessed, which reduces unnecessary imports and speeds up startup. * Updated `flowsint.enricher.__init__.py` to discover and lazily load plugins via entry points, printing loaded plugins and registry contents for debugging. * Introduced `EnricherABC` in `flowsint.enricher.abc.py`, which automatically registers subclasses with the enricher registry, supporting plugin discovery and registration. ### Example plugin implementation * Added a new example plugin package `flowsint-example-plugin` with entry point configuration and a sample `Enricher` class, demonstrating the new plugin system and type annotations for input/output. [[1]](diffhunk://#diff-2dc1986222d4c56d8056316cb601f335c4a7412b9f5a16dae211bb4f061660b7R1-R22) [[2]](diffhunk://#diff-a18e3a4a85d67a048c9dc931a0a698ad9c68eca9589ad0d7deeba1e884a44515R1-R38) [[3]](diffhunk://#diff-a118589c5d5dc8e5cbd7265b910e9a0cde830a18d63e9f89f2048a8c9c30c989R1-R2) ### Workspace and core package changes * Added workspace-compatible `pyproject.toml` files for `flowsint-core-next`, `flowsint-enrichers-next`, and `flowsint-types-next`, specifying Python version requirements and editable sources for modular development. [[1]](diffhunk://#diff-420719a24228e011621535ebd737b3a6cbe15a6778cbb8ab5545149f2b45800aR1-R16) [[2]](diffhunk://#diff-dc7871996df4c93cb1df69bfa9559d42200a8ff85fd335940c5335428c7e7870R1-R21) [[3]](diffhunk://#diff-66c29c9180e263191e58a981a147b951ff34a83984c41e027950686f94aa227eR1-R19) * Added `__init__.py` files to expose relevant symbols and updated the Python version to 3.13 in `.python-version` for compatibility with new features. [[1]](diffhunk://#diff-87676f4511c36ac0e9f30423599962c3ea0765c3bae43c552533bd8f548bf73bR1) [[2]](diffhunk://#diff-698245076f4723d53313a3bf7b50ead80321242680024fab8b50e8bb62325376R1) [[3]](diffhunk://#diff-36555f52c89477a2117c3cca9e1e559e0a4e2e53c7f03e79e45f1e9156c2991eR1) ### Redis connection initialization fix * Deferred Redis connection in `EventEmitter` by adding a `connect()` method, preventing premature connection during module import and fixing issues when importing enrichers. [[1]](diffhunk://#diff-c24bd20bab9e13ef289d3abd084e68cb31900123d3caba41b3ebdfa85b110c5bL14-R23) [[2]](diffhunk://#diff-4618ab6223a9b0c7dfdd39027059d78e2afe5210aba87195eea9bab5757cc442L20-R21) These changes lay the foundation for a scalable, plugin-based enrichment system with improved dependency management and startup efficiency. ## How to test the PR ```sh % cd python % uv sync % source .venv/bin/activate % python -c 'import flowsint.enricher' {'flowsint.example.enricher.mock:Enricher': <lazy entry point 'flowsint.example.enricher.mock:Enricher'>} EntryPoint(name='flowsint.example.enricher.mock:Enricher', value='flowsint.example.enricher.mock:Enricher', group='flowsint.enricher') <class 'flowsint.example.enricher.mock.Enricher'> {'my_enricher': <class 'flowsint.example.enricher.mock.Enricher'>} ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-19 13:10:04 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/flowsint#568