[GH-ISSUE #60] Add optional configuration options for transformers (e.g. active/passive modes) #58

New Issue

GiteaMirror · 2026-04-11T08:41:22-05:00

GiteaMirror commented

2026-04-11 08:41:22 -05:00

Originally created by @xhzeem on GitHub (Nov 15, 2025).
Original GitHub issue: https://github.com/reconurge/flowsint/issues/60

Description:
Please add support for optional configuration options in transformers, allowing them to adapt their behavior without needing separate transformer implementations.

This would make it possible to adjust how each transformer operates — for example, running in active or passive mode, changing scan depth, setting rate limits, or enabling specific modules — while still keeping safe defaults when no options are provided.

Motivation:
Currently, transformers have fixed behavior, which limits flexibility. Many OSINT and recon workflows benefit from having configurable modes — especially for tools that can operate in both passive and active contexts.

Adding optional parameters would make transformers more powerful, modular, and adaptable across different environments (from stealth OSINT to authorized assessments).

Example use case — Crawler with Katana:
The existing crawler transformer could integrate with Katana for web crawling, and expose optional parameters such as:

mode: "active" or "passive"
depth: recursion depth for crawling
js_parse: enable or disable JavaScript parsing
rate: requests per second

This would allow users to control how aggressive or deep the crawler runs, without changing its core logic.

Example concept:

Passive mode: get urls from public archives URLfinder
Active mode: perform full recursive crawling with Katana

Expected outcome:
All transformers should be able to define optional configuration parameters, which are:

Ignored if not provided (use defaults)
Respected when specified (allowing custom behavior)

The crawler would serve as a model example for implementing and testing this feature.

Benefits:

Enables configurable active/passive operation
Increases modularity and flexibility across transformers
Keeps safe defaults for minimal-risk OSINT collection
Simplifies maintenance — one transformer, multiple behaviors

Originally created by @xhzeem on GitHub (Nov 15, 2025). Original GitHub issue: https://github.com/reconurge/flowsint/issues/60 **Description:** Please add support for **optional configuration options** in transformers, allowing them to adapt their behavior without needing separate transformer implementations. This would make it possible to adjust how each transformer operates — for example, running in **active** or **passive** mode, changing scan depth, setting rate limits, or enabling specific modules — while still keeping safe defaults when no options are provided. **Motivation:** Currently, transformers have fixed behavior, which limits flexibility. Many OSINT and recon workflows benefit from having configurable modes — especially for tools that can operate in both passive and active contexts. Adding optional parameters would make transformers more powerful, modular, and adaptable across different environments (from stealth OSINT to authorized assessments). **Example use case — Crawler with [Katana](https://github.com/projectdiscovery/katana):** The existing crawler transformer could integrate with [Katana](https://github.com/projectdiscovery/katana) for web crawling, and expose optional parameters such as: * `mode`: `"active"` or `"passive"` * `depth`: recursion depth for crawling * `js_parse`: enable or disable JavaScript parsing * `rate`: requests per second This would allow users to control how aggressive or deep the crawler runs, without changing its core logic. Example concept: * Passive mode: get urls from public archives [URLfinder](https://github.com/projectdiscovery/urlfinder) * Active mode: perform full recursive crawling with Katana **Expected outcome:** All transformers should be able to define **optional configuration parameters**, which are: * Ignored if not provided (use defaults) * Respected when specified (allowing custom behavior) The crawler would serve as a model example for implementing and testing this feature. **Benefits:** * Enables configurable active/passive operation * Increases modularity and flexibility across transformers * Keeps safe defaults for minimal-risk OSINT collection * Simplifies maintenance — one transformer, multiple behaviors

GiteaMirror commented

2026-04-11 08:41:23 -05:00

@dextmorgn commented on GitHub (Nov 15, 2025):

Hey @xhzeem,

Thank you very much for this suggestion. This implies a lot of changes but it's exactly the direction we want to go !

We could make use of the params_schema defined in the transform base.

And I very much like the "safe" behavior by default, so that we can still quickly run transforms on the go, but go deeper with fine tuning the settings.

@dextmorgn commented on GitHub (Nov 15, 2025): Hey @xhzeem, Thank you very much for this suggestion. This implies a lot of changes but it's exactly the direction we want to go ! We could make use of the `params_schema` defined in the transform base. And I very much like the "safe" behavior by default, so that we can still quickly run transforms on the go, but go deeper with fine tuning the settings.