[PR #183] [MERGED] feat(types,enrichers): add Technology type and tech_detect transformer #3863

New Issue

GiteaMirror · 2026-07-14T19:50:46-05:00

GiteaMirror commented

2026-07-14 19:50:46 -05:00

📋 Pull Request Information

Original PR: https://github.com/reconurge/flowsint/pull/183
Author: @rachit367
Created: 6/9/2026
Status: ✅ Merged
Merged: 6/20/2026
Merged by: @dextmorgn

Base: main ← Head: feat/technology-type-and-tech-detect

📝 Commits (1)

63a97a7 feat(types,enrichers): add Technology type and tech_detect transformer

📊 Changes

4 files changed (+305 additions, -0 deletions)

View changed files

➕ flowsint-enrichers/src/flowsint_enrichers/website/to_technologies.py (+133 -0)
➕ flowsint-enrichers/tests/enrichers/test_tech_detect.py (+112 -0)
📝 flowsint-types/src/flowsint_types/__init__.py (+3 -0)
➕ flowsint-types/src/flowsint_types/technology.py (+57 -0)

📄 Description

What

Adds application-level technology fingerprinting as requested in #59:

a new Technology type, and
a new tech_detect transformer (Website → Technology) built on the existing httpx tool.

Closes #59

Why

Flowsint already models infrastructure (ip, port, website, ssl_certificate). This extends coverage into the application layer — what software/frameworks a site runs — which complements port/website data and lays groundwork for stack mapping and vuln correlation (the benefits called out in the issue).

Changes

flowsint-types/technology.py (new) — Technology(name, version, category, source), modeled on the existing WebTracker type. Registered via @flowsint_type, exported from flowsint_types/__init__.py, and added to TYPE_TO_MODEL. from_string accepts both bare names (nginx) and wappalyzer-style name:version pairs (PHP:8.1).

flowsint-enrichers/website/to_technologies.py (new) — TechDetectEnricher (name = "tech_detect"), auto-registered. Input Website, output Technology. Runs the existing HttpxTool with -td (httpx's wappalyzer-based tech detection), reads the tech field from the JSON output, de-dupes per probe, and links each technology to the source website with a USES_TECHNOLOGY relationship. Follows the established tool-backed enricher pattern (e.g. website_to_webtrackers, ip_to_ports).

The tech JSON field and the -td flag are per httpx's runner.Result struct (Technologies []string +"json:\"tech\""+`).

Testing

uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q
13 passed

Tests cover the Technology type (label, from_string, registry membership), the tech-entry parser, enricher registration/metadata, and scan() with the httpx Docker tool mocked — so the suite needs no Docker daemon.

Scope note: the issue mentions an optional port input and optional Wappalyzer integration. I kept this PR to the Website → Technology path using httpx's built-in wappalyzer dataset (-td), which is the smallest complete slice. Happy to follow up with a port-input variant if maintainers want it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/reconurge/flowsint/pull/183 **Author:** [@rachit367](https://github.com/rachit367) **Created:** 6/9/2026 **Status:** ✅ Merged **Merged:** 6/20/2026 **Merged by:** [@dextmorgn](https://github.com/dextmorgn) **Base:** `main` ← **Head:** `feat/technology-type-and-tech-detect` --- ### 📝 Commits (1) - [`63a97a7`](https://github.com/reconurge/flowsint/commit/63a97a76f77c60cf6210358a40ee892fe7b75e99) feat(types,enrichers): add Technology type and tech_detect transformer ### 📊 Changes **4 files changed** (+305 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `flowsint-enrichers/src/flowsint_enrichers/website/to_technologies.py` (+133 -0) ➕ `flowsint-enrichers/tests/enrichers/test_tech_detect.py` (+112 -0) 📝 `flowsint-types/src/flowsint_types/__init__.py` (+3 -0) ➕ `flowsint-types/src/flowsint_types/technology.py` (+57 -0) </details> ### 📄 Description ## What Adds application-level technology fingerprinting as requested in #59: - a new **`Technology`** type, and - a new **`tech_detect`** transformer (`Website` → `Technology`) built on the existing `httpx` tool. Closes #59 ## Why Flowsint already models infrastructure (`ip`, `port`, `website`, `ssl_certificate`). This extends coverage into the application layer — what software/frameworks a site runs — which complements port/website data and lays groundwork for stack mapping and vuln correlation (the benefits called out in the issue). ## Changes **`flowsint-types/technology.py`** (new) — `Technology(name, version, category, source)`, modeled on the existing `WebTracker` type. Registered via `@flowsint_type`, exported from `flowsint_types/__init__.py`, and added to `TYPE_TO_MODEL`. `from_string` accepts both bare names (`nginx`) and wappalyzer-style `name:version` pairs (`PHP:8.1`). **`flowsint-enrichers/website/to_technologies.py`** (new) — `TechDetectEnricher` (`name = "tech_detect"`), auto-registered. Input `Website`, output `Technology`. Runs the existing `HttpxTool` with `-td` (httpx's wappalyzer-based tech detection), reads the `tech` field from the JSON output, de-dupes per probe, and links each technology to the source website with a `USES_TECHNOLOGY` relationship. Follows the established tool-backed enricher pattern (e.g. `website_to_webtrackers`, `ip_to_ports`). The `tech` JSON field and the `-td` flag are per httpx's `runner.Result` struct (`Technologies []string `+"`json:\"tech\"`"+`). ## Testing ``` uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q 13 passed ``` Tests cover the `Technology` type (label, `from_string`, registry membership), the tech-entry parser, enricher registration/metadata, and `scan()` with the `httpx` Docker tool mocked — so the suite needs **no Docker daemon**. > Scope note: the issue mentions an optional `port` input and optional Wappalyzer integration. I kept this PR to the `Website` → `Technology` path using httpx's built-in wappalyzer dataset (`-td`), which is the smallest complete slice. Happy to follow up with a `port`-input variant if maintainers want it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-07-14 19:50:46 -05:00

GiteaMirror closed this issue

2026-07-14 19:50:47 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/flowsint#3863