[PR #186] feat(enrichers): make domain_to_website extractions opt-out #4056

New Issue

GiteaMirror · 2026-07-16T01:06:15-05:00

GiteaMirror commented

2026-07-16 01:06:15 -05:00

📋 Pull Request Information

Original PR: https://github.com/reconurge/flowsint/pull/186
Author: @rachit367
Created: 6/9/2026
Status: 🔄 Open

Base: main ← Head: feat/website-extraction-params

📝 Commits (1)

5a6b241 feat(enrichers): make domain_to_website extractions opt-out

📊 Changes

2 files changed (+276 additions, -94 deletions)

View changed files

📝 flowsint-enrichers/src/flowsint_enrichers/domain/to_website.py (+129 -94)
➕ flowsint-enrichers/tests/enrichers/test_domain_to_website.py (+147 -0)

📄 Description

What

Adds optional toggles to domain_to_website so users can skip the heavier extractions (page content, technologies, headers) for faster, lighter scans.

Closes #90

Why

From the issue thread: domain_to_website "extracts a lot and can be slow." Today it unconditionally fetches the page and runs full text extraction (up to 5000 chars), technology detection, and header capture on every run — even when a user only wants liveness + title. This follows the configurable-transformer convention from #60 (safe defaults, opt-in depth).

Changes

New get_params_schema() with three select params — extract_content, extract_technologies, extract_headers — all defaulting to "true", so existing behavior is unchanged. Setting any to "false" skips that work.
title, description, status_code, and active are always captured (they're cheap), so disabling the heavy options still gives a useful result.
Refactor: the two near-identical HTTPS/HTTP blocks in scan() are collapsed into a for scheme in ("https", "http") loop plus a _build_website_data() helper (so the param-gating lives in one place, not duplicated). Header selection moved to _extract_headers(). Behavior is preserved: HTTPS is tried first, HTTP is the fallback, and a failed/≥400 fetch yields an inactive Website. Also drops two unused imports.

Testing

uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q
10 passed

New tests (requests mocked, no network): params schema shape + safe defaults, default full extraction (regression), each toggle off independently, all-heavy-off still keeping core fields, and the request-failure → inactive path.

The issue also mentioned iocsearcher for richer extraction — the maintainer noted that as a "good suggestion for the future." This PR addresses the concrete follow-up in the thread (an opt-out to control cost); richer extractors can layer on top later.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/reconurge/flowsint/pull/186 **Author:** [@rachit367](https://github.com/rachit367) **Created:** 6/9/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/website-extraction-params` --- ### 📝 Commits (1) - [`5a6b241`](https://github.com/reconurge/flowsint/commit/5a6b24182deead4de9c778136cf77897ead0ffe2) feat(enrichers): make domain_to_website extractions opt-out ### 📊 Changes **2 files changed** (+276 additions, -94 deletions) <details> <summary>View changed files</summary> 📝 `flowsint-enrichers/src/flowsint_enrichers/domain/to_website.py` (+129 -94) ➕ `flowsint-enrichers/tests/enrichers/test_domain_to_website.py` (+147 -0) </details> ### 📄 Description ## What Adds optional toggles to `domain_to_website` so users can skip the heavier extractions (page content, technologies, headers) for faster, lighter scans. Closes #90 ## Why From the issue thread: `domain_to_website` "extracts a lot and can be slow." Today it unconditionally fetches the page and runs full text extraction (up to 5000 chars), technology detection, and header capture on every run — even when a user only wants liveness + title. This follows the configurable-transformer convention from #60 (safe defaults, opt-in depth). ## Changes - New `get_params_schema()` with three `select` params — `extract_content`, `extract_technologies`, `extract_headers` — **all defaulting to `"true"`**, so existing behavior is unchanged. Setting any to `"false"` skips that work. - `title`, `description`, `status_code`, and `active` are always captured (they're cheap), so disabling the heavy options still gives a useful result. - Refactor: the two near-identical HTTPS/HTTP blocks in `scan()` are collapsed into a `for scheme in ("https", "http")` loop plus a `_build_website_data()` helper (so the param-gating lives in one place, not duplicated). Header selection moved to `_extract_headers()`. Behavior is preserved: HTTPS is tried first, HTTP is the fallback, and a failed/≥400 fetch yields an inactive Website. Also drops two unused imports. ## Testing ``` uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q 10 passed ``` New tests (requests mocked, no network): params schema shape + safe defaults, default full extraction (regression), each toggle off independently, all-heavy-off still keeping core fields, and the request-failure → inactive path. > The issue also mentioned `iocsearcher` for richer extraction — the maintainer noted that as a "good suggestion for the future." This PR addresses the concrete follow-up in the thread (an opt-out to control cost); richer extractors can layer on top later. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-07-16 01:06:15 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/flowsint#4056