From the issue thread: domain_to_website "extracts a lot and can be slow." Today it unconditionally fetches the page and runs full text extraction (up to 5000 chars), technology detection, and header capture on every run — even when a user only wants liveness + title. This follows the configurable-transformer convention from #60 (safe defaults, opt-in depth).
Changes
New get_params_schema() with three select params — extract_content, extract_technologies, extract_headers — all defaulting to "true", so existing behavior is unchanged. Setting any to "false" skips that work.
title, description, status_code, and active are always captured (they're cheap), so disabling the heavy options still gives a useful result.
Refactor: the two near-identical HTTPS/HTTP blocks in scan() are collapsed into a for scheme in ("https", "http") loop plus a _build_website_data() helper (so the param-gating lives in one place, not duplicated). Header selection moved to _extract_headers(). Behavior is preserved: HTTPS is tried first, HTTP is the fallback, and a failed/≥400 fetch yields an inactive Website. Also drops two unused imports.
Testing
uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q
10 passed
New tests (requests mocked, no network): params schema shape + safe defaults, default full extraction (regression), each toggle off independently, all-heavy-off still keeping core fields, and the request-failure → inactive path.
The issue also mentioned iocsearcher for richer extraction — the maintainer noted that as a "good suggestion for the future." This PR addresses the concrete follow-up in the thread (an opt-out to control cost); richer extractors can layer on top later.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/reconurge/flowsint/pull/186
**Author:** [@rachit367](https://github.com/rachit367)
**Created:** 6/9/2026
**Status:** 🔄 Open
**Base:** `main` ← **Head:** `feat/website-extraction-params`
---
### 📝 Commits (1)
- [`5a6b241`](https://github.com/reconurge/flowsint/commit/5a6b24182deead4de9c778136cf77897ead0ffe2) feat(enrichers): make domain_to_website extractions opt-out
### 📊 Changes
**2 files changed** (+276 additions, -94 deletions)
<details>
<summary>View changed files</summary>
📝 `flowsint-enrichers/src/flowsint_enrichers/domain/to_website.py` (+129 -94)
➕ `flowsint-enrichers/tests/enrichers/test_domain_to_website.py` (+147 -0)
</details>
### 📄 Description
## What
Adds optional toggles to `domain_to_website` so users can skip the heavier extractions (page content, technologies, headers) for faster, lighter scans.
Closes #90
## Why
From the issue thread: `domain_to_website` "extracts a lot and can be slow." Today it unconditionally fetches the page and runs full text extraction (up to 5000 chars), technology detection, and header capture on every run — even when a user only wants liveness + title. This follows the configurable-transformer convention from #60 (safe defaults, opt-in depth).
## Changes
- New `get_params_schema()` with three `select` params — `extract_content`, `extract_technologies`, `extract_headers` — **all defaulting to `"true"`**, so existing behavior is unchanged. Setting any to `"false"` skips that work.
- `title`, `description`, `status_code`, and `active` are always captured (they're cheap), so disabling the heavy options still gives a useful result.
- Refactor: the two near-identical HTTPS/HTTP blocks in `scan()` are collapsed into a `for scheme in ("https", "http")` loop plus a `_build_website_data()` helper (so the param-gating lives in one place, not duplicated). Header selection moved to `_extract_headers()`. Behavior is preserved: HTTPS is tried first, HTTP is the fallback, and a failed/≥400 fetch yields an inactive Website. Also drops two unused imports.
## Testing
```
uv run --package flowsint-enrichers pytest flowsint-enrichers/tests/enrichers/ -q
10 passed
```
New tests (requests mocked, no network): params schema shape + safe defaults, default full extraction (regression), each toggle off independently, all-heavy-off still keeping core fields, and the request-failure → inactive path.
> The issue also mentioned `iocsearcher` for richer extraction — the maintainer noted that as a "good suggestion for the future." This PR addresses the concrete follow-up in the thread (an opt-out to control cost); richer extractors can layer on top later.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/reconurge/flowsint/pull/186
Author: @rachit367
Created: 6/9/2026
Status: 🔄 Open
Base:
main← Head:feat/website-extraction-params📝 Commits (1)
5a6b241feat(enrichers): make domain_to_website extractions opt-out📊 Changes
2 files changed (+276 additions, -94 deletions)
View changed files
📝
flowsint-enrichers/src/flowsint_enrichers/domain/to_website.py(+129 -94)➕
flowsint-enrichers/tests/enrichers/test_domain_to_website.py(+147 -0)📄 Description
What
Adds optional toggles to
domain_to_websiteso users can skip the heavier extractions (page content, technologies, headers) for faster, lighter scans.Closes #90
Why
From the issue thread:
domain_to_website"extracts a lot and can be slow." Today it unconditionally fetches the page and runs full text extraction (up to 5000 chars), technology detection, and header capture on every run — even when a user only wants liveness + title. This follows the configurable-transformer convention from #60 (safe defaults, opt-in depth).Changes
get_params_schema()with threeselectparams —extract_content,extract_technologies,extract_headers— all defaulting to"true", so existing behavior is unchanged. Setting any to"false"skips that work.title,description,status_code, andactiveare always captured (they're cheap), so disabling the heavy options still gives a useful result.scan()are collapsed into afor scheme in ("https", "http")loop plus a_build_website_data()helper (so the param-gating lives in one place, not duplicated). Header selection moved to_extract_headers(). Behavior is preserved: HTTPS is tried first, HTTP is the fallback, and a failed/≥400 fetch yields an inactive Website. Also drops two unused imports.Testing
New tests (requests mocked, no network): params schema shape + safe defaults, default full extraction (regression), each toggle off independently, all-heavy-off still keeping core fields, and the request-failure → inactive path.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.