Replace category drilldown with a search bar over a flat type list.
GET /types gains a flat=true param that flattens categories server-side
(custom types included); useActionItems(true) opts into it while every
other caller keeps the nested default.
The flow builder's params dialog rendered every non-vault-secret param as
<input type={param.type}>. For a select param that emits an invalid input
type, which the browser silently downgrades to a text box, and param.options
was never read. Enrichers that declare select params (ip_to_ports mode and
service_detection, domain_to_dns) were unusable there. Route select params
through the Select component instead, matching enricher-params-sheet.
The dialog also seeded a hardcoded settings object and wrote it onto the
saved node. Nothing renders it and no backend code reads a settings key from
flow node data, so drop the state and the Enricher.settings type field.
Both graph context menus keyed their enricher list on enricher.id, which is
absent on registry enrichers, so most rows shared an undefined key. Key on
name, as launch-enricher.tsx already does.
launch_enricher took payload.sketch_id on trust: any authenticated caller
could dispatch an enricher against a sketch belonging to someone else, and the
worker would write its findings into that sketch.
EnricherService gains get_sketch_for_launch, the same lookup launch_flow has
used since ac485e2c: resolve the sketch, then _check_permission for "update"
on its investigation. No role row means no access, so an unknown sketch 404s
and a viewer or a stranger gets 403, matching the flow route's responses.
The check runs before the graph read, so a denied caller never reaches Neo4j
and never queues a task.
Unlike FlowService's copy, the sketch id is parsed to a UUID first.
SketchRepository.get_by_id binds it as one, so a non-UUID string reached
SQLAlchemy and raised, which would have turned a bad id into a 500 instead of
a denial.
GET /api/enrichers listed templates through get_by_owner, while
POST /api/enrichers/{name}/launch resolves a name the registry does not know
through find_by_name_and_owner_or_public. Another user's public template was
therefore launchable but never listed, so the UI could not discover it and
never saw its params_schema.
The listing now uses the repository's existing get_by_owner_or_public, the same
visibility rule the launch lookup applies. The category filter is unchanged.
EnricherCreate/EnricherRead/EnricherUpdate had no importer anywhere in the
repo and were the whole content of app/api/schemas/enricher.py, so the module
goes with them.
get_enrichers' custom-type branch returns [] before reaching the
wobbly_type=True registry listing. The early return stays (why a published
custom type lists no enrichers is a separate question); the line it strands
does not.
EnricherBase.set_params() had no caller and no dynamic-dispatch site;
resolve_params/async_init assign self.params directly.
Same function, zero callers anywhere in the repo (see prior commit's
app/utils.py deletion for the third, now-removed copy). Also trims
flowsint_enrichers/utils.py down to get_root_domain, its only function
with any caller.
flowsint-api/app/utils.py — deleted entirely. Nothing imports it (its
one plausibly-used function, extract_input_schema_flow, is actually
imported from flowsint_core.utils everywhere it's called); it was a
never-referenced full duplicate of the core module.
flowsint_core/utils.py and flowsint_enrichers/utils.py — dropped
get_domain_from_ssl (identical in both, zero callers anywhere in the
repo) and its now-unused ssl/socket/Tuple/cast imports.
flowsint_enrichers/utils.py further trimmed to just get_root_domain,
its only function actually imported anywhere (domain/to_root_domain.py).
The rest (is_valid_ip, resolve_type, extract_enricher, flatten, ...)
had zero callers and zero test coverage — dead since the file was
apparently copy-pasted from flowsint_core/utils.py rather than reusing
it directly.
flowsint_core/utils.py duplicate
Touched by the TLS pin, which pulled this file into the changed-files
mypy ratchet — it had never been typed (is_valid_domain annotated -> str
while returning bool, resolve_type leaking Any, unguarded cert.get()
on a possibly-None cert, two untyped defs). Same fixes flowsint_core's
copy already got.
ssl.create_default_context() already excludes TLS1/1.1 by default in
modern Python/OpenSSL, but CodeQL flags the implicit reliance on that
default. Pin minimum_version explicitly instead of leaving it to
library defaults.
make lint (wired to run flowsint-app prettier/eslint in 88828c5f) needs
Node/yarn, but the job calling it only set up uv — prettier was missing
from PATH. Add setup-node + yarn install. Also drop the now-redundant
frontend-lint job: it duplicated the same checkout/install/format/lint
steps that make lint already runs in python-lint.
holehe/pydig/hibpwned were declared but never imported anywhere in the
codebase. ignorant was the only one actually used, by IgnorantEnricher
(phone_to_infos) — removed along with it. Eliminates all GPLv3
exposure from the dependency tree; remaining LGPL/MPL deps
(psycopg2-binary, stem, pycountry, certifi) are unmodified
weak-copyleft libs used as plain imports, no distribution obligation.
make lint was Python-only (PY_SRC never included flowsint-app) —
frontend prettier/eslint lived solely in flowsint-app's own yarn
scripts, disconnected from the repo's single lint entry point. Both
are 0-errors clean today (eslint's ~130 findings are all warnings),
same bar the Python side is already held to, so added as blocking
rather than a separate opt-in step. CONTRIBUTING.md updated to match.
.eslintrc.cjs + .eslintignore replaced by a single eslint.config.ts.
Requires ESLint 9 — flat config's TS-file loading and the plugins'
flat presets aren't available on 8.x, and v8 -> v9 is the only jump
that's actually clean: eslint-plugin-react (react/recommended,
jsx-runtime, and the two rules this project deliberately disables —
prop-types, display-name) hard-crashes on ESLint 10 (calls the
removed context.getFilename() API, confirmed by trying it — no
released or canary version fixes this). eslint-plugin-react-hooks and
typescript-eslint both declare and work fine on 10, but react doesn't,
so 9.39.5 is the newest version where all three plugins actually work
together. Every devDependency in this stack now pinned to an exact
version rather than a caret range, deliberately: this exact
combination is what got verified, and a silent minor/patch bump on
any one of them is exactly how this session hit the v10 breakage.
Rule set is unchanged, not just reorganized — verified the finding
count is identical before/after (129 problems: 0 errors, 129
warnings; 112 of them no-explicit-any, matching the pre-migration
count exactly) by resolving each plugin's flat preset explicitly
(tsPlugin.configs['flat/recommended'], react.configs.flat.recommended
+ ['jsx-runtime'], reactHooks.configs.flat.recommended — the plain
.recommended, not recommended-latest, since eslintrc's
'plugin:react-hooks/recommended' resolved to that same rule set, not
the newer superset) instead of guessing at equivalents. tsc --noEmit
output is also byte-identical before/after.
package.json's lint/lint:check scripts drop the now-meaningless --ext
flag — flat config determines matched files via the config's own
`files` patterns, not CLI extension lists (confirmed harmless either
way, but keeping it would just be dead noise). CI's per-file ratchet
invocation (yarn eslint <files>) and lint-staged's prettier-only setup
are untouched — neither depended on the config file format.
Split by Python vs frontend, matches exactly what CI actually gates
(make lint/typecheck, tsc/eslint/prettier/vitest/build) so it doesn't
drift into its own source of truth. Also notes what the lint-staged
pre-commit hook already covers (ruff, prettier) vs what's still manual
(mypy, eslint, tests), and why the mypy/eslint ratchets exist.
README's Poetry mention was stale (repo uses uv) — fixed in passing,
and pointed its Contributing section at the new file instead of
duplicating instructions.
The react-hooks/set-state-in-effect cleanup earlier in this branch
converted several "sync local form state from loaded data" effects to
the adjust-during-render pattern: track a prevValue via useState, and
when it differs from the current value, sync state during render
instead of in an effect.
That pattern has a real gap the original effects didn't: useEffect
runs on mount regardless of dependencies, but `useState(value)` for
the tracked prevValue captures the *current* value immediately, so on
mount `value === prevValue` already and the sync block never fires.
Any of these components that can mount with data already available
(query cache hit, or a prop that's simply non-null from the start)
silently kept its hardcoded-empty initial state forever — until the
underlying value later *changed* to something else.
Confirmed regression: node details panel's form (name/color/icon/...)
stayed empty on the first node selected after mount, only populating
once you clicked a second, different node. Same shape, same fix
(derive the initial useState from the current value via a lazy
initializer instead of a hardcoded default) applied to every sibling
with the same bug shape: sketch settings form, analysis title, user
profile form, and the custom-type editor (which also never called its
add-first-field fallback for a brand new type, for the same reason).
Audited every other adjust-during-render conversion from that branch;
the rest already seeded their initial state directly from the current
prop/value (e.g. useState(value) instead of useState('')), so they
don't have this gap.
Two useMemo deps arrays indexed columnSizeVars with a dynamic template-
literal key directly inline (columnSizeVars[\`--col-\${cell.column.id}-size\`]),
which react-hooks/exhaustive-deps can't statically verify — it errors
(not warns) on "Expected the dependency list to be an array of simple
expressions". Extracted each into a local variable above the useMemo
call so the dependency array only references plain identifiers.
This was the only actual error in the file; the remaining ~17 findings
are pre-existing warnings (unrelated missing deps, TanStack's
documented incompatible-library case) that don't fail the ratchet.
The ratchet was still red: every one of the 4 workspace packages
imports the others across package boundaries constantly, and none
ship a py.typed marker, so mypy treats each cross-package import as
untyped and flags it — not per-file, on almost every file in the
monorepo. Adding py.typed is the real fix but is its own project (see
the previous commits in this branch: tried it, it unmasked the full
~2200-error pre-existing backlog instead of scoping to what this
branch touches, reverted). Disabling the code is the honest call for
now — local logic within each file is still fully checked, only the
"can't verify types crossing this specific import" complaint is
silenced.
mypy now reports 0 errors across all 4 packages for files this branch
touches. Verified: full test suite green (397 + 32 + 6 + 54).
Last batch of real fixes plus one more scoped override:
- DockerTool.get_image() read cls.image as a class attribute, but
__init__ only ever sets it on the instance (self.image = ...) — real
bug (would AttributeError if get_image() were ever called before an
instance existed with a subclass not also declaring image at class
level). Declared `image: str` on the base to document what's
actually relied on: subclasses set it as a class attribute, __init__
shadows it per-instance with the tagged form.
- ReconCrawlTool.version() was declared -> str but its body was a bare
`pass` (returns None) — no callers exist, so raises NotImplementedError
with a comment instead of guessing a fake version string.
- ReconCrawlTool.launch()'s args param defaulted to None but the body
calls args.get(...) unconditionally — real crash risk if ever called
without args (never observed since the one caller always passes a
dict), guarded with `args = args or {}`.
- Extended flowsint-core's tests.* override (assignment, override) for
the same InputType/OutputType-pattern reason as the existing entries:
MockEnricher/ResolveEnricher fixtures reassign InputType/OutputType
and vary scan()/postprocess() signatures per the documented pattern,
same as production enrichers.
- permissions.py, alembic/env.py, and one alembic migration's untyped
params/standard-boilerplate arg-type gaps.
Ratchet is now clean except the accepted import-untyped debt (86
errors — cross-package imports needing py.typed markers project-wide,
deliberately out of scope: adding them earlier in this branch
ballooned the count from 478 to 1423 by unmasking the full ~2200
pre-existing backlog instead of shrinking it).
Verified: all 4 packages' test suites pass (397 + 32 + 6 + 54).
Without pydantic's mypy plugin, mypy doesn't understand Field()'s
required/optional semantics and flags every pydantic model
construction that omits a field with a default as a "missing named
argument" false positive (e.g. ASN(asn_str=...) flagged for not
passing nodeLabel/number/name/country/description, all of which have
defaults). Enabled across all 4 packages — fixes this whole category
at once rather than casting/ignoring each constructor call site.
Also: Enricher.postprocess's input_data was typed Optional[List[str]],
which never matched reality — every subclass's input_data is really
List[InputType] (per-subclass, same as results), and one subclass
(TemplateEnricher's social-account postprocess) explicitly used
List[Any], which is where the mismatch actually surfaced as an
override error. Widened to Optional[List[Any]], matching how
InputType/OutputType are already treated elsewhere in this class.
Also fixed that override's mutable default arg (List[Any] = []).
Rest is the same mechanical pattern as the previous commit (implicit-
Optional params, dict/list literal annotations) in yaml_loader.py,
template_enricher.py's __init__/_detect_type/async_init, and
test_template_enricher.py's helper functions.
Verified: flowsint-core's 397 tests pass unchanged.
Mechanical annotation additions (return types, Optional defaults for
implicit-None params, dict/list literal types) across ~15 files, plus
a handful of real bugs surfaced once the noise cleared:
- WhoxyTool.launch() was declared -> list[Dict] but always returns a
single dict (`return data` where data = resp.json(), and every
caller reads it as one: `infos.get("search_result")`). Fixed the
annotation to match actual behavior and every call site.
- get_inline_relationships() was declared -> List[str] but returns
List[Dict[str, Any]] ({source, edge, target} per relationship) —
this is the backend half of the sketch graph "inline" format the
frontend's sketchService.getGraphDataById overload already models;
SketchService.get_graph()'s return type now reflects the same
format-dependent Dict-or-List shape.
- get_db() used `yield db` (a generator, for FastAPI's Depends) but
was typed -> Session instead of -> Generator[Session, None, None],
breaking `next(get_db())` call sites like logger.py's batch flush.
- build_params_model() returns a dynamically-built model class via
pydantic's create_model(), typed -> BaseModel (an instance type)
instead of -> Type[BaseModel], which made self.ParamsModel(**kwargs)
look like calling an instance.
- Every *_dehashed.py/to_ports.py/to_domains.py enricher's __init__
had an untyped vault=None param — typed Optional[VaultProtocol],
matching the base Enricher.__init__.
Also adds a tools.* mypy override in flowsint-enrichers (disables
override) for the same reason as the InputType/OutputType one:
Tool.launch()'s abstract signature is deliberately loose
(*args, **kwargs) and every concrete tool (dnsx, httpx, subfinder,
whoxy, ...) defines its own real signature — not a mistake to fix
per-subclass.
Verified: flowsint-core (397) and flowsint-enrichers (32) tests pass
unchanged.
Pytest fixture params carry their real types from conftest.py fixture
definitions scattered across the tree, not the function signature —
fully annotating every test function's fixture params for marginal
safety on non-public test code (~150 errors, mostly in tools/network
test doubles and repository test suites) isn't worth the churn. src/
stays strict; this is a deliberate, scoped relaxation per package, not
a blanket one.
Also adds the missing tests/__init__.py to flowsint-api — its tests
dir was the only one without it, so its module name never actually
matched the tests.* override glob (mypy silently no-op'd it, flagged
via its own "unused section" note). The other three packages already
had this.
Enrichers declare InputType/OutputType as plain class attributes
(NotImplemented on the base, overridden per subclass) and use them in
annotations like List[InputType] and self.data.domain — the documented,
intentional API (see flowsint-enricher-builder skill), not something
to individually silence. mypy structurally can't verify a class
attribute doubling as a type alias without making Enricher generic,
which would mean touching every enricher's class declaration. Narrows
the specific checks that pattern trips (valid-type, attr-defined,
name-defined) via per-module overrides in flowsint-enrichers and
flowsint-core's tests (MockEnricher fixtures use the same pattern).
Separately, fixes real gaps in orchestrator.py surfaced once the noise
cleared: several dict literals (step_result, log_entry, results,
results_mapping, branch_results) were inferred as narrower types than
what the code actually assigns into them; execution_log_file and the
enrichers map were missing their real (Optional) types; _create_execution_log
used self.execution_log_file across an assignment mypy couldn't narrow,
fixed via a local variable. update_results_mapping's outputs param is
typed to match what it can actually receive (dict or list, per the
isinstance check upstream) with a comment on the pre-existing dict-only
assumption in its body — not silently "fixed" since that would change
behavior for list-shaped enricher outputs, unverified here. scan()'s
Liskov-incompatible override (sync, single-dict vs the base's async/
list-of-dicts contract) is documented and suppressed, not changed —
that's a behavioral call about how FlowOrchestrator is actually
invoked. graph_edge_to_neo4j_dict/serializer.py's from_obj/to_obj were
typed as generic BaseModel while the body only ever handles
FlowsintType or GraphNode — narrowed to that Union. And
prepare_enricher_inputs, which has no callers anywhere, is typed Any
rather than guessing at intended behavior for dead code whose two
return branches already disagree.
Verified: flowsint-core's 397 tests pass unchanged.
Calling \`uv run ruff\` twice per changed file (up to ~500 subprocess
spawns on this branch) was flaky: some invocations intermittently
returned empty output, making the byte-for-byte comparison fail and
the file wrongly counted as semantically changed. Resolving the
binary path once via \`uv run which ruff\` and invoking it directly
in the loop is both correct and faster. Verified stable across
repeated runs.
alembic/ was never in PY_SRC, so it never went through black/isort/
flake8 before, or ruff after — 31 files were unformatted. Adds it to
PY_SRC and reformats via ruff format + check --fix, plus a per-file-
ignore for env.py's star-import-from-models pattern (standard Alembic
autogenerate setup: needs every model's metadata visible under one
name, not enumerable without duplicating the model list here).
The ratchet gates mypy on files this branch "touches" vs BASE_REF,
to avoid blocking every PR on the ~2200 pre-existing errors. The ruff
adoption reformatted the whole tree, so every .py file differs from
main by formatting alone — the ratchet degraded into gating the
entire backlog on this one PR.
A file now only counts as changed if re-running ruff (format + fix)
on the BASE_REF version doesn't reproduce today's content
byte-for-byte. Verified locally: 578 errors across 31 files dropped
to the files actually edited this session.
make lint/lint-fix run \`uv run ruff\` from the workspace root, but
ruff was only declared in each package's own dev-dependency group —
never in the root project's. Worked locally by accident (shared venv
carried leftovers from earlier per-package syncs); a fresh CI clone
hit "Failed to spawn: ruff". Declare it in the root dev group too.
dextmorgn
github-actions[bot]