Commit Graph

840 Commits

Author SHA1 Message Date
dextmorgn
3885f8b4d1 Merge pull request #188 from Ghraven/fix/remove-frontend-debug-logs
fix(ui): remove frontend debug console logs
2026-06-11 12:37:11 +02:00
Ghraven
8de27796b0 fix(ui): remove frontend debug console logs 2026-06-11 13:55:17 +08:00
dextmorgn
12bf2937c1 Merge pull request #176 from reconurge/ci/python-tests
ci: run python tests on pull requests and main
v1.2.10
2026-06-05 15:47:32 +02:00
dextmorgn
612d39eadf test: provide dummy Neo4j credentials in conftest
Any code path reaching the Neo4jConnection singleton requires NEO4J_*
env vars at construction — tests only passed thanks to developers'
local .env files. Driver creation is lazy, nothing connects.

Verified by running all four suites with every .env removed:
458 passed.
2026-06-05 15:34:58 +02:00
dextmorgn
dc36ef633e test(core): factor out env-independent no-connection repo setup
Previous fix covered one occurrence; 17 other tests built
Neo4jGraphRepository(neo4j_connection=None), hitting the singleton
fallback that requires NEO4J_* env vars. Single helper, no env
dependency left.
2026-06-05 15:26:08 +02:00
dextmorgn
22c6908a0e test(core): stop relying on env credentials in no-connection test
Neo4jGraphRepository(neo4j_connection=None) falls back to the
Neo4jConnection singleton, which requires NEO4J_* env vars — the test
only passed locally thanks to .env. Pass a mock instead; the test
nulls _connection right after anyway.
2026-06-05 15:17:37 +02:00
dextmorgn
71e3a68830 fix(core): declare pyyaml dependency
flowsint_core imports yaml (template_generator_service, yaml_loader)
but never declared it — present locally only as a leftover in the
shared venv. CI's clean resolve exposed the missing declaration.
2026-06-05 15:10:18 +02:00
dextmorgn
a24551ed55 ci: provide REDIS_URL, also import-time required by flowsint_core 2026-06-05 14:43:37 +02:00
dextmorgn
17e36c9162 ci: provide AUTH_SECRET required at flowsint_core import 2026-06-05 14:32:12 +02:00
dextmorgn
07584ebc4a ci: run python tests on pull requests and main
Single entrypoint: CI calls `make test`, same as local development,
so the two can't drift. uv handles Python (.python-version) and
dependency sync; cache keyed on uv.lock.

Also add flowsint-api to `make test` — its suite existed but was
never wired in.
2026-06-05 14:24:34 +02:00
dextmorgn
db9f50d914 Merge pull request #175 from reconurge/fix/prebuilt-images-compose
fix(deploy): serve prod from pre-built images
2026-06-05 14:11:29 +02:00
dextmorgn
82925bec38 fix(deploy): serve prod from pre-built images
Frontend baked VITE_API_URL into the bundle at build time, so
published images pointed at whatever URL CI had — "fail to fetch"
for every other deployment. Bundle now defaults to same-origin
relative URLs and the frontend nginx proxies /api/ to the api
service over the Docker network; CORS becomes irrelevant.

- docker-compose.prod.yml: use ghcr images (FLOWSINT_VERSION,
  default latest), drop build sections; bind postgres/redis/neo4j/
  api ports to 127.0.0.1 — only 5173 faces the network
- nginx: /api/ reverse proxy, request-time DNS resolution, SSE
  support (proxy_buffering off)
- Makefile: prod pulls instead of building; remove dead deploy
  targets referencing nonexistent docker-compose.deploy.yml
- README: network/server deployment section (secrets checklist,
  version pinning, TLS via reverse proxy)
2026-06-05 14:04:16 +02:00
dextmorgn
f38d5d500d Merge pull request #174 from reconurge/chore/security-dependency-upgrades
fix(deps): upgrade Python dependencies to patch 27 known vulnerabilities
2026-06-05 13:21:03 +02:00
dextmorgn
9fcc60ca63 fix(deps): upgrade Python dependencies to patch 27 known vulnerabilities
pip-audit reported 27 vulnerabilities across 12 packages, including
cryptography PYSEC-2026-36 (CVSS 9.8) used by the vault, a starlette
Host-header path bypass, and python-multipart arbitrary file write.

- cryptography 45.0.7 -> 46.0.7 (pin was capped at <46)
- fastapi 0.115 -> 0.136, pulling starlette 0.46 -> 1.2
- python-multipart 0.0.20 -> 0.0.32
- pyjwt, urllib3, aiohttp, lxml, idna, mako and transitive bumps
- dev tools: pytest 8 -> 9 (requires pytest-httpx >=0.36), black 25 -> 26

All four module test suites pass (458 tests). pip-audit now reports
no known vulnerabilities.
2026-06-05 08:57:19 +02:00
dextmorgn
5cdfe727b4 Merge pull request #172 from melihdaskiran/fix/zip-and-vault-bugs
Fix zip alignment mismatch, phone validation return value, and vault …
2026-06-05 08:24:39 +02:00
Melih Daskiran
25eb3a0400 Fix zip alignment mismatch, phone validation return value, and vault owner_id type matching 2026-06-04 21:46:44 +03:00
dextmorgn
259019f23c fix(build): force LF on container files via .gitattributes
Windows checkouts with core.autocrlf=true convert entrypoint.sh to
CRLF, which Docker copies into the image. The shebang then reads
#!/bin/sh\r and exec fails with 'no such file or directory'.
Also covers .env.example (trailing \r in env values) and Makefile.
2026-06-04 18:23:53 +02:00
dextmorgn
7c082d733e docs(readme): make Windows install commands cmd-compatible
PowerShell foreach/Test-Path loop fails when pasted into cmd.exe.
Plain copy commands work in both shells.
2026-06-04 18:05:19 +02:00
dextmorgn
de93ae3059 docs(readme): add Windows install path without Make
Windows users hit failures with make-based install. Replicate
check-env + build + up targets as plain PowerShell/docker compose
commands for prod and dev.
2026-06-04 18:00:35 +02:00
dextmorgn
c6eaf08b66 Merge pull request #163 from Mubashirrrr/fix/json-import-apostrophe-corruption
fix(import): preserve apostrophes in valid JSON imports
2026-06-03 20:34:42 +02:00
dextmorgn
f380674955 Merge pull request #165 from reconurge/fix/sse-auth-header
fix(sse): auth header
2026-06-03 13:42:17 +02:00
Mubashir Rahim
1fdbdbd094 fix(import): preserve apostrophes in valid JSON imports
parse_json unconditionally replaced every single quote with a double
quote (`.replace("'", '"')`) before calling json.loads. This corrupted
any valid JSON string value containing an apostrophe, e.g. a person's
surname "Sarah O'Brien" became "Sarah O"Brien", causing the parse to
fail with "Invalid JSON" and the whole import to be rejected.

Parse strict JSON first so well-formed payloads (including apostrophes
in string values) import correctly, and only fall back to the lenient
single-quote replacement for Python-dict-style payloads that are not
valid JSON on their own.

Adds regression tests covering an apostrophe-bearing JSON value and the
retained single-quoted fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 14:34:22 +05:00
dextmorgn
b317e08e57 feat(api): authenticate SSE via header, drop query-param auth and dead route 2026-06-03 11:28:41 +02:00
dextmorgn
9db1e675d6 test(api): SSE auth-gate and endpoint-removal tests (red) 2026-06-03 11:22:58 +02:00
dextmorgn
4a75d6a924 refactor(app): stream graph status via connectSSE (token out of URL) 2026-06-03 11:08:11 +02:00
dextmorgn
d7f21e9384 refactor(app): stream logs via connectSSE (token out of URL) 2026-06-03 10:56:32 +02:00
dextmorgn
0b60c81361 Merge pull request #155 from Ghraven/fix-investigation-aware-timestamp
fix: use aware timestamp for investigation updates
2026-06-03 09:30:35 +02:00
dextmorgn
6e5f26aaf0 feat(app): add connectSSE helper with header auth and backoff 2026-06-03 09:28:49 +02:00
Ghraven
60cfa16b8d fix: use aware timestamp for investigation updates 2026-06-03 08:37:45 +08:00
dextmorgn
078ac77eee build(app): add fetch-event-source + vitest test runner 2026-06-02 22:45:36 +02:00
dextmorgn
ed8bfc5248 Merge pull request #149 from Ghraven/fix/timezone-aware-service-timestamps
Use timezone-aware UTC timestamps in core services
2026-06-02 21:11:45 +02:00
dextmorgn
438bbe18c4 Merge pull request #150 from Ghraven/fix-social-enricher-output-encoding
fix: read social enricher outputs as utf-8
2026-06-02 21:10:12 +02:00
Ghraven
4e14df6c61 fix: read social enricher outputs as utf-8 2026-06-01 00:32:48 +08:00
dextmorgn
249374e6d7 chore: update .gitignore v1.2.9 2026-05-31 14:53:34 +02:00
dextmorgn
a15ee54af0 chore: update skill name 2026-05-31 14:52:37 +02:00
Ghraven
b707961e68 fix: use timezone-aware service timestamps 2026-05-31 02:35:11 +08:00
dextmorgn
cc829704e5 Merge pull request #148 from reconurge/fix/phase1-critical
fix: security issues
2026-05-30 19:47:55 +02:00
dextmorgn
e8ca7b39bf fix(types): apply extra=allow via Pydantic v2 model_config
FlowsintType declared an inner `class ConfigDict: extra = "allow"`, which
Pydantic v2 treats as an ordinary nested class and ignores entirely. The
base therefore fell back to the default extra="ignore", silently dropping
user-supplied extra properties on every entity type. Replaced with
model_config = ConfigDict(extra="allow"). Extra keys are now retained.

flowsint-types (44), flowsint-enrichers (12) and flowsint-core (390) suites pass.
2026-05-30 19:03:30 +02:00
dextmorgn
1d409d7929 fix(scan): persist results to existing details column, errors to error column
The Scan model exposes details (JSON) and error (Text) columns, but the
enricher/flow tasks assigned to scan.results — a non-mapped attribute.
SQLAlchemy accepted the assignment silently, so commit() never persisted
it and the data was lost. Success paths now write scan.details; failure
paths write scan.error. ScanCreate.results renamed to details to match.

Scan repository tests pass.
2026-05-30 19:02:14 +02:00
dextmorgn
ee06d6cf3e fix(api): restrict CORS origins via ALLOWED_ORIGINS env
allow_origins was ["*"] together with allow_credentials=True, which is
both insecure and rejected by browsers for credentialed requests. Read a
comma-separated ALLOWED_ORIGINS env var instead, defaulting to the Vite
dev origin http://localhost:5173. Documented in .env.example.
2026-05-30 18:49:44 +02:00
dextmorgn
dd00bae9ce fix(api): run production container as non-root flowsint user
The production stage created the flowsint user and chowned all app
files to it, but the USER directive was commented out, so the API
ran as root. Uncommented it. Verified: image builds, container reports
whoami=flowsint and /health returns 200.
2026-05-30 18:49:43 +02:00
dextmorgn
c9d594a8e7 fix(api): load AUTH_SECRET from validated core.auth source
deps.py read AUTH_SECRET via os.getenv() before load_dotenv() ran,
yielding None when the secret was only present in .env. Import the
already-validated AUTH_SECRET from core.auth (loads dotenv first and
raises if unset) instead of re-reading it. Removes the dup + ordering bug.
2026-05-30 18:44:07 +02:00
dextmorgn
de4f4ea87f Update README with new image links
Added additional image links to the README.
2026-05-28 00:00:13 +02:00
dextmorgn
67ba8cfcc6 feat(chore): update SKILL.md 2026-05-21 11:09:28 +02:00
dextmorgn
8fbcf0c025 chore(skills): add flowsint-transform-builder claude skill
Project-scoped Claude Code skill guiding enricher and type creation.
Whitelists .claude/skills/flowsint-transform-builder/ in .gitignore so
the skill ships with the repo while keeping the rest of .claude/ ignored.
2026-05-15 18:55:56 +02:00
dextmorgn
d431ef4fae docs: sync with flowsint-types and flowsint-enrichers (v1.2.8)
- available-enrichers.mdx: add 17 missing enrichers, drop phantom entries
- managing-enrichers.mdx: fix stale Troubleshooting referencing EnricherRegistry.register()
- managing-types.mdx: note 8 registered-but-uncategorized types
- bump all frontmatter to version 1.2.8 / 2026-05-15
2026-05-15 18:55:25 +02:00
dextmorgn
5be480d4f5 Merge pull request #138 from z3rodaycve/feature/collection-enrichers
Release of my collection of Flowsint enrichers
2026-05-10 17:58:55 +02:00
dextmorgn
431d3997f4 Merge pull request #142 from harmsolo13/fix/celery-healthcheck
fix: replace celery container's inherited curl-based healthcheck
2026-04-29 11:35:17 +02:00
harmsolo13
d4dbe8d0fe fix: replace celery container's inherited curl-based healthcheck
The celery container builds from flowsint-api/Dockerfile which carries
a HEALTHCHECK directive that does `curl -f http://localhost:5001/health`.
The API container has an HTTP server on 5001 — celery doesn't, it's a
worker. So the inherited healthcheck always fails and `docker ps` shows
celery as (unhealthy) even when the worker is actively processing jobs.

This is cosmetic noise today but bites in two real ways: (1) restart
policies that key off health won't re-up celery on a real failure
because Docker can't tell good unhealthy from bad unhealthy, (2) any
service that adds `depends_on: celery: condition: service_healthy`
will refuse to start.

Fix: add a service-level healthcheck on celery in both compose files
(prod and dev) that uses celery's own `inspect ping` primitive
against the worker's broker. Compose-level healthcheck overrides the
Dockerfile-level one, so no Dockerfile change needed.

Smoke-tested locally: container goes from (unhealthy) to (healthy)
within ~30s of restart with no other changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 18:13:26 +09:30
z3rodaycve
f79403d361 Fixed to_ssl enricher to use httpX Docker Tool 2026-04-24 19:49:20 +02:00