[PR #163] [MERGED] fix(import): preserve apostrophes in valid JSON imports #3851

New Issue

GiteaMirror · 2026-07-14T19:50:24-05:00

GiteaMirror commented

2026-07-14 19:50:24 -05:00

📋 Pull Request Information

Original PR: https://github.com/reconurge/flowsint/pull/163
Author: @Mubashirrrr
Created: 6/3/2026
Status: ✅ Merged
Merged: 6/3/2026
Merged by: @dextmorgn

Base: main ← Head: fix/json-import-apostrophe-corruption

📝 Commits (1)

1fdbdbd fix(import): preserve apostrophes in valid JSON imports

📊 Changes

2 files changed (+46 additions, -2 deletions)

View changed files

📝 flowsint-core/src/flowsint_core/imports/json/parse_json.py (+9 -2)
📝 flowsint-core/tests/import/json/test_json_import.py (+37 -0)

📄 Description

Bug

parse_json (flowsint-core/src/flowsint_core/imports/json/parse_json.py) unconditionally rewrote the file before parsing:

my_bytes_value = file_bytes.decode().replace("'", '"')
graph = json.loads(my_bytes_value)

Replacing every single quote with a double quote corrupts any valid JSON string value that contains an apostrophe — extremely common in OSINT data (names, org names, free-text). For example a node label "Sarah O'Brien" becomes "Sarah O"Brien", which is no longer valid JSON, so json.loads raises and the whole import is rejected with Invalid JSON: Expecting ',' delimiter ....

Reproduction

from flowsint_core.imports.json.parse_json import parse_json

data = b'''{
  "nodes": [
    {"id": "1", "label": "Sarah O'Brien", "type": "individual"},
    {"id": "2", "label": "Bob", "type": "individual"}
  ],
  "edges": [{"source": "1", "target": "2", "label": "KNOWS"}]
}'''

parse_json(data, max_preview_rows=100)
# Before: Exception: Invalid JSON: Expecting ',' delimiter: line ... (char 41)
# After:  parses OK; the "Sarah O'Brien" label is preserved

This is well-formed JSON — it should always import.

Fix

Parse strict JSON first (handles all valid JSON, apostrophes included). Only if that fails, fall back to the original lenient single-quote→double-quote replacement, preserving support for Python-dict-style payloads that aren't valid JSON on their own. Minimal, surgical change confined to the decode/parse step.

Regression tests

Added to tests/import/json/test_json_import.py:

test_json_import_preserves_apostrophes_in_string_values — fails before this change (raises Invalid JSON), passes after.
test_json_import_still_accepts_single_quoted_payload — confirms the single-quoted fallback still works.

Full tests/import/ suite: 46 passed (44 pre-existing + 2 new) on Python 3.12.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/reconurge/flowsint/pull/163 **Author:** [@Mubashirrrr](https://github.com/Mubashirrrr) **Created:** 6/3/2026 **Status:** ✅ Merged **Merged:** 6/3/2026 **Merged by:** [@dextmorgn](https://github.com/dextmorgn) **Base:** `main` ← **Head:** `fix/json-import-apostrophe-corruption` --- ### 📝 Commits (1) - [`1fdbdbd`](https://github.com/reconurge/flowsint/commit/1fdbdbd094c78b239dd2795a4f34cbff6951adc1) fix(import): preserve apostrophes in valid JSON imports ### 📊 Changes **2 files changed** (+46 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `flowsint-core/src/flowsint_core/imports/json/parse_json.py` (+9 -2) 📝 `flowsint-core/tests/import/json/test_json_import.py` (+37 -0) </details> ### 📄 Description ## Bug `parse_json` (`flowsint-core/src/flowsint_core/imports/json/parse_json.py`) unconditionally rewrote the file before parsing: ```python my_bytes_value = file_bytes.decode().replace("'", '"') graph = json.loads(my_bytes_value) ``` Replacing **every** single quote with a double quote corrupts any *valid* JSON string value that contains an apostrophe — extremely common in OSINT data (names, org names, free-text). For example a node label `"Sarah O'Brien"` becomes `"Sarah O"Brien"`, which is no longer valid JSON, so `json.loads` raises and the whole import is rejected with `Invalid JSON: Expecting ',' delimiter ...`. ## Reproduction ```python from flowsint_core.imports.json.parse_json import parse_json data = b'''{ "nodes": [ {"id": "1", "label": "Sarah O'Brien", "type": "individual"}, {"id": "2", "label": "Bob", "type": "individual"} ], "edges": [{"source": "1", "target": "2", "label": "KNOWS"}] }''' parse_json(data, max_preview_rows=100) # Before: Exception: Invalid JSON: Expecting ',' delimiter: line ... (char 41) # After: parses OK; the "Sarah O'Brien" label is preserved ``` This is well-formed JSON — it should always import. ## Fix Parse strict JSON first (handles all valid JSON, apostrophes included). Only if that fails, fall back to the original lenient single-quote→double-quote replacement, preserving support for Python-dict-style payloads that aren't valid JSON on their own. Minimal, surgical change confined to the decode/parse step. ## Regression tests Added to `tests/import/json/test_json_import.py`: - `test_json_import_preserves_apostrophes_in_string_values` — **fails before** this change (raises `Invalid JSON`), **passes after**. - `test_json_import_still_accepts_single_quoted_payload` — confirms the single-quoted fallback still works. Full `tests/import/` suite: **46 passed** (44 pre-existing + 2 new) on Python 3.12. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-07-14 19:50:24 -05:00

GiteaMirror closed this issue

2026-07-14 19:50:25 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/flowsint#3851