* [AI] Detect CSV file encoding and add a per-account encoding selector
Read CSV/TSV files as raw bytes and decode them based on the detected
encoding instead of always assuming UTF-8: the byte order mark, BOM-less
UTF-16 (NUL byte analysis), strict UTF-8 validation, and a windows-1252
fallback that also decodes ISO-8859-1 content. A mostly-valid UTF-8 file
with a few corrupted bytes still decodes as UTF-8 with replacement
characters instead of falling back to windows-1252.
Add an encoding selector to the CSV import options, persisted per
account like the delimiter, for files in encodings that cannot be
detected automatically (e.g. windows-1250, ISO-8859-2).
Adds fixtures and tests for UTF-16 (with and without BOM), UTF-8 with
BOM, corrupted UTF-8, NUL-padded UTF-8, and windows-1252 content.
Fixes#6327
* Update VRT screenshots
Auto-generated by VRT workflow
PR: #8924
* Update VRT screenshots
Auto-generated by VRT workflow
PR: #8924
* [AI] Address review: require dominant NUL parity and translate encoding labels
Require one parity to clearly dominate (>= 90% of NUL bytes) before
detecting BOM-less UTF-16, so a UTF-8 file with dense contiguous NUL
padding (even split across parities) is no longer misdetected as
UTF-16. Adds a fixture with padding above the 10% density threshold.
Wrap the encoding selector labels in t() per the repository's
translated user-facing text requirement.
* [AI] Drop iso-8859-1 from the CSV encoding selector
TextDecoder resolves the iso-8859-1 label to the windows-1252 decoder
per the WHATWG encoding spec, so the option was redundant and could
mislead users into expecting true ISO-8859-1 decoding. windows-1252
already covers ISO-8859-1 content; keep the alias accepted in
decodeCsvBytes with a clarifying comment and an alias test.
* [AI] Make the iso-8859-1 alias test distinguish Windows-1252
The alias test now uses a fixture containing the 0x80 byte, which
windows-1252 decodes as the euro sign while a true ISO-8859-1 decoder
would produce a C1 control character, so the assertion actually
verifies the windows-1252 alias semantics.
* [AI] Simplify CSV encoding detection to BOM and explicit selection
Follow the review suggestion: the byte order mark selects UTF-16
LE/BE and everything else decodes as UTF-8; other encodings are
handled by the manual per-account selector this PR adds. Drop the
speculative BOM-less UTF-16, windows-1252 fallback, and corrupted
UTF-8 heuristics along with their fixtures and tests.
---------
Co-authored-by: François Lafleur <francois_lafleur@yahoo.ca>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Updates to the latest version of electron and moves the backend-frontend
communication from node-ipc to websockets. This resolves the previous
roadblock regarding `nodeIntegration` .
Done
- Remove node-ipc in favour of websockets.
- Move file copying out of `preload.js` to avoid importing module `fs`
there
- Bump all electron pacakge versions to the latest
- Added new package for finding open ports as node-ipc is gone
- Tweaked webpack config for above changes
Partially fixes#468
Questions/ Pending:
- Literally every single test fails for me, presumably some issue with
my setup/environment.
- The websocket communication is not using TLS. I'm not sure how to
enable this, or if we even need to as its all local.
- Still need to create the CI for building/deploying but I'm not sure
where start in this regard as i have no exp with it. Presumably we will
need to point the electron auto-updater to the github releases url's. If
people are happy with this PR I will look at adding the CI before its
merged.
- In dev mode only, I have disabled TLS security becuase my docker
container's cert is not signed. I _assume_ this will be true for other
people who spin up the server on thier own hardware. Perhaps I just need
to change my cert to one from letsencrypt or something...
Notes.
I have not touched javascript in eons so my apologies if the commit
trail is a bit fragmented. I tried to keep them fairly contained and
then there is a slightly gnarly final commit fixing all the linter
issues... Please let me know if you want me to squash some commits etc.
I initially tried to move this to web workers the same way the web app
does it but this was unsuccessful. I have found no way to spin up a
worker in one place (frontend/backend) and then pass this worker to the
other. The electron ipc channels don't allow you to directly pass
objects such as workers, everything is cloned/serialised. Passing a port
number so the other end can spin up its own socket works fine.
---------
Co-authored-by: Shazib Hussain <contact@shazib.com>
Co-authored-by: Jed Fox <git@jedfox.com>
Julian Dominguez-SchatzandGitHub
Joshua GranickandGitHub
Matiss Janis AboltinsandGitHub
Shazib HussainandGitHub
winklevos
James Long