kolaente f2d84ca6fc fix(migration): decode utf-16 and drop nul bytes in csv imports ...

A CSV saved as UTF-16 (Excel's "Unicode Text") was read as UTF-8, leaving
a NUL between every character. MySQL and SQLite store that, Postgres text
columns reject it with SQLSTATE 22021 and the whole import fails.

The file is now decoded to UTF-8 when read: a UTF-16 BOM switches to
UTF-16 decoding, remaining NUL bytes are dropped and invalid UTF-8 becomes
U+FFFD, matching what encoding/json does for the JSON importers.

Fixes API-CLOUD-4G

2026-10-13 21:45:53 +02:00

..

csv

fix(migration): decode utf-16 and drop nul bytes in csv imports

2026-10-13 21:45:53 +02:00

handler

feat(migration): keep queued import uploads in the file storage

2026-09-15 17:57:00 +02:00

microsoft-todo

feat(migration): typed error for failed upstream requests

2026-09-05 11:55:07 +02:00

planka

feat(migration): typed error for failed upstream requests

2026-09-05 11:55:07 +02:00

ticktick

refactor(migration): tidy the ticktick, wekan and csv read paths

2026-09-15 16:59:37 +02:00

todoist

fix(migration): preserve todoist task descriptions (#3819)

2026-09-08 08:23:34 +00:00

trello

fix(migration): route Trello API requests through the outgoing client

2026-09-30 18:26:56 +02:00

vikunja-file

fix(migration): keep a queued import upload out of the storage quota

2026-09-15 17:57:00 +02:00

wekan

refactor(migration): tidy the ticktick, wekan and csv read paths

2026-09-15 16:59:37 +02:00

create_from_structure_test.go

fix(import): create imported tasks in one batch with preserved indexes

2026-09-11 20:47:11 +02:00

create_from_structure.go

fix(migration): log migration start and completion at info level

2026-09-15 16:59:37 +02:00

db.go

…

errors.go

refactor(migration): parse the csv import config once per request

2026-09-15 16:59:37 +02:00

helpers_test.go

feat(migration): client-aware get and size-limited download helpers, bounded json decode

2026-08-20 08:45:42 +02:00

helpers.go

feat(migration): typed error for failed upstream requests

2026-09-05 11:55:07 +02:00

import_upload_test.go

feat(migration): keep queued import uploads in the file storage

2026-09-15 17:57:00 +02:00

import_upload.go

fix(migration): remove the partial blob of a failed import upload write

2026-09-15 17:57:00 +02:00

main_test.go

fix(migration): serialize imports per user

2026-08-31 18:37:03 +02:00

migration_status_test.go

feat(migration): record whether a migration succeeded

2026-09-15 16:59:37 +02:00

migration_status.go

feat(migration): keep queued import uploads in the file storage

2026-09-15 17:57:00 +02:00

migrator.go

fix(migration): run file imports in the background

2026-09-15 16:59:37 +02:00

testimage.jpg

…