A CSV saved as UTF-16 (Excel's "Unicode Text") was read as UTF-8, leaving
a NUL between every character. MySQL and SQLite store that, Postgres text
columns reject it with SQLSTATE 22021 and the whole import fails.
The file is now decoded to UTF-8 when read: a UTF-16 BOM switches to
UTF-16 decoding, remaining NUL bytes are dropped and invalid UTF-8 becomes
U+FFFD, matching what encoding/json does for the JSON importers.
Fixes API-CLOUD-4G
Detect, preview and migrate each repeated the size checks and file read.
Extracting readCSVFile gives the raw bytes a single entry point, and
projectsFromCSV makes the migrate conversion testable without a database.
The config was unmarshalled up front purely to produce a 400, then parsed
again by SetOptions, and a third time in the background job. Drop the
throwaway parse and give SetOptions a domain error that maps to 400 on
its own.
A file import ran inside the upload request. A big export takes minutes,
far longer than a reverse proxy holds a request open, so the client got a
502 while the import kept running and kept the user's migration slot, and
there was no way to cancel or retry it.
The upload request now only validates what is cheap to validate, claims
the slot, spools the upload into a per-process temp directory and queues a
FileMigrationRequestedEvent. FileMigrationListener runs the import through
the lifecycle it now shares with the OAuth migrations: stale event skip,
panic recovery, heartbeat, outcome and notifications. The vikunja-file
migrator rejects a non-zip or structurally broken export before queueing;
ticktick, wekan and csv would have to parse the whole upload, so they fail
in the job and report through the status and failure mail. The csv
migrator receives its column mapping as options carried with the event.
DetectCSVStructure, PreviewImport, and MigrateWithConfig each did
make([]byte, size) with size taken directly from the caller (an
uploaded file's declared size) with no upper bound, letting a
malicious or corrupt size value force an arbitrarily large
allocation. Cap it to the server's existing configured max upload
size before allocating.
Pull the StartMigration -> Migrate -> FinishMigration orchestration out of
the v1 echo handlers into handler.RunFileMigration and csv.RunMigration so
the v2 API can reuse the exact same business logic. v1 is refactored onto
them and stays byte-identical on the wire.
Also tag the CSV detect/preview/config DTOs with doc:/enum: so they carry
descriptions in the v2 OpenAPI schema (ignored by v1 swaggo/xorm).
Allow users to skip the first N data rows when importing CSV files.
This is useful when the CSV contains metadata rows before the actual
task data begins. Adds skip_rows to ImportConfig (backend) and a
number input in the parsing options UI (frontend).
Add a new CSV migration module that allows users to import tasks from
any CSV file with custom column mapping and parsing options.
Backend changes:
- New CSV migrator module with detection, preview, and import endpoints
- Auto-detection of delimiter, quote character, and date format
- Suggested column mappings based on column name patterns
- Transactional import using InsertFromStructure
Frontend changes:
- New CSV migration UI with two-step flow (upload -> mapping -> import)
- Column mapping selectors for all task attributes
- Live preview showing first 5 tasks with current mapping
- Parsing option controls for delimiter and date format
The CSV migrator creates a parent "Imported from CSV" project with
child projects based on the project column if provided, or a default
"Tasks" project for tasks without a specified project.
kolaente
Mauandkolaente
Claudeandkolaente