The user data export embedded every related task in full, repeating its
description once per relation. Related tasks are now exported as id and
project id only, since every related task is part of the export anyway.
The importer resolves such references across all imported projects and
creates relations once every project exists, so relations between
projects survive the round trip instead of duplicating the related task.
Exports with the full related task shape import as before.
- Reject a config that maps a column more than once, already when the
import is started or previewed.
- Import repeated labels of a row once and reject rows with more than
MaxLabelsPerTask (100) labels.
- Count items while converting rows so a file over the import item limit
fails before the whole structure is built.
- Walk mappings sorted by column so a row only visits the cells it has.
Lists, cards, checklists and attachments are looked up by id while
converting a board, so a repeated id imported the whole group once per
copy. Keep the first entry of each id and stop the conversion as soon as
the import item limit is exceeded instead of after building everything.
Review finding: CreateTaskRelationsForImport relied on an unenforced
invariant that every related task came from the same import. It now takes
the imported batch, builds the relations itself and rejects related tasks
outside the batch.
Planka instances on a private network were rejected by the SSRF guard with
a generic "could not reach" error. Return a dedicated error pointing at
outgoingrequests.allownonroutableips instead.
Refs #4234
Address review decision: a background upload queued for decode capacity
held its transaction for up to 30s. The permission check now runs in a
short read session, the image is decoded and encoded without a session,
and a fresh write session re-checks permission and stores the result.
Address pr-swarm finding: the item cap ran only after data.json was fully
decoded into tasks, so a file of millions of bare objects still exhausted
memory.
Checklist, custom field and link attachment names were concatenated raw
into the description markdown, so brackets, parens or HTML in names broke
links or were dropped.
A CSV saved as UTF-16 (Excel's "Unicode Text") was read as UTF-8, leaving
a NUL between every character. MySQL and SQLite store that, Postgres text
columns reject it with SQLSTATE 22021 and the whole import fails.
The file is now decoded to UTF-8 when read: a UTF-16 BOM switches to
UTF-16 decoding, remaining NUL bytes are dropped and invalid UTF-8 becomes
U+FFFD, matching what encoding/json does for the JSON importers.
Fixes API-CLOUD-4G
Detect, preview and migrate each repeated the size checks and file read.
Extracting readCSVFile gives the raw bytes a single entry point, and
projectsFromCSV makes the migrate conversion testable without a database.
The files row is inserted before the blob is written, and the local backend creates the file before copying into it, so a failed copy left a partial blob behind once the row was rolled back.
The vikunja-file import checks the user's stored files against
migration.vikunjafile.maxuserstorage. The upload is now one of those
files, so an import counted its own zip against the quota. Exclude files
referenced as a migration upload.
The upload was spooled to the receiving instance's temp dir and only its
name travelled with the event. That holds while events stay in-process,
but it ties the job to the instance that accepted the upload.
Store it through the files package on the configured backend and record
its id on the migration status, so the event carries only the status id
and any instance can run the import. The job copies it to a local temp
file first: the zip importer needs random access, which an S3 object body
cannot give.
The upload is removed once the import is over. An hourly cron deletes
leftovers from events lost on restart or instances that died mid-import.
The config was unmarshalled up front purely to produce a 400, then parsed
again by SetOptions, and a third time in the background job. Drop the
throwaway parse and give SetOptions a domain error that maps to 400 on
its own.
A file import ran inside the upload request. A big export takes minutes,
far longer than a reverse proxy holds a request open, so the client got a
502 while the import kept running and kept the user's migration slot, and
there was no way to cancel or retry it.
The upload request now only validates what is cheap to validate, claims
the slot, spools the upload into a per-process temp directory and queues a
FileMigrationRequestedEvent. FileMigrationListener runs the import through
the lifecycle it now shares with the OAuth migrations: stale event skip,
panic recovery, heartbeat, outcome and notifications. The vikunja-file
migrator rejects a non-zip or structurally broken export before queueing;
ticktick, wekan and csv would have to parse the whole upload, so they fail
in the job and report through the status and failure mail. The csv
migrator receives its column mapping as options carried with the event.
Split opening the zip and indexing its entries out of Migrate, so every
structural check that doesn't need entry contents runs in one place before
anything is imported. The missing VERSION check moves into that scan.
A finished migration_status row looked the same whether the import worked
or failed, so the client could only guess. Rows now carry error_kind, a
closed set the client translates (reported, interrupted, credentials,
queue, upload, detail), and error_message, set only for "detail": the
error caused by the user's own data, the same text the failure mail sends.
A failure we report to Sentry stores just the kind, so server internals
never reach the status.
The migration was already finished and its claim released when the done
notification failed, yet the error was returned and the listener then sent
a failure mail and overwrote the finished row. Log it instead. The failure
mail's own log line also printed the migration error instead of the
notification error.
The claim timeout was measured from started_at, so a long import either
kept a dead claim for a day (the old 24h default) or had its slot taken
over while still running. A running migration now refreshes heartbeat_at
at a tenth of the timeout, clamped to 1s..30s, and a claim only counts as
stale once its last beat is older than the timeout. Rows that never beat
fall back to started_at. The default timeout drops to 5m accordingly.
Also bind the stale cutoff in the configured timezone: xorm writes times in
the engine's zone but binds query args as-is, which broke the comparison on
SQLite.
A second import sent while the first claim was still being written could
fail the insert with whatever the driver reports under contention, which
surfaced as a 500. The claim now reads the running row before writing so a
losing request stays read-only, re-reads the live claim whenever the insert
fails, retries lock collisions a few times, and probes for a stale claim
before taking a write lock to release it.
Top-level and related tasks are collected first and created in one batch
so preset indexes survive, related references resolve to the created
struct instead of a mutable id lookup, untitled relation stubs are skipped,
and positions and buckets remap through one old-id map.
The api error handler, the poison queue logger and the migration listener all
captured through the same frame, so API-OSS-2P collected six unrelated MySQL
errors under one title. Set a derived fingerprint and an error.type tag on the
scope before capturing.
Unsafe paths, unusable file ids, a missing or unparseable VERSION and
unreadable data/filter json all describe a broken upload, not a server
fault, but they were raw fmt.Errorf values and surfaced as a 500. They now
return ErrInvalidImportFile; an export from a version we can no longer
read gets its own ErrImportFromUnsupportedVersion (14013) so clients can
tell the two apart. The http messages stay generic and never echo a path
from the archive.
ErrCodeNotAZipFile and ErrCodeFileIsEmpty shared 14001 and 14002 with
ErrCodeAPITokenInvalid and ErrCodeInvalidAPITokenPermission, so clients
mapping codes to messages rendered the api token text for a failed import.
The api token codes are the older ones, so the migration errors move to
14011 and 14012.
BREAKING CHANGE: a failed import now returns 14011 instead of 14001 and
14012 instead of 14002.
kolaente
Wouter BlancquaertandGitHub