A CSV saved as UTF-16 (Excel's "Unicode Text") was read as UTF-8, leaving
a NUL between every character. MySQL and SQLite store that, Postgres text
columns reject it with SQLSTATE 22021 and the whole import fails.
The file is now decoded to UTF-8 when read: a UTF-16 BOM switches to
UTF-16 decoding, remaining NUL bytes are dropped and invalid UTF-8 becomes
U+FFFD, matching what encoding/json does for the JSON importers.
Fixes API-CLOUD-4G
Detect, preview and migrate each repeated the size checks and file read.
Extracting readCSVFile gives the raw bytes a single entry point, and
projectsFromCSV makes the migrate conversion testable without a database.
The files row is inserted before the blob is written, and the local backend creates the file before copying into it, so a failed copy left a partial blob behind once the row was rolled back.
The vikunja-file import checks the user's stored files against
migration.vikunjafile.maxuserstorage. The upload is now one of those
files, so an import counted its own zip against the quota. Exclude files
referenced as a migration upload.
The upload was spooled to the receiving instance's temp dir and only its
name travelled with the event. That holds while events stay in-process,
but it ties the job to the instance that accepted the upload.
Store it through the files package on the configured backend and record
its id on the migration status, so the event carries only the status id
and any instance can run the import. The job copies it to a local temp
file first: the zip importer needs random access, which an S3 object body
cannot give.
The upload is removed once the import is over. An hourly cron deletes
leftovers from events lost on restart or instances that died mid-import.
The config was unmarshalled up front purely to produce a 400, then parsed
again by SetOptions, and a third time in the background job. Drop the
throwaway parse and give SetOptions a domain error that maps to 400 on
its own.
A file import ran inside the upload request. A big export takes minutes,
far longer than a reverse proxy holds a request open, so the client got a
502 while the import kept running and kept the user's migration slot, and
there was no way to cancel or retry it.
The upload request now only validates what is cheap to validate, claims
the slot, spools the upload into a per-process temp directory and queues a
FileMigrationRequestedEvent. FileMigrationListener runs the import through
the lifecycle it now shares with the OAuth migrations: stale event skip,
panic recovery, heartbeat, outcome and notifications. The vikunja-file
migrator rejects a non-zip or structurally broken export before queueing;
ticktick, wekan and csv would have to parse the whole upload, so they fail
in the job and report through the status and failure mail. The csv
migrator receives its column mapping as options carried with the event.
Split opening the zip and indexing its entries out of Migrate, so every
structural check that doesn't need entry contents runs in one place before
anything is imported. The missing VERSION check moves into that scan.
A finished migration_status row looked the same whether the import worked
or failed, so the client could only guess. Rows now carry error_kind, a
closed set the client translates (reported, interrupted, credentials,
queue, upload, detail), and error_message, set only for "detail": the
error caused by the user's own data, the same text the failure mail sends.
A failure we report to Sentry stores just the kind, so server internals
never reach the status.
The migration was already finished and its claim released when the done
notification failed, yet the error was returned and the listener then sent
a failure mail and overwrote the finished row. Log it instead. The failure
mail's own log line also printed the migration error instead of the
notification error.
The claim timeout was measured from started_at, so a long import either
kept a dead claim for a day (the old 24h default) or had its slot taken
over while still running. A running migration now refreshes heartbeat_at
at a tenth of the timeout, clamped to 1s..30s, and a claim only counts as
stale once its last beat is older than the timeout. Rows that never beat
fall back to started_at. The default timeout drops to 5m accordingly.
Also bind the stale cutoff in the configured timezone: xorm writes times in
the engine's zone but binds query args as-is, which broke the comparison on
SQLite.
A second import sent while the first claim was still being written could
fail the insert with whatever the driver reports under contention, which
surfaced as a 500. The claim now reads the running row before writing so a
losing request stays read-only, re-reads the live claim whenever the insert
fails, retries lock collisions a few times, and probes for a stale claim
before taking a write lock to release it.
Top-level and related tasks are collected first and created in one batch
so preset indexes survive, related references resolve to the created
struct instead of a mutable id lookup, untitled relation stubs are skipped,
and positions and buckets remap through one old-id map.
The api error handler, the poison queue logger and the migration listener all
captured through the same frame, so API-OSS-2P collected six unrelated MySQL
errors under one title. Set a derived fingerprint and an error.type tag on the
scope before capturing.
Unsafe paths, unusable file ids, a missing or unparseable VERSION and
unreadable data/filter json all describe a broken upload, not a server
fault, but they were raw fmt.Errorf values and surfaced as a 500. They now
return ErrInvalidImportFile; an export from a version we can no longer
read gets its own ErrImportFromUnsupportedVersion (14013) so clients can
tell the two apart. The http messages stay generic and never echo a path
from the archive.
ErrCodeNotAZipFile and ErrCodeFileIsEmpty shared 14001 and 14002 with
ErrCodeAPITokenInvalid and ErrCodeInvalidAPITokenPermission, so clients
mapping codes to messages rendered the api token text for a failed import.
The api token codes are the older ones, so the migration errors move to
14011 and 14012.
BREAKING CHANGE: a failed import now returns 14011 instead of 14001 and
14012 instead of 14002.
Errors like planka's ErrInvalidCredentials or ErrNoPlankaAtURL surface
inside the migration listener and map to a 4xx, so they are the user's
problem to fix, not ours.
encoding/json reports a document that ends mid-value as io.ErrUnexpectedEOF
rather than a *json.SyntaxError, so those still reached the error handler as a
500.
A WeKan JSON export saved with a UTF-8 BOM made encoding/json return a raw
*json.SyntaxError from the migrator, which the shared error handler turned into
a 500 and forwarded to Sentry. Map decode failures of the uploaded file to a
400 in RunFileMigration (shared by v1 and v2), and skip a leading BOM so an
otherwise valid export imports instead of being rejected.
Fixes API-CLOUD-4B
A 4xx from the migration source (disabled mailbox, expired token, deleted
item) is not a bug on our side, but every one of them was captured as an
exception. Those users now get the failure notification with the upstream
message instead of the "we have been notified" one.
Non-2xx responses from the service we migrate from were plain fmt.Errorf
values, so callers could not tell an upstream problem apart from a bug.
ErrUpstreamRequestFailed carries the migrator, status code and a truncated
body and is now returned by the shared http helpers, microsoft-todo,
todoist and planka.
Preflight over the declared uncompressed sizes (overflow-safe), a file
count cap and a per-user storage quota bound the import before
anything is read; the actual decompressed bytes are still counted
against the same budget while reading. Attachments and backgrounds
now stream through a lazy provider: one zip entry is decompressed
into a private temp file at a time. A failed import deletes the file
blobs it wrote, which were orphaned for every migrator before.
Per-job budget: 128 MiB aggregate response bytes (counted while
decoding, not from Content-Length), 64 MiB retained attachments,
50,000 decoded entities and 2,000 outbound request attempts counted
at the transport level so retries cannot multiply work. The
per-response cap drops to 4 MiB. Budget exhaustion aborts the import
with a typed error; ordinary 404/timeout attachment failures keep the
log-and-skip.
Credentials migrators are verified synchronously before the async migration is
queued (CredentialsChecker). Routes live on v2 only; the listener learns the
migrator via a route-free RegisterMigratorForEvents, which the oauth v2
registrar uses as well. Shared status/migrate registration moved to
migration_shared.go.
Everything lands under a "Migrated from Planka" root project; projects with
several boards become a parent with one child per board. Lists become kanban
buckets (trash skipped), closed/archived cards are done, labels keep their
Planka colours, checklists, custom fields and link attachments are rendered
into the description, comments are prefixed with the author unless it is the
importing user. No assignees, memberships or other users' data.
Pages archived cards and comments (Planka requires both cursor fields, pages
until empty with a hard cap, partial results are kept), reads base custom
field definitions from the projects payload and rejects Planka v1 payloads.
Authenticates with an API key, a JWT or username + password (Planka checks
Bearer before X-Api-Key, so only one header is ever sent; downloads use the
accessToken cookie). The synchronous credential check has one 15s deadline and
no retries. Redirects to another host or to plain http are refused for the
api, followed without credentials for attachment downloads. Response bodies
are capped, url userinfo is stripped, pending login steps (totp, terms) and
non-planka endpoints are reported as distinct client errors (142xx).
Migrators that talk to a user-supplied host need a client with a redirect
policy, a cap on response bodies and no retries on permanent errors
(utils.ErrDoNotRetry). Diagnostic body reads in the shared helpers are capped.
Creating a done task puts it in the view's default done bucket; moving it into
the imported bucket flipped it back to open. The imported state (incl. done_at)
is restored in bulk after the task loop.
Vikunja-file exports carry task assignees with user ids from the source
instance. On import those ids were looked up on the target instance and
permission-checked, which fails with "User does not have access to the
project" whenever the id belongs to someone else (e.g. importing a
self-hosted export into Vikunja Cloud). Since ba980b1b8 that error aborts
the whole import.
Match assignees against the importing user by email, then username, and
drop everyone else – foreign user ids have no meaning on this instance.
Fixes#3476
Todoist returns opaque identifiers instead of urls in file_url for
attachments it does not host itself (mail attachments for example).
Passing those to the http client failed with "unsupported protocol
scheme" and aborted the entire migration.
Skip attachments without an http(s) url and log-and-continue when a
single download fails instead of failing the whole migration.
Non-empty-title errors from task creation fell through silently and the
loop kept using a task with ID 0, attaching relations and labels to id
0. Return the error instead; empty titles keep being skipped.
DetectCSVStructure, PreviewImport, and MigrateWithConfig each did
make([]byte, size) with size taken directly from the caller (an
uploaded file's declared size) with no upper bound, letting a
malicious or corrupt size value force an arbitrarily large
allocation. Cap it to the server's existing configured max upload
size before allocating.
sortParentsBeforeChildren marked a task as placed only after recursing
into its parent, so a parentId cycle in an uploaded TickTick export made
place() recurse forever. That is a Go stack overflow, a runtime fatal
error the recover middleware cannot catch, so a two-line CSV from any
authenticated user took down the whole process.
Track a tri-state per task and mark it before recursing, which breaks
the cycle. Acyclic input is unaffected.
Tasks imported without a position are inserted in front of the lowest one by
halving it. That reaches the minimum spacing every few dozen inserts, which
recalculates every position in the view and makes a large import O(n²) - a
10k task TickTick export with an empty Order column never finished.
Numbering the tasks up front skips the halving entirely and keeps the order
the export was written in.
Closes#3297
kolaente
Wouter BlancquaertandGitHub
Mauandkolaente
