The upload was spooled to the receiving instance's temp dir and only its
name travelled with the event. That holds while events stay in-process,
but it ties the job to the instance that accepted the upload.
Store it through the files package on the configured backend and record
its id on the migration status, so the event carries only the status id
and any instance can run the import. The job copies it to a local temp
file first: the zip importer needs random access, which an S3 object body
cannot give.
The upload is removed once the import is over. An hourly cron deletes
leftovers from events lost on restart or instances that died mid-import.
A file import ran inside the upload request. A big export takes minutes,
far longer than a reverse proxy holds a request open, so the client got a
502 while the import kept running and kept the user's migration slot, and
there was no way to cancel or retry it.
The upload request now only validates what is cheap to validate, claims
the slot, spools the upload into a per-process temp directory and queues a
FileMigrationRequestedEvent. FileMigrationListener runs the import through
the lifecycle it now shares with the OAuth migrations: stale event skip,
panic recovery, heartbeat, outcome and notifications. The vikunja-file
migrator rejects a non-zip or structurally broken export before queueing;
ticktick, wekan and csv would have to parse the whole upload, so they fail
in the job and report through the status and failure mail. The csv
migrator receives its column mapping as options carried with the event.
A finished migration_status row looked the same whether the import worked
or failed, so the client could only guess. Rows now carry error_kind, a
closed set the client translates (reported, interrupted, credentials,
queue, upload, detail), and error_message, set only for "detail": the
error caused by the user's own data, the same text the failure mail sends.
A failure we report to Sentry stores just the kind, so server internals
never reach the status.
The migration was already finished and its claim released when the done
notification failed, yet the error was returned and the listener then sent
a failure mail and overwrote the finished row. Log it instead. The failure
mail's own log line also printed the migration error instead of the
notification error.
The claim timeout was measured from started_at, so a long import either
kept a dead claim for a day (the old 24h default) or had its slot taken
over while still running. A running migration now refreshes heartbeat_at
at a tenth of the timeout, clamped to 1s..30s, and a claim only counts as
stale once its last beat is older than the timeout. Rows that never beat
fall back to started_at. The default timeout drops to 5m accordingly.
Also bind the stale cutoff in the configured timezone: xorm writes times in
the engine's zone but binds query args as-is, which broke the comparison on
SQLite.
The api error handler, the poison queue logger and the migration listener all
captured through the same frame, so API-OSS-2P collected six unrelated MySQL
errors under one title. Set a derived fingerprint and an error.type tag on the
scope before capturing.
Errors like planka's ErrInvalidCredentials or ErrNoPlankaAtURL surface
inside the migration listener and map to a 4xx, so they are the user's
problem to fix, not ours.
encoding/json reports a document that ends mid-value as io.ErrUnexpectedEOF
rather than a *json.SyntaxError, so those still reached the error handler as a
500.
A WeKan JSON export saved with a UTF-8 BOM made encoding/json return a raw
*json.SyntaxError from the migrator, which the shared error handler turned into
a 500 and forwarded to Sentry. Map decode failures of the uploaded file to a
400 in RunFileMigration (shared by v1 and v2), and skip a leading BOM so an
otherwise valid export imports instead of being rejected.
Fixes API-CLOUD-4B
A 4xx from the migration source (disabled mailbox, expired token, deleted
item) is not a bug on our side, but every one of them was captured as an
exception. Those users now get the failure notification with the upstream
message instead of the "we have been notified" one.
Credentials migrators are verified synchronously before the async migration is
queued (CredentialsChecker). Routes live on v2 only; the listener learns the
migrator via a route-free RegisterMigratorForEvents, which the oauth v2
registrar uses as well. Shared status/migrate registration moved to
migration_shared.go.
Pull the StartMigration -> Migrate -> FinishMigration orchestration out of
the v1 echo handlers into handler.RunFileMigration and csv.RunMigration so
the v2 API can reuse the exact same business logic. v1 is refactored onto
them and stays byte-identical on the wire.
Also tag the CSV detect/preview/config DTOs with doc:/enum: so they carry
descriptions in the v2 OpenAPI schema (ignored by v1 swaggo/xorm).
Add /api/v2 auth/status/migrate endpoints for the three OAuth-based
migrators. One generic helper registers all three ops per migrator
behind its static config gate, so there's no copy-pasted block per
migrator.
The migrate kick-off orchestration (already-running guard + event
dispatch) is extracted into migrationHandler.StartMigration so v1 and
v2 share it; v1's wire output is unchanged. The guard now surfaces as a
typed migration.ErrMigrationAlreadyRunning (412) so v2 can translate it
through the standard error bridge.
Task titles, project titles, team names, doer/assignee names, and API
token titles were interpolated raw into Line(...) calls whose content is
rendered to HTML by goldmark and then sanitized with bluemonday UGCPolicy.
UGCPolicy intentionally allows safe <a href> and <img src> with
http/https URLs, so a title containing Markdown link or image syntax
would survive sanitization as a working phishing link or tracking pixel
in a legitimate Vikunja email.
Introduce notifications.EscapeMarkdown, which prefixes every CommonMark
§2.4 backslash-escapable ASCII punctuation character — including '<' so
autolinks like `<https://evil.com>` are neutralized before reaching
goldmark — with a backslash. Apply it to every user-controlled argument
of every Line(...) call in pkg/models that feeds into an i18n template,
and to the hand-built "* [title](url) (project)" Markdown link in the
overdue-tasks digest notification.
Also escape the migration error string in MigrationFailedNotification,
an additional sink not listed in the advisory (error messages can carry
user-controlled content from the external migration source).
Subject(...), Greeting(...), and CreateConversationalHeader(...) are
left unchanged: Subject is passed directly to the mail library and is
not markdown-rendered, Greeting is rendered via html/template's built-in
HTML escaping without markdown, and the conversational header is
sanitized as raw HTML by bluemonday in mail_render.go.
Fixes GHSA-45q4-x4r9-8fqj.
This changes the error handling to a centralized HTTP error handler in `pkg/routes/error_handler.go` that converts all error types to proper HTTP responses. This simplifies the overall error handling because http handler now only need to return the error instead of calling HandleHTTPError as previously.
It also removes the duplication between handling errors with and without Sentry.
🐰 Hop along, dear errors, no more wrapping today!
We've centralized handlers in a shiny new way,
From scattered to unified, the code flows so clean,
ValidationHTTPError marshals JSON supreme!
Direct propagation hops forward with glee,
A refactor so grand—what a sight to see! 🎉
This change introduces notifications via mail when a migration fails. It will contain the error message and a hint to post it in the forum when Sentry is disabled, otherwise the error message will be sent directly to sentry and the notification will inform accordingly.
I've tried to balance "this thing failed, go figure it out" with "here is what we know and how you can get help", we'll see how well that approach works.
This change fixes a bug where Vikunja would not correctly check if a migration was already running. That meant it was not possible for users who had never before migrated anything to start a migration, because Vikunja assumed they already had a migration running for them.
This state was neither properly reflected in the frontend, which is now fixed as well.
Increase golangci timeout
Fix installing golangci-lint in ci
Remove mage targets replaced by golangci
Run golint in ci
Add goheader linter
Enable & fix more linters
Fix lint issues
Add mage target to automagically fix issues found by golangci
golangci-lint run --fix
Add golangci config
Add golangci mage target
Co-authored-by: kolaente <k@knt.li>
Reviewed-on: https://kolaente.dev/vikunja/api/pulls/676
Co-Authored-By: konrad <konrad@kola-entertainments.de>
Co-Committed-By: konrad <konrad@kola-entertainments.de>
fix copyright date
Add more user tests
More user tests
More user tests
Start refactoring user tests
Docs
Fix lint
Fix db fixtures init in tests
Fix models test
Fix loading fixtures
Fix ineffasign
Fix lint
Fix integration tests
Fix init of test engine creation
Fix user related tests
Better handling of creating test enging
Moved all fixtures to db package
Moved all fixtures to db package
Moved user related stuff to seperate package
Co-authored-by: kolaente <k@knt.li>
Reviewed-on: https://kolaente.dev/vikunja/api/pulls/123
Update swagger docs
Update docs
Let the wunderlist migrator use the registerRoutes function
Add migration status table
Add migration status
Co-authored-by: kolaente <k@knt.li>
Reviewed-on: https://kolaente.dev/vikunja/api/pulls/122
Go mod tidy
[skip ci] Add modules/migration to docs
[skip ci] update date
fmt
Merge branch 'master' into feature/migration
# Conflicts:
# pkg/routes/api/v1/info.go
Add docs on how to create a migrator
Add available migrators to /info endpoint
Return a message once everything was migrated successfully
Add swagger docs for wunderlist migration
Docs for migration [skip ci]
Fix due date fixture in migration test
Fix staticcheck
Fix lint
Logging and cleanup
Make the migrator work with real data
Add routes for migration
Fix misspell
Add method to store a full vikunja structure into vikunja
Add getting all data from wunderlist
Add attachment migration from wunderlist
Add done and done at to wunderlist migrator
Add todo
Add wunderlist auth url implementation
Fix lint
Finish wunderlist migration
Added all structs for the wunderlist migratior
Fix owner field being null for user shared namespaces (#119)
Update copyright year (#118)
Add option to disable registration (#117)
Added migrator interface
Co-authored-by: kolaente <k@knt.li>
Reviewed-on: https://kolaente.dev/vikunja/api/pulls/120
kolaenteandTink
Dominik PschenitschniandGitHub
konradandkolaente