diff --git a/.agents/skills/running-vrts b/.agents/skills/running-vrts new file mode 120000 index 0000000000..87cbefb6f3 --- /dev/null +++ b/.agents/skills/running-vrts @@ -0,0 +1 @@ +../../.claude/skills/running-vrts \ No newline at end of file diff --git a/.claude/skills/running-vrts/SKILL.md b/.claude/skills/running-vrts/SKILL.md new file mode 100644 index 0000000000..16aadf288a --- /dev/null +++ b/.claude/skills/running-vrts/SKILL.md @@ -0,0 +1,101 @@ +--- +name: running-vrts +description: Use whenever adding, updating, regenerating, running, or debugging visual regression tests (VRTs) / screenshot tests in the Actual Budget repo — including phrases like "add a VRT", "add a screenshot test", "update the snapshots", "regenerate the VRT screenshots", "the VRT is failing", "yarn vrt", "vrt:docker", or "/update-vrt", and any time a UI change needs screenshot coverage. VRT snapshots must be generated inside the Linux docker image (never on the host) and snapshot updates must be scoped to the changed test only — getting either wrong produces snapshots CI ignores or rewrites every screenshot in the repo. +--- + +# Running VRTs (Visual Regression Tests) in actualbudget/actual + +VRTs are ordinary Playwright e2e tests in `packages/desktop-client/e2e/*.test.ts` that call: + +```ts +await expect(target).toMatchThemeScreenshots(); +``` + +`toMatchThemeScreenshots()` is a custom matcher defined in +`packages/desktop-client/e2e/fixtures.ts`. `target` is a `Page` or a `Locator` — +**prefer a tight locator** (e.g. a popover or panel) over the whole page to +reduce noise from charts and data. + +## Key mechanics + +- **No-op without `VRT=true`.** In a plain e2e run the matcher passes + immediately, so `yarn workspace @actual-app/web run playwright test -g ""` + verifies the interaction flow cheaply without needing snapshots — always do + this first. +- **With `VRT=true`, each call captures three themes** (`auto` i.e. light, + `dark`, `midnight`), producing numbered snapshots: + `.test.ts-snapshots/--N-chromium-linux.png`. + One test with 2 assertions → 6 PNGs. +- **Masking:** elements with `data-vrt-mask="true"` are masked out of + screenshots. Use it for volatile content (real dates, version numbers). +- **Diff tolerance is strict:** `maxDiffPixels: 5`, `threshold: 0.05` + (`packages/desktop-client/playwright.config.ts`). +- **Snapshots are Linux-only.** Committed snapshots end in + `-chromium-linux.png` and must be generated inside the Playwright docker + image. A host run on macOS writes `-chromium-darwin.png`, which CI ignores. + +## Writing a new VRT + +Import from the local fixtures, not `@playwright/test`: + +```ts +import { expect, test } from './fixtures'; +``` + +Patterns to copy: + +- Date-picker popover VRT: `e2e/transactions.test.ts` ("by date"). +- Report VRTs: `e2e/reports.test.ts`. +- Tests use the demo budget (`ConfigurationPage.createTestFile()`), whose data + is pinned (date ranges show 2016), so snapshots are largely date-stable. +- Popovers render with a `[data-popover]` attribute — a good screenshot + locator. +- Page models live in `e2e/page-models/`. + +## Generating snapshots locally (docker) + +Run all yarn commands from the repo root. + +1. Start the dev server with HTTPS (background it; ready when + `https://localhost:3001` returns 200 — `ignoreHTTPSErrors` is already set in + the Playwright config): + + ```sh + HTTPS=true yarn start + ``` + +2. The container runs with `--network host`. On Linux it can reach + `https://localhost:3001` directly. On macOS/Windows it cannot — get the + machine's LAN IP (`ipconfig getifaddr en0` on macOS) and use + `https://:3001` instead. + +3. Run the Playwright docker image directly. The `bin/run-vrt` wrapper passes + `-it`, which fails without a TTY, so in an agent shell invoke docker + yourself: + + ```sh + docker run --rm --network host -v "$(pwd)":/work/ -w /work/ \ + mcr.microsoft.com/playwright:v1.59.1-jammy /bin/bash -c \ + "E2E_START_URL=https://:3001 yarn vrt --update-snapshots -g '' .test.ts" + ``` + + Pin the image tag to the repo's Playwright version (`@playwright/test` in + `packages/desktop-client/package.json` — currently 1.59.1 → + `v1.59.1-jammy`). The equivalent user-facing wrapper is + `yarn vrt:docker --e2e-start-url https://:3001 --update-snapshots`. + +4. **Always scope `--update-snapshots`** with `-g`/file args to the new or + changed test only — an unscoped update rewrites every screenshot in the + repo. + +5. Verify by re-running the exact same command **without** + `--update-snapshots`; it must pass. + +6. Optionally Read one generated PNG to sanity-check it shows the intended UI, + then commit the test + PNGs (commit messages need the `[AI]` prefix — see + the committing-actual-changes skill). + +## Fallback without docker + +Commit only the test file; CI generates the snapshots when someone comments +`/update-vrt` on the PR (`.github/workflows/vrt-update-generate.yml`). diff --git a/upcoming-release-notes/add-vrt-skill.md b/upcoming-release-notes/add-vrt-skill.md new file mode 100644 index 0000000000..c48e5a572c --- /dev/null +++ b/upcoming-release-notes/add-vrt-skill.md @@ -0,0 +1,6 @@ +--- +category: Maintenance +authors: [MatissJanis] +--- + +Add an agent skill documenting how to run and update visual regression tests