[Bug]: Server is crashing seemingly randomly... how to debug? #971

New Issue

2026-02-28T19:26:50-06:00

GiteaMirror commented

2026-02-28 19:26:50 -06:00

Originally created by @jat255 on GitHub (Mar 7, 2024).

Verified issue does not already exist?

I have searched and found no existing issue

Is this related to GoCardless, Simplefin or another bank-sync provider?

I have checked my server logs and could not see any errors there
I will be attaching my server logs to this issue
I will be attaching my client-side (browser) logs to this issue
I understand that this issue will be automatically closed if insufficient information is provided

What happened?

I'm running 24.3.0 installed via Yarn on a Linux system (node 18.18.0). I'm running actual via the following systemd user unit file:

[Unit]
Description=Actual Server

[Service]
Type=simple
Restart=on-failure
WorkingDirectory=/home/username/actual-server
ExecStart=/usr/bin/yarn start
Environment="ACTUAL_SERVER_FILES=/mnt/actual_budget/server_files"
Environment="ACTUAL_USER_FILES=/mnt/actual_budget/user_files"
Environment="DEBUG=actual:config,actual-sensitive:config"

[Install]
WantedBy=multi-user.target

Approximately once per day, the server application is crashing (and not restarting, which is odd but that's probably a systemd thing). Most recently it crashed at around 1:00AM when there was definitely no one accessing the system via any clients. In the journal for my actual systemd unit, I have logs such as the following:

Mar 04 10:30:51 actual-arch systemd[16950]: Started Actual Server.
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.339Z actual:config project root: '/home/username/actual-server'
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config default data directory: '/home/username/actual-server'
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config loading config from default path: '/home/username/actual-server/config.json'
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config config file '/home/username/actual-server/config.json' not found, ignoring.
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config using port 5006
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using hostname ::
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using server files directory /mnt/actual_budget/server_files
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using user files directory /mnt/actual_budget/user_files
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using web root directory /home/username/actual-server/node_modules/@actual-app/web/build
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using file sync limit 20mb
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using sync encrypted file limit 50mb
Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using file limit 20mb
Mar 04 10:30:52 actual-arch yarn[17256]: Checking if there are any migrations to run for direction "up"...
Mar 04 10:30:52 actual-arch yarn[17256]: Migrations: DONE
Mar 04 10:30:52 actual-arch yarn[17256]: Listening on :::5006...
Mar 05 01:14:27 actual-arch systemd[16950]: Stopping Actual Server...
Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Main process exited, code=exited, status=1/FAILURE
Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Failed with result 'exit-code'.
Mar 05 01:14:27 actual-arch systemd[16950]: Stopped Actual Server.
Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Consumed 3.222s CPU time.

Is there another place I should be looking for logs, or some way to turn the debug level up? I have set the DEBUG=actual:config,actual-sensitive:config environment variable, but that doesn't give tons of information. If I restart the systemd unit, it works fine until it crashes again some time later. Any ideas here would be super appreciated.

I also tried manually killing the yarn process with kill and kill -9 to see what the systemd logs look like in that case, and it results in the same log where it says "systemd: Stopping Actual Server", so I don't think it's systemd that's shutting things down (and I don't have any clue as to why it would).

I had posted about this in the Discord, but opening a bug here for better visibility/searching.

What error did you receive?

This issue appears unrelated to the client, so there are no errors other than seeing this the next time I open the app when the server has crashed:

This is because the /sync endpoint returns a 502 from my Caddy reverse proxy since the server is not running.

Where are you hosting Actual?

Locally via Yarn

What browsers are you seeing the problem on?

No response

Operating System

Linux

Originally created by @jat255 on GitHub (Mar 7, 2024). ### Verified issue does not already exist? - [X] I have searched and found no existing issue ### Is this related to GoCardless, Simplefin or another bank-sync provider? - [X] I have checked my server logs and could not see any errors there - [X] I will be attaching my server logs to this issue - [ ] I will be attaching my client-side (browser) logs to this issue - [X] I understand that this issue will be automatically closed if insufficient information is provided ### What happened? I'm running 24.3.0 installed via Yarn on a Linux system (node 18.18.0). I'm running actual via the following systemd user unit file: ``` [Unit] Description=Actual Server [Service] Type=simple Restart=on-failure WorkingDirectory=/home/username/actual-server ExecStart=/usr/bin/yarn start Environment="ACTUAL_SERVER_FILES=/mnt/actual_budget/server_files" Environment="ACTUAL_USER_FILES=/mnt/actual_budget/user_files" Environment="DEBUG=actual:config,actual-sensitive:config" [Install] WantedBy=multi-user.target ``` Approximately once per day, the server application is crashing (and not restarting, which is odd but that's probably a systemd thing). Most recently it crashed at around 1:00AM when there was definitely no one accessing the system via any clients. In the journal for my actual systemd unit, I have logs such as the following: ``` Mar 04 10:30:51 actual-arch systemd[16950]: Started Actual Server. Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.339Z actual:config project root: '/home/username/actual-server' Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config default data directory: '/home/username/actual-server' Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config loading config from default path: '/home/username/actual-server/config.json' Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config config file '/home/username/actual-server/config.json' not found, ignoring. Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.340Z actual:config using port 5006 Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using hostname :: Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using server files directory /mnt/actual_budget/server_files Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using user files directory /mnt/actual_budget/user_files Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using web root directory /home/username/actual-server/node_modules/@actual-app/web/build Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using file sync limit 20mb Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using sync encrypted file limit 50mb Mar 04 10:30:52 actual-arch yarn[17256]: 2024-03-04T17:30:52.341Z actual:config using file limit 20mb Mar 04 10:30:52 actual-arch yarn[17256]: Checking if there are any migrations to run for direction "up"... Mar 04 10:30:52 actual-arch yarn[17256]: Migrations: DONE Mar 04 10:30:52 actual-arch yarn[17256]: Listening on :::5006... Mar 05 01:14:27 actual-arch systemd[16950]: Stopping Actual Server... Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Main process exited, code=exited, status=1/FAILURE Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Failed with result 'exit-code'. Mar 05 01:14:27 actual-arch systemd[16950]: Stopped Actual Server. Mar 05 01:14:27 actual-arch systemd[16950]: actual.service: Consumed 3.222s CPU time. ``` Is there another place I should be looking for logs, or some way to turn the debug level up? I have set the `DEBUG=actual:config,actual-sensitive:config` environment variable, but that doesn't give tons of information. If I restart the systemd unit, it works fine until it crashes again some time later. Any ideas here would be super appreciated. I also tried manually killing the yarn process with `kill` and `kill -9` to see what the systemd logs look like in that case, and it results in the same log where it says "systemd: Stopping Actual Server", so I don't think it's systemd that's shutting things down (and I don't have any clue as to why it would). I had posted about this in the Discord, but opening a bug here for better visibility/searching. ### What error did you receive? This issue appears unrelated to the client, so there are no errors other than seeing this the next time I open the app when the server has crashed: ![image](https://github.com/actualbudget/actual/assets/1278301/6d9ddf09-fe20-4da3-b022-6dd512a55a47) This is because the `/sync` endpoint returns a 502 from my Caddy reverse proxy since the server is not running. ### Where are you hosting Actual? Locally via Yarn ### What browsers are you seeing the problem on? _No response_ ### Operating System Linux

GiteaMirror added the bug label 2026-02-28 19:26:50 -06:00

GiteaMirror closed this issue

2026-02-28 19:26:50 -06:00

GiteaMirror commented

2026-02-28 19:26:51 -06:00

@jat255 commented on GitHub (Mar 7, 2024):

Of course, immediately after I post this I believe I figured it out...

I had been leaving an ssh connection open to my server to watch the logs in the hopes of catching it failing in real time, but never noticed it failing until the next morning. It just dawned on me that it was shutting down when my ssh session was terminating.

This is because I was running actual as a user systemd unit without setting the following in /etc/systemd/logind.conf (https://unix.stackexchange.com/a/521542):

[Login]
…
KillUserProcesses=no
…

I'm pretty confident that was my issue, so I'll close this. Sorry for the noise.

EDIT, I think it's also necessary to run loginctl enable-linger if you want the service to start at boot: https://wiki.archlinux.org/title/Systemd/User#Automatic_start-up_of_systemd_user_instances

@jat255 commented on GitHub (Mar 7, 2024): Of course, immediately after I post this I believe I figured it out... I had been leaving an ssh connection open to my server to watch the logs in the hopes of catching it failing in real time, but never noticed it failing until the next morning. It just dawned on me that it was shutting down when my ssh session was terminating. This is because I was running actual as a user systemd unit without setting the following in `/etc/systemd/logind.conf` (https://unix.stackexchange.com/a/521542): ``` [Login] … KillUserProcesses=no … ``` I'm pretty confident that was my issue, so I'll close this. Sorry for the noise. EDIT, I think it's also necessary to run `loginctl enable-linger` if you want the service to start at boot: https://wiki.archlinux.org/title/Systemd/User#Automatic_start-up_of_systemd_user_instances