mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2026-05-24 00:53:23 -05:00
[GH-ISSUE #3912] rocket segfault make vaultwarden exit #18490
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Asutorufa on GitHub (Sep 22, 2023).
Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/3912
Subject of the issue
rocket segfault make vaultwarden exit
Deployment environment
Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page)
Show Running Config
Environment settings which are overridden: ADMIN_TOKEN
Steps to reproduce
random
Expected behaviour
normal running
Actual behaviour
random exit
Troubleshooting data
@BlackDex commented on GitHub (Sep 22, 2023):
Are there any logs of Vaultwarden it self? The current log's posted are not useful for us to even start figuring out what could be the issue.
@Asutorufa commented on GitHub (Sep 23, 2023):
I see no panic in log, it just exit and then i restart it. This behavior only in the 1.29.2, 1.29.1 everything is ok.
@BlackDex commented on GitHub (Sep 23, 2023):
And what about the
testing-alpinetagged image?@Asutorufa commented on GitHub (Sep 23, 2023):
OK, I'll try.
@mirolm commented on GitHub (Sep 23, 2023):
I saw this several times when clients fetch alot of icons from added entries. Sometimes it crashes 2-3 times before client gives up and leaves the site without icon. Below are some of the sites that have missing icons after such refreshes:
https://bulbankonline.bg/
https://brunata.bg/
Not sure this is the exact cause tho, but hope this helps. The above two sites had icons before the new icon code was introduced.
@BlackDex commented on GitHub (Sep 23, 2023):
@mirolm, which deployment do you use? Docker? Debian, Alpine? Which architecture?
Which version are you using? Is it the current
stableortesting?@mirolm commented on GitHub (Sep 23, 2023):
I use standalone install with executables extracted from testing-alpine running on debian 11 x64. Build my client shows is 1.29.2-bbd630f1.
@BlackDex commented on GitHub (Sep 23, 2023):
@mirolm could you please try the build from yesterday? It has some updated libraries.
@mirolm commented on GitHub (Sep 23, 2023):
Now my client reports build 1.29.2-bc26bfa5. It works so far without crashes, but is there a way to invalidate the server icon cache so i can test my guess?
I noticed the server is pretty rock solid in it's daily operation. The only times i noticed it crash is when it pulls tons of icons all at once.
@BlackDex commented on GitHub (Sep 23, 2023):
@mirolm this can be done in two ways.
bc26bfa589/.env.template (L63..L66)The first one setting to something like 1 or 2 second would probably work.
@mirolm commented on GitHub (Sep 23, 2023):
Yea it crashed after i cleared the browser cache too. Last rows of the log are:
Sep 23 13:24:09 altair vaultwarden[1522133]: [2023-09-23 13:24:09.942][request][INFO] GET /icons/brunata.bg/icon.png
Sep 23 13:24:10 altair systemd[1]: vaultwarden.service: Main process exited, code=killed, status=11/SEGV
Sep 23 13:24:10 altair systemd[1]: vaultwarden.service: Failed with result 'signal'.
[578896.207012] traps: rocket-worker-t[1522143] general protection fault ip:7f04448f3da4 sp:7f0441a4bdf0 error:0 in vaultwarden[7f0442a72000+244f000]
[578977.951013] traps: rocket-worker-t[1522351] general protection fault ip:7f384ac6cc05 sp:7f3847f81a58 error:0
[579160.640653] traps: rocket-worker-t[1522448] general protection fault ip:7f63c4dec8ff sp:7f63c1b3ebd8 error:0 in vaultwarden[7f63c2f6b000+244f000]
Sooo i guess this site makes it crash for me.
@BlackDex commented on GitHub (Sep 23, 2023):
Ill see of i can replicate it.
@BlackDex commented on GitHub (Sep 23, 2023):
I can replicate it. It seems to be an Alpine thingy only. It doesn't break on the Debian container for me. Ill have to check it out.
@BlackDex commented on GitHub (Sep 23, 2023):
So, 1.29.1 seems to work just fine. Everything after that fails.
It could be multiple items.
@BlackDex commented on GitHub (Sep 23, 2023):
Found the issue. It seems to be a combination of the OpenSSL version and the OpenSSL Crate which doesn't seem to use v3.x.x OpenSSL version. This probably causes some API calls to be incompatible and break.
@BlackDex commented on GitHub (Sep 23, 2023):
Hmmm, thought it was that, but for some reason it broke again.
I now see that if i increase
ROCKET_WORKERSit doesn't fail. So it must be something with threading.@sidewinder94 commented on GitHub (Sep 28, 2023):
@BlackDex This is probably the issue I had and asked about on the matrix channel (Irwene). Feel free to ask if you want me to try things on my end
@BlackDex commented on GitHub (Sep 28, 2023):
I have figured it out already. And can reproduce it. Now, i only need to fix it fully. Running into some compile issues.
@BlackDex commented on GitHub (Sep 29, 2023):
Ok, i didn't knew me merging my PR on my Repo would close the issue in this repo :).
But technically it is fixed, though no new release yet of Vaultwarden which contains this fix.
@bo0tzz commented on GitHub (Oct 2, 2023):
Should this issue be reopened until the fix is in Vaultwarden itself?
@BlackDex commented on GitHub (Oct 2, 2023):
No, not really. Since there is nothing to fix in Vaultwarden. We should only trigger a new build, for which i am also working on some changed.
@Crashdummyy commented on GitHub (Oct 6, 2023):
@BlackDex
So how can I handle this in the meantime ?
I disabled icon downloads and increased the WorkerPool to 30 it however still crashes about 200 times a day
@BlackDex commented on GitHub (Oct 6, 2023):
Revert to v1.29.1 or use debian instead of alpine.
@Crashdummyy commented on GitHub (Oct 6, 2023):
Thanks for the quick reply.
I am in somewhat the same situation as this issuer.
I'm on postgres and when I change the base image from
testing-alpinetotestingI cant start the server../vaultwarden: error while loading shared libraries: libmariadb.so.3: cannot open shared object file: No such file or directoryIs there a way to supress this or do I need to install the mariadb
@BlackDex commented on GitHub (Oct 6, 2023):
say what? That is on the current testing?
@Crashdummyy commented on GitHub (Oct 6, 2023):
on the non testing as well.
I deploy the binaries like this.
before that I always used
server:alpine-testingwhich can be started but crashes for now@BlackDex commented on GitHub (Oct 6, 2023):
Ah, you extract the binary. That will not work with the debian based containers, as they are linked to other shared libraries.
I'm working on the rebuild of the building right now.
One other thing you could do is build the container your self right now and extract it from there.
That will use the new Alpine build containers and solve your issue.
@Crashdummyy commented on GitHub (Oct 6, 2023):
I guess Ill do that for now unitl the new release is available.
Thanks a lot for your help :)
@BlackDex commented on GitHub (Oct 6, 2023):
The problem is, if i trigger a new testing now, it will still have a broken debian build for armv6.
I want to have both solved instead of creating two different testing releases. That only makes it confusing.