mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2026-07-20 21:25:16 -05:00
Sqlite database corruption #5037
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/vaultwarden#5037
Reference in New Issue
Block a user
Originally created by @lettore on GitHub (Oct 21, 2022).
I suffered a silent DB Sqlite corruption, silent because there's no sign to understand that Vaultwarden was running with a corrupted database.
I'm using Vaultwarden through Chrome extensions and the Android App, both are connecting to Vaultwarden to keep data in sync, and it's working at least writing, never had a error warning.
Today I logged in with the web-ui just to find my vault empty.
In the container log I see lot of errors:
thread '<unnamed>' panicked at 'Error loading ciphers: DatabaseError(__Unknown, "database disk image is malformed")', src/db/models/cipher.rs:470:41This SHOULD exit the container immediately, what's the point running a program storing sensitive and important data that is accepting writing to a corrupted database?
Why don't send an email saying that there's a problem with the server?
Why don't raise an alert in the webpage saying that your data is lost?
You can login, add a new entry and nothing tell you about the problem, after you added the new entry you can see it in the page, but after a reload is gone.
I have 2 backups of the DB file, both running every 6 hours, because is very sensitive data.
I found that the only backup working is from 6 months ago.
I'm really lucky that the Chrome extension and the Android app still have all my passwords updated so I can export them and rebuild all.
I kindly ask you to exit the container immediately after a panic error like this, so it's possible to diagnose correctly the problem.
@BlackDex commented on GitHub (Oct 22, 2022):
I have tried to break the database while running, and i was not able to do that, maybe i need to wait a bit longer or something, but even editing the file with a hex editor didn't made it broken while running. However, when i stopped and started vaultwarden again it didn't started and stopped with an error.
Also, if you say that you were able to store and sync the ciphers between different clients, then up-until that point the database should have been ok. I even was able, after modifying the database file with a hex editor to create a backup via the admin interface.
So, it's hard for me to actually try and debug this. I have kept the server running for a while, i even shredded the wal and shm files etc, and still was able to create a backup, probably because it has some stuff in memory which was helpful in fixing the database i think.
Also, besides all this, I don't know how you create your backups, but if you for example copy only the .sqlite3 file while there are shm and wal files, that will definitely break it. Also, if you somehow copy an old DB over the existing DB, that could cause some strange stuff.
And if you are using a network filesystem, like Samba/CIFS or NFS, that also has issues with sqlite3 files.
In any case, I kinda agree that it should stop somehow, but i would need to be able to mimic this somehow, which i'm not.
I would suggest to read the following wiki articles regarding on how to backup:
Currently I'm afraid we can't really do anything about this until we have some way on reproducing this same kind of behavior.
@lettore commented on GitHub (Oct 22, 2022):
Database is located on ZFS filesystem, one backup is done with ZFS snapshot and another with the command sqlite3 db.sqlite3 .dump.
I'm quite experienced to know how to backup a database, and not mounting it on NFS or copying it while it's being written.
I was able finally to recover the corrupted file, it had 4 error in ciphers index and 2 in devices. I reindexed the ciphers index and cleared the device table. Then I still had a page error in database so I made a sql dump and recreated the database.
I'm able to read from the corrupted database with SQL tools, so it was not completely corrupted and it makes sense why Vaultwarden was able to open it, using it to check the login password etc. the problem was in the ciphers index, when it needs to load all the ciphers it get into panic. This not happens when it just need to write a new entry, so it's possible that the Chrome extension was sending the new password and Vaultwarden correctly write it, but when asking for changes it gets no reply as Vaultwarden was not able to read correctly the ciphers.
I still have the corrupted database so I have no problem to do tests if you need to understand what's happening.
My request anyway it's really simple, beside how is corrupted the database, Vaultwarden in Docker log throws a panic error. This is an error that anyone can understand as critical, so it should be handled properly. Vaultwarden shouldn't just show an empty page just because is not able to read the ciphers table, it SHOULD report the error in the frontend explaining why it's unable to load the ciphers table, and possibly deny any further write to that database. The best would be also to reply with an error 5xx to any API requests so also the Android app or the Chrome extension will be aware of a problem with the server.
@BlackDex commented on GitHub (Oct 22, 2022):
In general the
.dumpis discouraged on running and opened databases. It's much better and safer to use either.backuporVACUUM INTO, since with dump you still need to store the wal and shm as far as i know, and you really really need to be sure that there are no transactions activeAlso, same goes for ZFS snapshotting, that isn't instantaneous, and the wal/shm and sqlite3 could get out-off-sync during the snapshot, but most of the time that isn't going to be an issue, but it could.
As for the error catching, it's more easier said then done tracking these specific errors and acting correctly upon those messages. And, it would still be easier if we were able to reproduce something like this to be sure it could be tackled.
Also, you are running a very old version of Vaultwarden 1.24.0 which is from January this year. In the mean time there have been several large updates done on whole Vaultwarden codebase including async functions, database handling and connections, library updates, newer rust versions which could have bugs fixed, base container image etc.
So, is hard to tell for me if this issue is still valid using the latest version available. And it would be better if you are able to provide a more detailed log during an issue like this, so with
LOG_LEVEL=traceandRUST_BACKTRACE=fullfor example, and then the whole backtrace and a few lines before and after, instead of just one line (but, only using the latest stable or testing version, not an older one).@lettore commented on GitHub (Oct 22, 2022):
As said, I made a backup every 6 hours and I save a new password no more twice for week, so it's quite difficult to snapshot the database while writing. I have 360 snapshots of the database all with the same problem. The original file on disk is corrupt, then all the backups of 6 months were corrupted in the same way, 6 wrong index. So the backup process was perfect in making a exact copy of an unfortunate corrupted file.
Not sure why you say I'm running version 1.24.0 while I always used the latest Docker tag, I can see ver 2.25.0 in UI.
I'm not trying to diagnose why the database became corrupted. I'm just kindly ask to handle a panic error in a different way. I will add to my backup process a PRAGMA INTEGRITY CHECK on the output just to be sure to have a valid backup file and be notified otherwise.
@BlackDex commented on GitHub (Oct 23, 2022):
I'm just trying to help produce better backups and prevent a new corrupted database file, using
.dumpcould cause a corruption in some rare cases,.backupandVACUUM INTOare preferred.And while you maybe are not adding a new password there are some background jobs active which could be doing database stuff on that same moment.
The reason i say you are running 1.24.0 is because of the error message you gave. That specific line only matches 1.24.0 or older.
If this was an old log line from before you updated, then i can understand it, but if this log line is at the end of the log then there is something wrong with your running docker container also.
You mentioned that you still have the old database which is corrupted. I kindly asked if you could provide some more logs including the whole backtrace and a few lines (10 or 20) before and after the whole backtrace with the modified environment variables and with the latest version 1.26.0 or testing so that we have more details using the up-to-date version to try and help you and all other people who might encounter something like this.
And even with that, it's going to be difficult for us to really tackle and test this, since we can't reproduce this easily and make sure the application is doing what it should be in this specific case.
@lettore commented on GitHub (Oct 23, 2022):
My fault, I was using an old image 1.23.1. Using the latest 1.26.0 it goes not beyond trying to do the migration script, so no way to see how the new version deals a corrupted database.
This is the log file while running the old version, as you can see after the error the server continues to reply with 200 OK to other requests.
mws3f3yy92ndwo9zo3bmffjcq_logs.txt
In Admin page I can see users with the correct number of entries and organizations.
As the new image is not able to migrate the database and exits I can't say if the issue is still present.
@BlackDex commented on GitHub (Oct 23, 2022):
Thanks for the logs, hopefully there is something useful we can get out of it.
Also as a side-note. The 1.23.1 version was also using an older SQLite library 3.31.0 and the current stable uses 3.35.4, with in between a lot of changes, bug fixes, optimizations etc... Which could also in turn help, or would have helped preventing the file from getting corrupt.
The current testing branch is even using the a very new version 3.39.2.
@mhzawadi commented on GitHub (Oct 24, 2022):
I too am getting DB corruption, have had it over night. I have hourly backups setup so can normally see when it has issues.
Will grab logs next time and post here, as I have only just found this and have already got things working
@BlackDex commented on GitHub (Oct 24, 2022):
Besides logs, we also need to know the version of Vaultwarden etc..
But besides all that, Vaultwarden isn't corrupting the database. It is mostly using a wrong way of backing up the database, depleted storage, memory issues, or sudden shutdown etc...
@mhzawadi commented on GitHub (Oct 24, 2022):
might have found my issue 🤦🏼 , I use
bruceforce/vaultwarden-backupas the backup container. That was setup to delete files older then 3 days, as indelete * older than 3 days.my bad
@BlackDex commented on GitHub (Oct 24, 2022):
Ow my, that is very bad.
@androidacy-user commented on GitHub (Oct 31, 2022):
We just lost our entire vaultwarden instance because it decided the sqlite database was corrupt overnight and the database was not recoverable through any means i could find. Database is only ever touched by the docker instance itself so can rule that out.
@BlackDex commented on GitHub (Oct 31, 2022):
Vaultwarden doesn't corrupt a database either. So that is also something to rule out. Most other items could be backups, filesystem issues, memory issues and some other items.
Maybe there is a bit more info in the logs? Also, which version of Vaultwarden were you using.
@lettore commented on GitHub (Oct 31, 2022):
Trying recovery mine I saw that it was not corrupted totally, it's just Vaultwarden that refuses to open it. It should be added a tool inside the admin UI to export a corrupted database. With a Windows MySql utility I was able to see all the database correctly, only the last 3 ciphers entries were missing. Try to diagnose where the db is corrupted, as mine had lot of errors in devices table that can be safely deleted.
@androidacy-user commented on GitHub (Dec 1, 2022):
We couldn't access ours with SQLite3 either.
@BlackDex quite frankly no idea as we restored to a known good point after digging through our backups to find one that also had Vaultwarden (it's not included in our daily backups as it's low priority and not expected to randomly fail). You'll understand our first priority was getting back our data.