Connection to remote mysql just drops for some reason and never comes back
Your environment
Bitwarden_rs version:
Install method: Docker Compose
Reverse proxy and version: Apache 2.4.29
Version of mysql/postgresql:
Other relevant information:
Steps to reproduce
My connection string in docker-compose.yml
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs"
Expected behaviour
Connect all happy as can be?
Actual behaviour
Eventually runs if I do docker compose down; docker compose up -d
Relevant logs
bitwardenrs | [2020-10-28 09:40:17.254][r2d2][ERROR] Unknown MySQL server host 'mysql.server.net' (-3)
After restarting I see this. I have to do a complete down and up to get bitwarden_rs working again.
bitwardenrs | [2020-10-28 12:55:10.450][bitwarden_rs::util][WARN] Can't connect to database, retrying: DieselConError.
bitwardenrs | [CAUSE] BadConnection(
bitwardenrs | "Unknown MySQL server host 'mysql.server.net' (-3)",
bitwardenrs | )
Originally created by @misilot on GitHub (Oct 28, 2020).
Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/1204
<!--
# ###
NOTE: Please update to the latest version of bitwarden_rs before reporting an issue!
This saves you and us a lot of time and troubleshooting.
See: https://github.com/dani-garcia/bitwarden_rs/issues/1180
# ###
-->
<!--
Please fill out the following template to make solving your problem easier and faster for us.
This is only a guideline. If you think that parts are unnecessary for your issue, feel free to remove them.
Remember to hide/obfuscate personal and confidential information,
such as names, global IP/DNS addresses and especially passwords, if necessary.
-->
### Subject of the issue
Connection to remote mysql just drops for some reason and never comes back
### Your environment
<!-- The version number, obtained from the logs or the admin diagnostics page -->
<!-- Remember to check your issue on the latest version first! -->
* Bitwarden_rs version:
<!-- How the server was installed: Docker image / package / built from source -->
* Install method: Docker Compose
* Reverse proxy and version: Apache 2.4.29
* Version of mysql/postgresql:
* Other relevant information:
### Steps to reproduce
<!-- Tell us how to reproduce this issue. What parameters did you set (differently from the defaults)
and how did you start bitwarden_rs? -->
My connection string in docker-compose.yml
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs"
### Expected behaviour
<!-- Tell us what should happen -->
Connect all happy as can be?
### Actual behaviour
<!-- Tell us what happens instead -->
Eventually runs if I do docker compose down; docker compose up -d
### Relevant logs
<!-- Share some logfiles, screenshots or output of relevant programs with us. -->
bitwardenrs | [2020-10-28 09:40:17.254][r2d2][ERROR] Unknown MySQL server host 'mysql.server.net' (-3)
After restarting I see this. I have to do a complete down and up to get bitwarden_rs working again.
bitwardenrs | [2020-10-28 12:55:10.450][bitwarden_rs::util][WARN] Can't connect to database, retrying: DieselConError.
bitwardenrs | [CAUSE] BadConnection(
bitwardenrs | "Unknown MySQL server host \'mysql.server.net\' (-3)",
bitwardenrs | )
So restarting bitwarden_rs doesn't fix it, you need to restart the whole docker compose group? Sounds some kind of networking issue in docker, can you share your compose file?
<!-- gh-comment-id:723501843 -->
@dani-garcia commented on GitHub (Nov 7, 2020):
So restarting bitwarden_rs doesn't fix it, you need to restart the whole docker compose group? Sounds some kind of networking issue in docker, can you share your compose file?
#---#Docker-compose file for Bitwarden_rs#---version:"3"services:bitwardenrs:restart:always# Dani Garcia image https://github.com/dani-garcia/bitwarden_rsimage:"bitwardenrs/server:latest"container_name:bitwardenrsenvironment:# Timezone settings, important for Fail2ban to workTZ:'America/Chicago'# Logging connection attempsLOG_FILE:'/data/bitwarden.log'EXTENDED_LOGGING:'true'LOG_LEVEL:'warn'# Beef up a bitROCKET_WORKERS:20WEBSOCKET_ENABLED:'true'# Hardening a bitSIGNUPS_ALLOWED:'false'ADMIN_TOKEN:'<removed>'DOMAIN:'https://key.domain'SHOW_PASSWORD_HINT:'false'DISABLE_ICON_DOWNLOAD:'true'SMTP_HOST:'apprelay.smtp.domain'SMTP_PORT:25SMTP_SSL:'false'SMTP_FROM:'adm@domain'SMTP_FROM_NAME:'IT'DUO_IKEY:'dafsdf'DUO_SKEY:'fasfsad'DUO_HOST:'api-.duosecurity.com'INVITATION_ORG_NAME:'Password Management'ORG_CREATION_USERS:'libadm@domain'networks:- bitwarden_netports:- '9080:80'- '9082:3012'volumes:- "/klib/data/bw-data:/data"volumes:Certfiles:networks:bitwarden_net:
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs" was in there as well, but I had to switch to sqlite backend for now.
I did end up removing the networks: bitwarden_net configuration currently as well trying to troubleshoot #1203
<!-- gh-comment-id:723503610 -->
@misilot commented on GitHub (Nov 7, 2020):
```yaml
#---
#Docker-compose file for Bitwarden_rs
#---
version: "3"
services:
bitwardenrs:
restart: always
# Dani Garcia image https://github.com/dani-garcia/bitwarden_rs
image: "bitwardenrs/server:latest"
container_name: bitwardenrs
environment:
# Timezone settings, important for Fail2ban to work
TZ: 'America/Chicago'
# Logging connection attemps
LOG_FILE: '/data/bitwarden.log'
EXTENDED_LOGGING: 'true'
LOG_LEVEL: 'warn'
# Beef up a bit
ROCKET_WORKERS: 20
WEBSOCKET_ENABLED: 'true'
# Hardening a bit
SIGNUPS_ALLOWED: 'false'
ADMIN_TOKEN: '<removed>'
DOMAIN: 'https://key.domain'
SHOW_PASSWORD_HINT: 'false'
DISABLE_ICON_DOWNLOAD: 'true'
SMTP_HOST: 'apprelay.smtp.domain'
SMTP_PORT: 25
SMTP_SSL: 'false'
SMTP_FROM: 'adm@domain'
SMTP_FROM_NAME: 'IT'
DUO_IKEY: 'dafsdf'
DUO_SKEY: 'fasfsad'
DUO_HOST: 'api-.duosecurity.com'
INVITATION_ORG_NAME: 'Password Management'
ORG_CREATION_USERS: 'libadm@domain'
networks:
- bitwarden_net
ports:
- '9080:80'
- '9082:3012'
volumes:
- "/klib/data/bw-data:/data"
volumes:
Certfiles:
networks:
bitwarden_net:
```
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs" was in there as well, but I had to switch to sqlite backend for now.
I did end up removing the networks: bitwarden_net configuration currently as well trying to troubleshoot #1203
My docker-compose file is based on https://github.com/sosandroid/docker-bitwarden_rs-caddy-synology/blob/master/docker-compose_bitwarden-caddy.yml
[2020-11-08 12:02:19.733][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.733][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.734][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.734][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
But bitwarden_rs has no trouble re-opening connections to my remote MySQL server and there is no bug visible to end users.
Have you tried to add the hostname of your mysql server to container's /etc/hosts file?
My docker compose file has a volume for /etc/hosts, it helps on managing remote IPs (warning: it may not be best practices):
volumes:
- '/etc/hosts:/etc/hosts'
<!-- gh-comment-id:723580475 -->
@jean-io commented on GitHub (Nov 8, 2020):
Strange, I often get theses errors in my logs:
```
[2020-11-08 12:02:19.733][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.733][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.734][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.734][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
[2020-11-08 12:02:19.735][r2d2][ERROR] MySQL server has gone away
```
But bitwarden_rs has no trouble re-opening connections to my remote MySQL server and there is no bug visible to end users.
Have you tried to add the hostname of your mysql server to container's `/etc/hosts` file?
My docker compose file has a volume for `/etc/hosts`, it helps on managing remote IPs **(warning: it may not be best practices)**:
```
volumes:
- '/etc/hosts:/etc/hosts'
```
Sounds like a DNS issue to resolve the MySQL host name indeed. Is the DNS entry somehow dynamic (e.g. provided by Docker, Kubernetes, Cloud provider, etc)? When you have the issue, can you attach to the container and try to resolve the MySQL's FQDN? How often the problem appears?
<!-- gh-comment-id:725262627 -->
@sgissi commented on GitHub (Nov 11, 2020):
Sounds like a DNS issue to resolve the MySQL host name indeed. Is the DNS entry somehow dynamic (e.g. provided by Docker, Kubernetes, Cloud provider, etc)? When you have the issue, can you attach to the container and try to resolve the MySQL's FQDN? How often the problem appears?
@sgissi hostname for me the hostname was static and not provided by docker. The server is running as a standalone server. The machine running docker has other applications (not via docker) as well as other systems that talk to and use mysql constantly, and those never seem to skip a beat.
The problem was occurring every few hours. I wasn't using the system all the time but when it failed the connection, I wasn't able to login to the vault using my credentials until I stopped and started the container again.
I thought I tried doing a nslookup but the command didn't work (it has been a little while since I tried). Should that work, or is there another way to test and resolve the FQDN when it fails. I can look at creating a new container that uses the mysql backend to test again. We switched to sqlite as we wanted to make the switch sooner rather than later.
<!-- gh-comment-id:725412325 -->
@misilot commented on GitHub (Nov 11, 2020):
@sgissi hostname for me the hostname was static and not provided by docker. The server is running as a standalone server. The machine running docker has other applications (not via docker) as well as other systems that talk to and use mysql constantly, and those never seem to skip a beat.
The problem was occurring every few hours. I wasn't using the system all the time but when it failed the connection, I wasn't able to login to the vault using my credentials until I stopped and started the container again.
I thought I tried doing a nslookup but the command didn't work (it has been a little while since I tried). Should that work, or is there another way to test and resolve the FQDN when it fails. I can look at creating a new container that uses the mysql backend to test again. We switched to sqlite as we wanted to make the switch sooner rather than later.
@misilot to do a DNS lookup from within the container you could run the following command on the docker host.
# IPv4 only
docker exec -it bitwarden getent ahosts mysql.example.net
# IPv6 when available
docker exec -it bitwarden getent hosts mysql.example.net
Now i actually do not know if the mysql library uses IPv6 to connect, but just to be sure you can check if it resolves to something.
There also is a DNS check in the admin interface under the diagnostics page which does a DNS check for github.com.
Other then that i have no clue on what it could be. Maybe enabling debug log_level for a while could result in some more info.
<!-- gh-comment-id:729786092 -->
@BlackDex commented on GitHub (Nov 18, 2020):
@misilot to do a DNS lookup from within the container you could run the following command on the docker host.
```bash
# IPv4 only
docker exec -it bitwarden getent ahosts mysql.example.net
# IPv6 when available
docker exec -it bitwarden getent hosts mysql.example.net
```
Now i actually do not know if the mysql library uses IPv6 to connect, but just to be sure you can check if it resolves to something.
There also is a DNS check in the admin interface under the diagnostics page which does a DNS check for github.com.
Other then that i have no clue on what it could be. Maybe enabling debug log_level for a while could result in some more info.
@misilot, have you solved this issue? Is this still relevant using the latest or testing version?
<!-- gh-comment-id:743782807 -->
@BlackDex commented on GitHub (Dec 12, 2020):
@misilot, have you solved this issue? Is this still relevant using the `latest` or `testing` version?
No I haven't been able to, we switched to sqllite and that appears to be
working for fine for us. If you want you can close this, and I can try and
circle back to it in the near future.
<!-- gh-comment-id:746901847 -->
@misilot commented on GitHub (Dec 16, 2020):
No I haven't been able to, we switched to sqllite and that appears to be
working for fine for us. If you want you can close this, and I can try and
circle back to it in the near future.
On Sat, Dec 12, 2020 at 10:50 AM Mathijs van Veluw <notifications@github.com>
wrote:
> @misilot <https://github.com/misilot>, have you solved this issue? Is
> this still relevant using the latest or testing version?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dani-garcia/bitwarden_rs/issues/1204#issuecomment-743782807>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AALBHSDV3YJLNX6EOUA7OODSUONLVANCNFSM4TCWMTEQ>
> .
>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @misilot on GitHub (Oct 28, 2020).
Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/1204
Subject of the issue
Connection to remote mysql just drops for some reason and never comes back
Your environment
Steps to reproduce
My connection string in docker-compose.yml
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs"
Expected behaviour
Connect all happy as can be?
Actual behaviour
Eventually runs if I do docker compose down; docker compose up -d
Relevant logs
bitwardenrs | [2020-10-28 09:40:17.254][r2d2][ERROR] Unknown MySQL server host 'mysql.server.net' (-3)
After restarting I see this. I have to do a complete down and up to get bitwarden_rs working again.
bitwardenrs | [2020-10-28 12:55:10.450][bitwarden_rs::util][WARN] Can't connect to database, retrying: DieselConError.
bitwardenrs | [CAUSE] BadConnection(
bitwardenrs | "Unknown MySQL server host 'mysql.server.net' (-3)",
bitwardenrs | )
@dani-garcia commented on GitHub (Nov 7, 2020):
So restarting bitwarden_rs doesn't fix it, you need to restart the whole docker compose group? Sounds some kind of networking issue in docker, can you share your compose file?
@misilot commented on GitHub (Nov 7, 2020):
DATABASE_URL: "mysql://bitwarden_rs:password@mysql.example.net:53306/bitwarden_rs" was in there as well, but I had to switch to sqlite backend for now.
I did end up removing the networks: bitwarden_net configuration currently as well trying to troubleshoot #1203
My docker-compose file is based on https://github.com/sosandroid/docker-bitwarden_rs-caddy-synology/blob/master/docker-compose_bitwarden-caddy.yml
@jean-io commented on GitHub (Nov 8, 2020):
Strange, I often get theses errors in my logs:
But bitwarden_rs has no trouble re-opening connections to my remote MySQL server and there is no bug visible to end users.
Have you tried to add the hostname of your mysql server to container's
/etc/hostsfile?My docker compose file has a volume for
/etc/hosts, it helps on managing remote IPs (warning: it may not be best practices):@sgissi commented on GitHub (Nov 11, 2020):
Sounds like a DNS issue to resolve the MySQL host name indeed. Is the DNS entry somehow dynamic (e.g. provided by Docker, Kubernetes, Cloud provider, etc)? When you have the issue, can you attach to the container and try to resolve the MySQL's FQDN? How often the problem appears?
@misilot commented on GitHub (Nov 11, 2020):
@sgissi hostname for me the hostname was static and not provided by docker. The server is running as a standalone server. The machine running docker has other applications (not via docker) as well as other systems that talk to and use mysql constantly, and those never seem to skip a beat.
The problem was occurring every few hours. I wasn't using the system all the time but when it failed the connection, I wasn't able to login to the vault using my credentials until I stopped and started the container again.
I thought I tried doing a nslookup but the command didn't work (it has been a little while since I tried). Should that work, or is there another way to test and resolve the FQDN when it fails. I can look at creating a new container that uses the mysql backend to test again. We switched to sqlite as we wanted to make the switch sooner rather than later.
@BlackDex commented on GitHub (Nov 18, 2020):
@misilot to do a DNS lookup from within the container you could run the following command on the docker host.
Now i actually do not know if the mysql library uses IPv6 to connect, but just to be sure you can check if it resolves to something.
There also is a DNS check in the admin interface under the diagnostics page which does a DNS check for github.com.
Other then that i have no clue on what it could be. Maybe enabling debug log_level for a while could result in some more info.
@BlackDex commented on GitHub (Dec 12, 2020):
@misilot, have you solved this issue? Is this still relevant using the
latestortestingversion?@misilot commented on GitHub (Dec 16, 2020):
No I haven't been able to, we switched to sqllite and that appears to be
working for fine for us. If you want you can close this, and I can try and
circle back to it in the near future.
On Sat, Dec 12, 2020 at 10:50 AM Mathijs van Veluw notifications@github.com
wrote:
@BlackDex commented on GitHub (Jan 31, 2021):
Closing this issue as requested.