Secret rotation / secret loss recovery #7759

New Issue

GiteaMirror · 2025-11-02T07:35:45-06:00

GiteaMirror commented

2025-11-02 07:35:45 -06:00

Originally created by @clarfonthey on GitHub (Aug 26, 2021).

I've run into issues like #1851 in the past and didn't realise it was due to the SECRET_KEY changing when recreating a config. It would be nice if we could somehow add some mitigations to this.

Recovery from secret loss

If the secret is changed without access to the previous secret, and things like 2FA secrets can't be decrypted, then there should be some easy way to mitigate this. Right now, the only solution is to delete the corrupted rows in the DB manually.

A few potential options:

A gitea doctor command to delete 2FA data, potentially notifying users who had 2FA enabled
A flag on users in the database to prompt them to reconfigure 2FA on login

Proactive secret rotation

If the user wishes to proactively change the secret, there should be an option to include (at least) two secrets in the config: the new secret for future operations, and the old secret for past operations.

A few options here:

Secrets could be configured in optional pairs, where there's a "primary" secret and a "backup" secret. The backup secret is exclusively for decryption, and anything decrypted with it should be re-encrypted with the primary secret.
There could also be the option for more than one backup secret.
A gitea doctor command to re-encrypt things encrypted with a backup secret.
A cron job to re-encrypt things encrypted with a backup secret.
The ability to automatically remove the backup secret once all its data is updated.

The actual action upon rotation of the secret depends on the reason for the rotation. If there's some kind of breach of security, things that were protected by the secret like 2FA keys should be regenerated. So, a few options here:

Secrets could be able to be marked as compromised, and this could also be used with the user flag mentioned in the first section, prompting users to reconfigure 2FA on login.
This could potentially up the priority on the cron job for re-encrypting things, putting it to a high priority.

Originally created by @clarfonthey on GitHub (Aug 26, 2021). I've run into issues like #1851 in the past and didn't realise it was due to the `SECRET_KEY` changing when recreating a config. It would be nice if we could somehow add some mitigations to this. ## Recovery from secret loss If the secret is changed without access to the previous secret, and things like 2FA secrets can't be decrypted, then there should be some easy way to mitigate this. Right now, the only solution is to delete the corrupted rows in the DB manually. A few potential options: * A `gitea doctor` command to delete 2FA data, potentially notifying users who had 2FA enabled * A flag on users in the database to prompt them to reconfigure 2FA on login ## Proactive secret rotation If the user wishes to proactively change the secret, there should be an option to include (at least) two secrets in the config: the new secret for future operations, and the old secret for past operations. A few options here: * Secrets could be configured in optional pairs, where there's a "primary" secret and a "backup" secret. The backup secret is exclusively for decryption, and anything decrypted with it should be re-encrypted with the primary secret. * There could also be the option for more than one backup secret. * A `gitea doctor` command to re-encrypt things encrypted with a backup secret. * A cron job to re-encrypt things encrypted with a backup secret. * The ability to automatically remove the backup secret once all its data is updated. The actual action upon rotation of the secret depends on the reason for the rotation. If there's some kind of breach of security, things that were protected by the secret like 2FA keys should be regenerated. So, a few options here: * Secrets could be able to be marked as compromised, and this could also be used with the user flag mentioned in the first section, prompting users to reconfigure 2FA on login. * This could potentially up the priority on the cron job for re-encrypting things, putting it to a high priority.

GiteaMirror added the issue/critical issue/confirmed type/proposal issue/workaround labels 2025-11-02 07:35:45 -06:00

GiteaMirror commented

2025-11-02 07:35:46 -06:00

@NotNite commented on GitHub (Apr 22, 2023):

For any people unfortunate enough to be frantically Googling this at 11 PM like I was: I was moving my infrastructure to Nix, and the NixOS module for Gitea automatically generated a secret key for me, and so I couldn't log in because it wasn't using the default.

In case anyone needs it, the default secret key is here.

@NotNite commented on GitHub (Apr 22, 2023): For any people unfortunate enough to be frantically Googling this at 11 PM like I was: I was moving my infrastructure to Nix, and the NixOS module for Gitea automatically generated a secret key for me, and so I couldn't log in because it wasn't using the default. In case anyone needs it, the default secret key is [here](https://github.com/go-gitea/gitea/blob/fb32b4c5340a1b5934e2a5164b0847265167b8b8/modules/setting/security.go#L108).

GiteaMirror commented

2025-11-02 07:35:46 -06:00

@SvartrShell commented on GitHub (Dec 13, 2023):

id also like to be able to change this from the default

@SvartrShell commented on GitHub (Dec 13, 2023): id also like to be able to change this from the default

GiteaMirror commented

2025-11-02 07:35:46 -06:00

@techknowlogick commented on GitHub (Dec 13, 2023):

An approach we could take is what minio does where they will take old and new key as config option to rotate

@techknowlogick commented on GitHub (Dec 13, 2023): An approach we could take is what minio does where they will take old and new key as config option to rotate

GiteaMirror commented

2025-11-02 07:35:46 -06:00

@clarfonthey commented on GitHub (Dec 14, 2023):

Note that this is included in the original description, albeit probably not clearly enough. I'll take another pass through it and see what I can do to make that clearer.

EDIT: I've updated the original post to use bulleted lists wherever possible, so that it's hopefully clearer to skim and see the potential tasks people could work on. IMHO we should probably create separate issues for the individual pieces.

@clarfonthey commented on GitHub (Dec 14, 2023): Note that this is included in the original description, albeit probably not clearly enough. I'll take another pass through it and see what I can do to make that clearer. EDIT: I've updated the original post to use bulleted lists wherever possible, so that it's hopefully clearer to skim and see the potential tasks people could work on. IMHO we should probably create separate issues for the individual pieces.

GiteaMirror commented

2025-11-02 07:35:47 -06:00

@r10r commented on GitHub (Jan 24, 2024):

I have a similar issue after upgrading an installation from the gitea helm chart. In the installation the SECRET_KEY was not set and so values in the database are not encrypted. But after an
upgrade the helm chart generated a fresh non-emptySECRET_KEY resulting in the error when trying to access /admin/auths

...s/web/admin/auths.go:53:Authentications() [E] auth.Sources: failed to decrypt by secret, the key (maybe SECRET_KEY?)

Setting theSECRET_KEY to an empty value works - but how can I start using a non-empty SECRET_KEY now ?

@r10r commented on GitHub (Jan 24, 2024): I have a similar issue after upgrading an installation from the [gitea helm chart](https://gitea.com/gitea/helm-chart). In the installation the `SECRET_KEY` was not set and so values in the database are not encrypted. But after an upgrade the helm chart generated a fresh non-empty`SECRET_KEY` resulting in the error when trying to access **/admin/auths** >...s/web/admin/auths.go:53:Authentications() [E] auth.Sources: failed to decrypt by secret, the key (maybe SECRET_KEY?) Setting the`SECRET_KEY` to an empty value works - but how can I start using a non-empty `SECRET_KEY` now ?

GiteaMirror commented

2025-11-02 07:35:47 -06:00

@3bbbeau commented on GitHub (Jul 7, 2024):

For me I was getting this issue on every runner following a restored PostgreSQL dump for Gitea:

level=error msg="failed to fetch task" error="unknown: rpc error: code = Internal desc = pick task: GetSecretsOfTask: failed to decrypt by secret, the key (maybe SECRET_KEY?) might be incorrect: AesDecrypt invalid decrypted base64 string: illegal base64 data at input byte 1"

I removed the secrets for the org/repo (simply updating them did not work), re-added them, and then the error went away and CI jobs ran as expected.

@3bbbeau commented on GitHub (Jul 7, 2024): For me I was getting this issue on every runner following a restored PostgreSQL dump for Gitea: >level=error msg="failed to fetch task" error="unknown: rpc error: code = Internal desc = pick task: GetSecretsOfTask: failed to decrypt by secret, the key (maybe SECRET_KEY?) might be incorrect: AesDecrypt invalid decrypted base64 string: illegal base64 data at input byte 1" I removed the secrets for the org/repo (simply updating them did not work), re-added them, and then the error went away and CI jobs ran as expected.

GiteaMirror commented

2025-11-02 07:35:47 -06:00

@Brawl345 commented on GitHub (Sep 17, 2024):

For any people unfortunate enough to be frantically Googling this at 11 PM like I was: I was moving my infrastructure to Nix, and the NixOS module for Gitea automatically generated a secret key for me, and so I couldn't log in because it wasn't using the default.

In case anyone needs it, the default secret key is here.

And what needs to be set? Copied my old gitea data where there is no SECRET_KEY inside the app.ini. I tried setting the SECRET_KEY to the default value but it still doesn't work, still getting this AesDecrypt error. Can't login anymore, this is very frustrating and undocumented...

EDIT: "Fixed" it by deleting the relevant entries from the two_factor database table and regenerating the 2FA token...

@Brawl345 commented on GitHub (Sep 17, 2024): > For any people unfortunate enough to be frantically Googling this at 11 PM like I was: I was moving my infrastructure to Nix, and the NixOS module for Gitea automatically generated a secret key for me, and so I couldn't log in because it wasn't using the default. > > In case anyone needs it, the default secret key is [here](https://github.com/go-gitea/gitea/blob/fb32b4c5340a1b5934e2a5164b0847265167b8b8/modules/setting/security.go#L108). And what needs to be set? Copied my old gitea data where there is no SECRET_KEY inside the app.ini. I tried setting the SECRET_KEY to the default value but it still doesn't work, still getting this AesDecrypt error. Can't login anymore, this is very frustrating and undocumented... EDIT: "Fixed" it by deleting the relevant entries from the two_factor database table and regenerating the 2FA token...

GiteaMirror commented

2025-11-02 07:35:47 -06:00

@Erwyn commented on GitHub (Dec 12, 2024):

Just went through this as well. The gitea service module let you write the configuration normally found in the app.ini file as per Options documentation so basically you would have to do something like:

services.gitea = {
    enable = true;
    settings = {
      security = {
        SECRET_KEY = "YOURSECRETKEY";
      };
    };
  };

Now, because it is already defined i the module, on build Nix will not be happy stating that you have conflicting values, yours and the one from the package. So you can use something like lib.mkForce to state that yours take precedence:

services.gitea = {
    enable = true;
    settings = {
      security = {
        SECRET_KEY = lib.mkForce "YOURSECRETKEY";
      };
    };
  };

Better yet, in order to not have the key in the Nix store you can use a secret manager like Agenix or Nix-sops and rely on the SECRET_KEY_URI as stated in Gitea's documentation: https://docs.gitea.com/1.19/administration/config-cheat-sheet#security-security. Please do note in the example that I still lib.mkForce on an empty value for SECRET_KEY it is because as per code Gitea is loading both simultaneously (fb32b4c534/modules/setting/security.go (L104)) and will be unhappy if both values are present. So first, I empty the SECRET_KEY so that SECRET_KEY_URI can be taken into account:

services.gitea = {
    enable = true;
    settings = {
      security = {
        SECRET_KEY = lib.mkForce "";
        SECRET_KEY_URI = "file:${config.age.secrets.giteaSecretKey.path}";
      };
    };
  };

Please do note that I am no NixOS expert so maybe it's not the best way to deal with this. If so please do let me know.

@Erwyn commented on GitHub (Dec 12, 2024): Just went through this as well. The gitea service module let you write the configuration normally found in the `app.ini` file as per [Options documentation](https://search.nixos.org/options?channel=24.11&show=services.gitea.settings&from=0&size=50&sort=relevance&type=packages&query=services.gitea) so basically you would have to do something like: ``` services.gitea = { enable = true; settings = { security = { SECRET_KEY = "YOURSECRETKEY"; }; }; }; ``` Now, because it is already defined i the module, on build Nix will not be happy stating that you have conflicting values, yours and the one from the package. So you can use something like `lib.mkForce` to state that yours take precedence: ``` services.gitea = { enable = true; settings = { security = { SECRET_KEY = lib.mkForce "YOURSECRETKEY"; }; }; }; ``` Better yet, in order to not have the key in the Nix store you can use a secret manager like Agenix or Nix-sops and rely on the `SECRET_KEY_URI` as stated in Gitea's documentation: https://docs.gitea.com/1.19/administration/config-cheat-sheet#security-security. Please do note in the example that I still `lib.mkForce` on an empty value for `SECRET_KEY` it is because as per code Gitea is loading both simultaneously (https://github.com/go-gitea/gitea/blob/fb32b4c5340a1b5934e2a5164b0847265167b8b8/modules/setting/security.go#L104) and will be unhappy if both values are present. So first, I empty the `SECRET_KEY` so that `SECRET_KEY_URI` can be taken into account: ``` services.gitea = { enable = true; settings = { security = { SECRET_KEY = lib.mkForce ""; SECRET_KEY_URI = "file:${config.age.secrets.giteaSecretKey.path}"; }; }; }; ``` Please do note that I am no NixOS expert so maybe it's not the best way to deal with this. If so please do let me know.

GiteaMirror commented

2025-11-02 07:35:48 -06:00

@theAkito commented on GitHub (Mar 9, 2025):

In my case, I migrated Gitea from one Helm Chart installation on Kubernetes, to another and it wasn't immediately clear to me, that it is absolutely crucial to migrate SECRET_KEY in the first place, as well.

Now, 2FA via TOTP does not work and am getting the 500 Internal Server Error issue, as stated before.

My previous app.ini had GITEA__SECURITY__ISECRET_KEY(YES, there is an I prefix and it seems to have been correct, at that time) set, which probably got set automatically, as I don't remember setting it myself and if I would've set it myself, I would've recorded it in some password store.

Do we need to modify the database to make the old secret work? I just want to apply the old secret, remove the new one and having it done, as the whole point of this operation is, not having to re-generate all 2FA setups for each user.

@theAkito commented on GitHub (Mar 9, 2025): In my case, I migrated Gitea from one Helm Chart installation on Kubernetes, to another and it wasn't immediately clear to me, that it is absolutely crucial to migrate `SECRET_KEY` in the first place, as well. Now, 2FA via TOTP does not work and am getting the [500 Internal Server Error issue](https://github.com/go-gitea/gitea/issues/1851), as stated before. My previous `app.ini` had `GITEA__SECURITY__ISECRET_KEY`(YES, there is an `I` prefix and it seems to have been correct, at that time) set, which probably got set automatically, as I don't remember setting it myself and if I would've set it myself, I would've recorded it in some password store. Do we need to [modify the database](https://github.com/go-gitea/gitea/issues/1851#issuecomment-691018223) to make the old secret work? I just want to apply the old secret, remove the new one and having it done, as the whole point of this operation is, not having to re-generate all 2FA setups for each user.

GiteaMirror commented

2025-11-02 07:35:48 -06:00

@fabiang commented on GitHub (Apr 23, 2025):

If you have trouble with 500 errors after changing SECRET_KEY while using LDAP for authentication you can do the following:

Connect to your database
Delete the LDAP source from table login_source
Re-add your LDAP source from the Gitea admin panel
Update existing users with the new ID from login_source: UPDATE user u SET u.login_source = <new_id> WHERE u.login_source = <old_id>;

@fabiang commented on GitHub (Apr 23, 2025): If you have trouble with 500 errors after changing `SECRET_KEY` while using LDAP for authentication you can do the following: 1. Connect to your database 2. Delete the LDAP source from table `login_source` 3. Re-add your LDAP source from the Gitea admin panel 4. Update existing users with the new ID from `login_source`: `UPDATE user u SET u.login_source = <new_id> WHERE u.login_source = <old_id>; `

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#7759