gitea dump does not respect --tempdir option #4363

@JacquesOfAllTrades commented on GitHub (Nov 26, 2019):

I'm not a Gitea dev, nor have I tested this, but I'd suggest trying

export TMPDIR=/var/backups/gitea/

before running the dump, then run the dump with the --tempdir option still in place.

Based on the code in this code from [https://github.com/go-gitea/gitea/blob/release/v1.8/cmd/dump.go#L81]:

	// work-around #1103
	if os.Getenv("TMPDIR") == "" {
		os.Setenv("TMPDIR", tmpWorkDir)
	}

there's presumably something in the dump process that relies on the TMPDIR environment variable. Gitea will set TMPDIR to the directory you provide via --tempdir, but only if it's not already set on your system.

Could be worth a shot.

@JacquesOfAllTrades commented on GitHub (Nov 26, 2019): I'm not a Gitea dev, nor have I tested this, but I'd suggest trying ``` export TMPDIR=/var/backups/gitea/ ``` before running the dump, then run the dump **with** the `--tempdir` option still in place. Based on the code in this code from [https://github.com/go-gitea/gitea/blob/release/v1.8/cmd/dump.go#L81]: ``` // work-around #1103 if os.Getenv("TMPDIR") == "" { os.Setenv("TMPDIR", tmpWorkDir) } ``` there's presumably something in the dump process that relies on the TMPDIR environment variable. Gitea will set TMPDIR to the directory you provide via `--tempdir`, but *only* if it's not already set on your system. Could be worth a shot.

GiteaMirror commented

@JacquesOfAllTrades commented on GitHub (Dec 4, 2019):

Just a quick follow-up: I can confirm that both Gitea 1.9.1 and 1.9.5 respect the --tempdir option on my system. I'm running a command like

$sudo -u git gitea dump -c /path/to/app.ini --tempdir /jacques/backups/temp

and the dump contents are correctly staged in /jacques/backup/temp before being assembled into the final .zip file.

Note that the directory passed to --tempdir has to exist and be writable by your git (or gitea) user.

Also, the TMPDIR environment variable on my system is empty, so the code snippet I included in my previous comment will set it to the value provided to the --tempdir argument. If your TMPDIR is already set before running gitea dump, e.g., to /tmp, it won't be modified.

If clearing your TMPDIR var or setting it to the value you're providing to --tempdir works for you, that does seem like a bug, because it would mean the dump process has an undocumented (and undesirable) dependency on that var.

@JacquesOfAllTrades commented on GitHub (Dec 4, 2019): Just a quick follow-up: I can confirm that both Gitea 1.9.1 and 1.9.5 respect the `--tempdir` option on my system. I'm running a command like ``` $sudo -u git gitea dump -c /path/to/app.ini --tempdir /jacques/backups/temp ``` and the dump contents are correctly staged in `/jacques/backup/temp` before being assembled into the final .zip file. Note that the directory passed to `--tempdir` has to exist and be writable by your `git` (or `gitea`) user. Also, the `TMPDIR` environment variable on my system is empty, so the code snippet I included in my previous comment will set it to the value provided to the `--tempdir` argument. If your `TMPDIR` is already set before running `gitea dump`, e.g., to `/tmp`, it won't be modified. If clearing your `TMPDIR` var or setting it to the value you're providing to `--tempdir` works for you, that does seem like a bug, because it would mean the dump process has an undocumented (and undesirable) dependency on that var.

GiteaMirror commented

@nodiscc commented on GitHub (Dec 4, 2019):

I have grown the partition so I can't see the disk full error anymore, but

I have changed my dump script to this:

export TMPDIR=/var/backups/gitea/
cd /var/backups/gitea/
sudo -u gitea gitea dump --tempdir /var/backups/gitea/ -c /etc/gitea/app.ini

ran the script and watched files open by the gitea dump process (watch -n 2 sudo lsof -p 19206). It starts backing up repositories and opens /var/backups/gitea/gitea-dump-597340154/gitea-repo.zip, /var/backups/gitea/gitea-dump-1575490742.zip, all fine. At some point it opens

/tmp/user/998/cae/gitea-dump-1575490742.zip
/tmp/user/998/cae/gitea-dump-1575490742.zip/gitea-repo.zip`.

In the process environment I see

$ sudo cat /proc/19206/environ
....TMP=/tmp/user/998TMPDIR=/tmp/user/998TEMP=/tmp/user/998TEMPDIR=/tmp/user/998

I don't know what sets TMPDIR to this path. Maybe it because it's run through sudo. Maybe related to libpam-tmpdir being enabled. I will do more checks.

@nodiscc commented on GitHub (Dec 4, 2019): I have grown the partition so I can't see the disk full error anymore, but I have changed my dump script to this: ```bash export TMPDIR=/var/backups/gitea/ cd /var/backups/gitea/ sudo -u gitea gitea dump --tempdir /var/backups/gitea/ -c /etc/gitea/app.ini ``` ran the script and watched files open by the `gitea dump` process (`watch -n 2 sudo lsof -p 19206`). It starts backing up repositories and opens `/var/backups/gitea/gitea-dump-597340154/gitea-repo.zip, /var/backups/gitea/gitea-dump-1575490742.zip`, all fine. At some point it opens ``` /tmp/user/998/cae/gitea-dump-1575490742.zip /tmp/user/998/cae/gitea-dump-1575490742.zip/gitea-repo.zip`. ``` In the process environment I see ```bash $ sudo cat /proc/19206/environ ....TMP=/tmp/user/998TMPDIR=/tmp/user/998TEMP=/tmp/user/998TEMPDIR=/tmp/user/998 ``` I don't know what sets TMPDIR to this path. Maybe it because it's run through `sudo`. Maybe related to `libpam-tmpdir` being enabled. I will do more checks.

GiteaMirror commented

@JacquesOfAllTrades commented on GitHub (Dec 4, 2019):

Maybe it because it's run through sudo.

That's a good thought. You'd probably have to enter a sudo shell as the 'gitea' user before setting TMPDIR and then running gitea dump.

On the other hand, I guess I can't say with absolute certainty that no files are getting put into /tmp at any point during my dump process. I didn't see any files left in /tmp when the dump aborted due to an error (whereas I did see files left in the --tempdir directory), but it's possible that some files got briefly put into /tmp later in the process once my error was fixed.

Of course, all of the rigmarole with the TMPDIR variable was just an attempt to work around your original issue, which I agree looks like a bug in the handling of the --tempdir arg.

@JacquesOfAllTrades commented on GitHub (Dec 4, 2019): > Maybe it because it's run through sudo. That's a good thought. You'd probably have to enter a sudo shell as the 'gitea' user before setting TMPDIR and then running `gitea dump`. On the other hand, I guess I can't say with absolute certainty that no files are getting put into `/tmp` at any point during my dump process. I didn't see any files left in `/tmp` when the dump aborted due to an error (whereas I did see files left in the `--tempdir` directory), but it's possible that some files got briefly put into `/tmp` later in the process once my error was fixed. Of course, all of the rigmarole with the `TMPDIR` variable was just an attempt to work around your original issue, which I agree looks like a bug in the handling of the `--tempdir` arg.

GiteaMirror commented

See https://serverfault.com/a/479081/54402

@guillep2k commented on GitHub (Dec 5, 2019):

TMPDIR is (or should be) considered a security setting, as many programs could leak information if this env variable is crafted maliciously. As a consequence, sudo will most likely clean it up.

@guillep2k commented on GitHub (Dec 5, 2019): `TMPDIR` is (or should be) considered a security setting, as many programs could leak information if this env variable is crafted maliciously. As a consequence, `sudo` will most likely clean it up. See https://serverfault.com/a/479081/54402

GiteaMirror commented

@lunny commented on GitHub (Dec 5, 2019):

@nodiscc could you try a recent version, i.e. ~~v1.10.0~~ v1.10.1

@lunny commented on GitHub (Dec 5, 2019): @nodiscc could you try a recent version, i.e. ~v1.10.0~ v1.10.1

GiteaMirror commented

@nodiscc commented on GitHub (Dec 6, 2019):

I will try ASAP

@nodiscc commented on GitHub (Dec 6, 2019): I will try ASAP

GiteaMirror commented

@stale[bot] commented on GitHub (Feb 6, 2020):

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

@stale[bot] commented on GitHub (Feb 6, 2020): This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

GiteaMirror commented

@nodiscc commented on GitHub (Feb 8, 2020):

I no longer use the gitea dump command as it has some crippling limitations in my opinion:

it must be run from the gitea data directory which is not easily done from a cron job with proper use of sudo (I always ended up getting permission denied messages or similar). I strongly suggest that you make it able to run from any directory, and not rely on PWD to create logs, etc. directories.
zips the archive hence preventing any kind of deduplication by my backup program (rsnapshot)

Instead I simply backup the gitea data directory and database:

# /etc/rsnapshot.d/gitea.conf
# rsnapshot configuration for gitea backups
backup          /var/lib/gitea  localhost/
backup_script   /usr/bin/mysqldump --single-transaction -h localhost -u root gitea > gitea.sql  gitea_db/

@nodiscc commented on GitHub (Feb 8, 2020): I no longer use the `gitea dump` command as it has some crippling limitations in my opinion: - it must be run from the gitea data directory which is not easily done from a cron job with proper use of sudo (I always ended up getting `permission denied` messages or similar). I strongly suggest that you make it able to run from any directory, and not rely on `PWD` to create logs, etc. directories. - zips the archive hence preventing any kind of deduplication by my backup program (rsnapshot) Instead I simply backup the gitea data directory and database: ``` # /etc/rsnapshot.d/gitea.conf # rsnapshot configuration for gitea backups backup /var/lib/gitea localhost/ backup_script /usr/bin/mysqldump --single-transaction -h localhost -u root gitea > gitea.sql gitea_db/ ```

GiteaMirror commented

@guillep2k commented on GitHub (Feb 8, 2020):

I no longer use the gitea dump command as it has some crippling limitations in my opinion:

I strongly suggest that you make it able to run from any directory, and not rely on PWD to create logs, etc. directories.

You can always use the --work-dir flag in the command line, specify a WORK_DIR environment variable or set that up in app.ini.

@guillep2k commented on GitHub (Feb 8, 2020): > > > I no longer use the `gitea dump` command as it has some crippling limitations in my opinion: > > I strongly suggest that you make it able to run from any directory, and not rely on `PWD` to create logs, etc. directories. You can always use the `--work-dir` flag in the command line, specify a WORK_DIR environment variable or set that up in `app.ini`.

GiteaMirror commented

@nodiscc commented on GitHub (Feb 9, 2020):

You can always use the --work-dir flag in the command line

This is good to know, it's not documented at https://docs.gitea.io/en-us/backup-and-restore/ though. Maybe raise another issue?

However the dump command is not really satisfying for me:

zipping of the backup requires twice the size of the data directory in disk space on the server (even if I just need to move the zip to another machine, that space is still required during the dump)
the dump archive contains another gitea-repo.zip which makes this worse (and also eats disk space when restoring), and is also bad performance-wise (zip in zip)
zipping of the backup prevents my backup software from performing file-based deduplication

For these reasons I am switching to the backup procedure I described above (dump the database, backup the db dump and gitea data directory).

@nodiscc commented on GitHub (Feb 9, 2020): >You can always use the --work-dir flag in the command line This is good to know, it's not documented at https://docs.gitea.io/en-us/backup-and-restore/ though. Maybe raise another issue? However the dump command is not really satisfying for me: - zipping of the backup requires twice the size of the data directory in disk space on the server (even if I just need to move the zip to another machine, that space is still required during the dump) - the dump archive contains another `gitea-repo.zip` which makes this worse (and also eats disk space when restoring), and is also bad performance-wise (zip in zip) - zipping of the backup prevents my backup software from performing file-based deduplication For these reasons I am switching to the backup procedure I described above (dump the database, backup the db dump and gitea data directory).

GiteaMirror commented

@nodiscc commented on GitHub (Feb 9, 2020):

In addition, git .pack files are already compressed which makes the zip operation a waste of time, disk space and CPU cycles (.packs in .zip in .zip).

@nodiscc commented on GitHub (Feb 9, 2020): In addition, git `.pack` files are already compressed which makes the zip operation a waste of time, disk space and CPU cycles (`.packs` in `.zip` in `.zip`).

GiteaMirror commented