web UI remains empty (Quick Guide) after pushing from existing git repository #7819

Closed
opened 2025-11-02 07:37:53 -06:00 by GiteaMirror · 24 comments
Owner

Originally created by @frnmst on GitHub (Sep 8, 2021).

  • Gitea version (or commit ref): 1.15.2 (affects 1.15.1 and 1.15.0 as well)

  • Git version: 2.20.1

  • Operating system: Debian 10 using deb [arch=amd64] https://packaging.gitlab.io/gitea buster stable as APT source

  • Database (use [x]):

    • PostgreSQL
    • MySQL (I have now migrated to PostgreSQL but the problem existed with Mariadb as well)
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:

    • Yes (provide example URL)
    • No
  • Log gist:

file log

2021/09/08 18:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch
2021/09/08 18:23:41 ...s/context/context.go:185:HTML() [D] Template: repo/empty
2021/09/08 18:23:42 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a
2021/09/08 18:23:42 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng
2021/09/08 18:23:54 ...ters/private/serv.go:380:ServCommand() [D] Serv Results:
	IsWiki: false
	IsDeployKey: false
	KeyID: 1	KeyName: vm@vm-2021-02-17
	UserName: frnmst
	UserID: 2
	OwnerName: frnmst
	RepoName: m2
	RepoID: 223
2021/09/08 18:23:54 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch
2021/09/08 18:23:54 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch-check
2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 symbolic-ref HEAD refs/heads/master
2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch
2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch-check
2021/09/08 18:23:58 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a
2021/09/08 18:23:58 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng
2021/09/08 18:23:59 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a
2021/09/08 18:23:59 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng
2021/09/08 18:24:01 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a
2021/09/08 18:24:01 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng

console log

2021/09/08 18:43:36 Started GET /api/internal/serv/command/1/frnmst/m2?mode=2&verb=git-receive-pack for 127.0.0.1:53866
2021/09/08 18:43:36 Completed GET /api/internal/serv/command/1/frnmst/m2?mode=2&verb=git-receive-pack 200 OK in 3.415025ms
2021/09/08 18:43:37 Started POST /api/internal/hook/pre-receive/frnmst/m2 for 127.0.0.1:53870
2021/09/08 18:43:37 Completed POST /api/internal/hook/pre-receive/frnmst/m2 200 OK in 4.340007ms
2021/09/08 18:43:37 Started POST /api/internal/hook/post-receive/frnmst/m2 for 127.0.0.1:53874
2021/09/08 18:43:37 Completed POST /api/internal/hook/post-receive/frnmst/m2 200 OK in 1.419227ms
2021/09/08 18:43:37 Started POST /api/internal/hook/set-default-branch/frnmst/m2/master for 127.0.0.1:53876
2021/09/08 18:43:37 Completed POST /api/internal/hook/set-default-branch/frnmst/m2/master 200 OK in 3.992819ms
2021/09/08 18:43:37 Started POST /api/internal/ssh/1/update/224 for 127.0.0.1:53878
2021/09/08 18:43:37 Completed POST /api/internal/ssh/1/update/224 200 OK in 3.032989ms

Description

Hello,

when I create an empty repository and I push an existing one from git, the web UI does not update. This did not happen with version 1.14.7. I made a copy of the database and tried migrating like this:

1.14.7 -> 1.15.0 -> 1.15.1 -> 1.15.2

and

1.14.7 -> 1.15.2

  1. the problem happens both using a systemd service and

    $ HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini web --verbose

  2. I read that the problem might be that the git hooks are not running because the partition is marked as noexec. Not my case:

    UUID=5aad5fe7-0d6f-449e-83e5-326cc43372e3  /var/lib/gitea  btrfs  auto,rw,compress=zstd,autodefrag,commit=60,thread_pool=60 0 0
    
  3. SSH keys are all in order because it works with version 1.14.7

gitea doctor says

[12] Check git-daemon-export-ok files
 - [I] Checked 203 repositories, 0 need updates.
OK

Any ideas?

Originally created by @frnmst on GitHub (Sep 8, 2021). <!-- NOTE: If your issue is a security concern, please send an email to security@gitea.io instead of opening a public issue --> <!-- 1. Please speak English, this is the language all maintainers can speak and write. 2. Please ask questions or configuration/deploy problems on our Discord server (https://discord.gg/gitea) or forum (https://discourse.gitea.io). 3. Please take a moment to check that your issue doesn't already exist. 4. Make sure it's not mentioned in the FAQ (https://docs.gitea.io/en-us/faq) 5. Please give all relevant information below for bug reports, because incomplete details will be handled as an invalid report. --> - Gitea version (or commit ref): 1.15.2 (affects 1.15.1 and 1.15.0 as well) - Git version: 2.20.1 - Operating system: Debian 10 using `deb [arch=amd64] https://packaging.gitlab.io/gitea buster stable` as APT source <!-- Please include information on whether you built gitea yourself, used one of our downloads or are using some other package --> <!-- Please also tell us how you are running gitea, e.g. if it is being run from docker, a command-line, systemd etc. ---> <!-- If you are using a package or systemd tell us what distribution you are using --> - Database (use `[x]`): - [x] PostgreSQL - [x] MySQL (I have now migrated to PostgreSQL but the problem existed with Mariadb as well) - [ ] MSSQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - Log gist: file log ``` 2021/09/08 18:23:41 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch 2021/09/08 18:23:41 ...s/context/context.go:185:HTML() [D] Template: repo/empty 2021/09/08 18:23:42 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a 2021/09/08 18:23:42 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng 2021/09/08 18:23:54 ...ters/private/serv.go:380:ServCommand() [D] Serv Results: IsWiki: false IsDeployKey: false KeyID: 1 KeyName: vm@vm-2021-02-17 UserName: frnmst UserID: 2 OwnerName: frnmst RepoName: m2 RepoID: 223 2021/09/08 18:23:54 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch 2021/09/08 18:23:54 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch-check 2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 symbolic-ref HEAD refs/heads/master 2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch 2021/09/08 18:23:55 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] /var/lib/gitea/repos/frnmst/m2.git: /usr/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch-check 2021/09/08 18:23:58 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a 2021/09/08 18:23:58 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng 2021/09/08 18:23:59 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a 2021/09/08 18:23:59 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng 2021/09/08 18:24:01 ...s/context/context.go:740:1() [D] Session ID: 627b25cd1142c05a 2021/09/08 18:24:01 ...s/context/context.go:741:1() [D] CSRF Token: UDBsDo5BcKhQ08z9kk9m8NsNBb86MTYzMTExNjM3NjU0MDc2MTc2Ng ``` console log ``` 2021/09/08 18:43:36 Started GET /api/internal/serv/command/1/frnmst/m2?mode=2&verb=git-receive-pack for 127.0.0.1:53866 2021/09/08 18:43:36 Completed GET /api/internal/serv/command/1/frnmst/m2?mode=2&verb=git-receive-pack 200 OK in 3.415025ms 2021/09/08 18:43:37 Started POST /api/internal/hook/pre-receive/frnmst/m2 for 127.0.0.1:53870 2021/09/08 18:43:37 Completed POST /api/internal/hook/pre-receive/frnmst/m2 200 OK in 4.340007ms 2021/09/08 18:43:37 Started POST /api/internal/hook/post-receive/frnmst/m2 for 127.0.0.1:53874 2021/09/08 18:43:37 Completed POST /api/internal/hook/post-receive/frnmst/m2 200 OK in 1.419227ms 2021/09/08 18:43:37 Started POST /api/internal/hook/set-default-branch/frnmst/m2/master for 127.0.0.1:53876 2021/09/08 18:43:37 Completed POST /api/internal/hook/set-default-branch/frnmst/m2/master 200 OK in 3.992819ms 2021/09/08 18:43:37 Started POST /api/internal/ssh/1/update/224 for 127.0.0.1:53878 2021/09/08 18:43:37 Completed POST /api/internal/ssh/1/update/224 200 OK in 3.032989ms ``` <!-- It really is important to provide pertinent logs --> <!-- Please read https://docs.gitea.io/en-us/logging-configuration/#debugging-problems --> <!-- In addition, if your problem relates to git commands set `RUN_MODE=dev` at the top of app.ini --> ## Description <!-- If using a proxy or a CDN (e.g. CloudFlare) in front of gitea, please disable the proxy/CDN fully and connect to gitea directly to confirm the issue still persists without those services. --> Hello, when I create an empty repository and I push an existing one from git, the web UI does not update. This did not happen with version 1.14.7. I made a copy of the database and tried migrating like this: 1.14.7 -> 1.15.0 -> 1.15.1 -> 1.15.2 and 1.14.7 -> 1.15.2 1. the problem happens both using a systemd service and `$ HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini web --verbose` 2. I read that the problem might be that the git hooks are not running because the partition is marked as `noexec`. Not my case: ``` UUID=5aad5fe7-0d6f-449e-83e5-326cc43372e3 /var/lib/gitea btrfs auto,rw,compress=zstd,autodefrag,commit=60,thread_pool=60 0 0 ``` 3. SSH keys are all in order because it works with version 1.14.7 gitea doctor says ``` [12] Check git-daemon-export-ok files - [I] Checked 203 repositories, 0 need updates. OK ``` Any ideas?
Author
Owner

@delvh commented on GitHub (Sep 8, 2021):

What confuses me the most is that Gitea apparently updated the web UI on the first push previously…
As far as I know, you always have to reload Gitea to see the new current state.
I don't see why it should be different for the first push.
Unlike GitHub, where you get notified about every single action immediately.

Or do you perhaps mean that the web UI remains empty even after a page reload?

@delvh commented on GitHub (Sep 8, 2021): What confuses me the most is that Gitea apparently updated the web UI on the first push previously… As far as I know, you always have to reload Gitea to see the new current state. I don't see why it should be different for the first push. Unlike GitHub, where you get notified about every single action immediately. Or do you perhaps mean that the web UI remains empty even after a page reload?
Author
Owner

@frnmst commented on GitHub (Sep 8, 2021):

Yes, it remains empty after each reload (F5 on browser, tried on two different browsers). Sometimes if I restart gitea changes are loaded (no more Quick Guide and actual content) but most of the time it does not work.

@frnmst commented on GitHub (Sep 8, 2021): Yes, it remains empty after each reload (F5 on browser, tried on two different browsers). Sometimes if I restart gitea changes are loaded (no more Quick Guide and actual content) but most of the time it does not work.
Author
Owner

@frnmst commented on GitHub (Sep 8, 2021):

If I do the same actions using version 1.14.7 the page refreshes correctly.

@frnmst commented on GitHub (Sep 8, 2021): If I do the same actions using version 1.14.7 the page refreshes correctly.
Author
Owner

@delvh commented on GitHub (Sep 8, 2021):

Does it even persist after a force refresh (on Firefox Ctrl+Shift+R)?

Because if not this issue is most likely cache related, otherwise it is more serious…

@delvh commented on GitHub (Sep 8, 2021): Does it even persist after a force refresh (on Firefox `Ctrl+Shift+R`)? Because if not this issue is most likely cache related, otherwise it is more serious…
Author
Owner

@frnmst commented on GitHub (Sep 8, 2021):

Yes it persists

@frnmst commented on GitHub (Sep 8, 2021): Yes it persists
Author
Owner

@delvh commented on GitHub (Sep 8, 2021):

Not good.
I hope one of the core maintainers can help you better than I can,
because this was the easiest and only thing I could think of that might have resolved this error.

@delvh commented on GitHub (Sep 8, 2021): Not good. I hope one of the core maintainers can help you better than I can, because this was the easiest and only thing I could think of that might have resolved this error.
Author
Owner

@frnmst commented on GitHub (Sep 8, 2021):

Yes it's really strange. There is another problem which might be related so i don't know if I need to open a new issue: when I add a new repository as a migration (tested with "Git" only, both mirror and simple migration) it is stuck on Migrating from .... Tested with https://github.com/thockin/test which is 120KB so it should be instant

@frnmst commented on GitHub (Sep 8, 2021): Yes it's really strange. There is another problem which might be related so i don't know if I need to open a new issue: when I add a new repository as a migration (tested with "Git" only, both mirror and simple migration) it is stuck on `Migrating from ...`. Tested with https://github.com/thockin/test which is 120KB so it should be instant
Author
Owner

@frnmst commented on GitHub (Sep 8, 2021):

I installed redis yesterday for performance. By trial and error, adding these settings to the config seem to solve the problem somehow. Something does not work with queue on simple files maybe

[queue]
TYPE = redis
CONN_STR = redis://127.0.0.1:6379/0
@frnmst commented on GitHub (Sep 8, 2021): I installed redis yesterday for performance. By trial and error, adding these settings to the config seem to solve the problem somehow. Something does not work with queue on simple files maybe ```ini [queue] TYPE = redis CONN_STR = redis://127.0.0.1:6379/0 ```
Author
Owner

@lunny commented on GitHub (Sep 9, 2021):

Have you checked the files permission that owners are git?

@lunny commented on GitHub (Sep 9, 2021): Have you checked the files permission that owners are `git`?
Author
Owner

@frnmst commented on GitHub (Sep 9, 2021):

permissions are these:

drwxr-xr-x 1 gitea gitea 156 Sep  7 11:44 /var/lib/gitea/data/data/queues/
@frnmst commented on GitHub (Sep 9, 2021): permissions are these: drwxr-xr-x 1 gitea gitea 156 Sep 7 11:44 /var/lib/gitea/data/data/queues/
Author
Owner

@idealism-xxm commented on GitHub (Sep 9, 2021):

I ran into a similar problem after Gitea was updated from v1.14.7 to v.1.15.0

After I pushed a commit, I didn't see anything in web UI and the result of API (/repos/{owner}/{repo}/contents/{filepath}) was also empty.

But I could get all files and all commits when I clone it into a new directory.

I also have found a strange thing twice: I created many repositories and pushed commits, but I could only saw the records of creating repositories at Home page.

Suddenly all records appeared at the same time and all created time were same.

@idealism-xxm commented on GitHub (Sep 9, 2021): I ran into a similar problem after Gitea was updated from v1.14.7 to v.1.15.0 After I pushed a commit, I didn't see anything in web UI and the result of API (`/repos/{owner}/{repo}/contents/{filepath}`) was also empty. But I could get all files and all commits when I clone it into a new directory. I also have found a strange thing twice: I created many repositories and pushed commits, but I could only saw the records of creating repositories at Home page. Suddenly all records appeared at the same time and all created time were same.
Author
Owner

@idealism-xxm commented on GitHub (Sep 16, 2021):

This problem happened again with version v1.15.2

Missing records appear after I delete new pod and start a new pod.

image

So I thought there was something stuck and I found two running process.

Screen Shot 2021-09-16 at 15 34 29
@idealism-xxm commented on GitHub (Sep 16, 2021): This problem happened again with version v1.15.2 Missing records appear after I delete new pod and start a new pod. ![image](https://user-images.githubusercontent.com/16055078/133572862-861358c5-13b7-42ef-bd39-0a25818d08c4.png) So I thought there was something stuck and I found two running process. <img width="1159" alt="Screen Shot 2021-09-16 at 15 34 29" src="https://user-images.githubusercontent.com/16055078/133573792-dab13656-383d-40c8-a9c5-3fd4c102d61a.png">
Author
Owner

@zeripath commented on GitHub (Sep 22, 2021):

OK, so we're not sure what's causing this - and without some more logging we're kinda stuck...

However, one of the PRs that went into 1.15.3 means that we now can tell what is opening these repos.

So would it be possible to upgrade to 1.15.3 and try again - and try to give us DEBUG logs from when the cat-file batch is opened and tell us what opened the cat-file batch too.

@zeripath commented on GitHub (Sep 22, 2021): OK, so we're not sure what's causing this - and without some more logging we're kinda stuck... However, one of the PRs that went into 1.15.3 means that we now can tell what is opening these repos. So would it be possible to upgrade to 1.15.3 and try again - and try to give us DEBUG logs from when the cat-file batch is opened and tell us what opened the cat-file batch too.
Author
Owner

@idealism-xxm commented on GitHub (Sep 23, 2021):

Thanks for your reply. I'll try again with v1.15.3

I find another way that could handle this: flushing queue with name push_update-channel on page /admin/monitor/queue/{qid} will make all of missing records appear. (This action will post /admin/monitor/queue/{qid}/flush )

And I find there is something missing in the failure log. Maybe the later task won't add a new worker.

Success:

2021/09/22 07:11:54 Started POST /api/v1/repos/server/template-common/generate for 10.136.200.79:45430
2021/09/22 07:11:54 models/repo.go:1159:CreateRepository() [E] Failed to create /data/git/repositories/server/bb3de708d2a5480eae05881b596b8db5.git/git-daemon-export-ok: open /data/git/repositories/server/bb3de708d2a5480eae05881b596b8db5.git/git-daemon-export-ok: no such file or directory
2021/09/22 07:11:55 Completed POST /api/v1/repos/server/template-common/generate 201 Created in 656.666594ms
2021/09/22 07:11:55 Started PUT /api/v1/repos/server/bb3de708d2a5480eae05881b596b8db5/collaborators/collaborator for 10.136.200.79:45430
2021/09/22 07:11:55 Completed PUT /api/v1/repos/server/bb3de708d2a5480eae05881b596b8db5/collaborators/collaborator 204 No Content in 23.010222ms
2021/09/22 07:11:55 Started GET /server/bb3de708d2a5480eae05881b596b8db5.git/info/refs?service=git-receive-pack for 10.136.200.79:45430
2021/09/22 07:11:55 Completed GET /server/bb3de708d2a5480eae05881b596b8db5.git/info/refs?service=git-receive-pack 200 OK in 59.733867ms
2021/09/22 07:11:55 Started POST /server/bb3de708d2a5480eae05881b596b8db5.git/git-receive-pack for 10.136.200.79:45430
2021/09/22 07:11:56 Started POST /api/internal/hook/pre-receive/server/bb3de708d2a5480eae05881b596b8db5 for [::1]:55642
2021/09/22 07:11:56 Completed POST /api/internal/hook/pre-receive/server/bb3de708d2a5480eae05881b596b8db5 200 OK in 3.116331ms
2021/09/22 07:11:56 Started POST /api/internal/hook/post-receive/server/bb3de708d2a5480eae05881b596b8db5 for [::1]:55650
2021/09/22 07:11:56 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 1 (for push_update-channel) has zero workers - adding 1 temporary workers for 5m0s
2021/09/22 07:11:56 Completed POST /api/internal/hook/post-receive/server/bb3de708d2a5480eae05881b596b8db5 200 OK in 2.523509ms
2021/09/22 07:11:56 Started POST /api/internal/hook/set-default-branch/server/bb3de708d2a5480eae05881b596b8db5/master for [::1]:55654
2021/09/22 07:11:56 Completed POST /api/internal/hook/set-default-branch/server/bb3de708d2a5480eae05881b596b8db5/master 200 OK in 52.226075ms
2021/09/22 07:11:57 Completed POST /server/bb3de708d2a5480eae05881b596b8db5.git/git-receive-pack 200 OK in 1.777624302s
2021/09/22 07:11:57 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 11 (for repo_stats_update-channel) has zero workers - adding 1 temporary workers for 5m0s

Failure:

2021/09/22 07:05:22 Started POST /api/v1/repos/server/template-common/generate for 10.136.200.79:40448
2021/09/22 07:05:22 models/repo.go:1159:CreateRepository() [E] Failed to create /data/git/repositories/server/7f1536ca958841bab9054f424240ad43.git/git-daemon-export-ok: open /data/git/repositories/server/7f1536ca958841bab9054f424240ad43.git/git-daemon-export-ok: no such file or directory
2021/09/22 07:05:22 Completed POST /api/v1/repos/server/template-common/generate 201 Created in 609.753909ms
2021/09/22 07:05:22 Started PUT /api/v1/repos/server/7f1536ca958841bab9054f424240ad43/collaborators/collaborator for 10.136.200.79:40448
2021/09/22 07:05:22 Completed PUT /api/v1/repos/server/7f1536ca958841bab9054f424240ad43/collaborators/collaborator 204 No Content in 21.357616ms
2021/09/22 07:05:22 Started GET /server/7f1536ca958841bab9054f424240ad43.git/info/refs?service=git-receive-pack for 10.136.200.79:40448
2021/09/22 07:05:23 Completed GET /server/7f1536ca958841bab9054f424240ad43.git/info/refs?service=git-receive-pack 200 OK in 64.134938ms
2021/09/22 07:05:23 Started POST /server/7f1536ca958841bab9054f424240ad43.git/git-receive-pack for 10.136.200.79:40448
2021/09/22 07:05:23 Started POST /api/internal/hook/pre-receive/server/7f1536ca958841bab9054f424240ad43 for [::1]:36206
2021/09/22 07:05:23 Completed POST /api/internal/hook/pre-receive/server/7f1536ca958841bab9054f424240ad43 200 OK in 3.097823ms
2021/09/22 07:05:24 Started POST /api/internal/hook/post-receive/server/7f1536ca958841bab9054f424240ad43 for [::1]:36238
2021/09/22 07:05:24 Completed POST /api/internal/hook/post-receive/server/7f1536ca958841bab9054f424240ad43 200 OK in 2.161102ms
2021/09/22 07:05:24 Started POST /api/internal/hook/set-default-branch/server/7f1536ca958841bab9054f424240ad43/master for [::1]:36240
2021/09/22 07:05:24 Completed POST /api/internal/hook/set-default-branch/server/7f1536ca958841bab9054f424240ad43/master 200 OK in 52.543197ms
2021/09/22 07:05:24 Completed POST /server/7f1536ca958841bab9054f424240ad43.git/git-receive-pack 200 OK in 1.881910978s

I notice that the repo_path of the stuck process mentioned before isn't a existent repo.

Is it casued by deleting a repo when pushing to remote?

@idealism-xxm commented on GitHub (Sep 23, 2021): Thanks for your reply. I'll try again with v1.15.3 I find another way that could handle this: flushing queue with name `push_update-channel` on page `/admin/monitor/queue/{qid}` will make all of missing records appear. (This action will post `/admin/monitor/queue/{qid}/flush` ) And I find there is something missing in the failure log. Maybe the later task won't add a new worker. Success: ``` 2021/09/22 07:11:54 Started POST /api/v1/repos/server/template-common/generate for 10.136.200.79:45430 2021/09/22 07:11:54 models/repo.go:1159:CreateRepository() [E] Failed to create /data/git/repositories/server/bb3de708d2a5480eae05881b596b8db5.git/git-daemon-export-ok: open /data/git/repositories/server/bb3de708d2a5480eae05881b596b8db5.git/git-daemon-export-ok: no such file or directory 2021/09/22 07:11:55 Completed POST /api/v1/repos/server/template-common/generate 201 Created in 656.666594ms 2021/09/22 07:11:55 Started PUT /api/v1/repos/server/bb3de708d2a5480eae05881b596b8db5/collaborators/collaborator for 10.136.200.79:45430 2021/09/22 07:11:55 Completed PUT /api/v1/repos/server/bb3de708d2a5480eae05881b596b8db5/collaborators/collaborator 204 No Content in 23.010222ms 2021/09/22 07:11:55 Started GET /server/bb3de708d2a5480eae05881b596b8db5.git/info/refs?service=git-receive-pack for 10.136.200.79:45430 2021/09/22 07:11:55 Completed GET /server/bb3de708d2a5480eae05881b596b8db5.git/info/refs?service=git-receive-pack 200 OK in 59.733867ms 2021/09/22 07:11:55 Started POST /server/bb3de708d2a5480eae05881b596b8db5.git/git-receive-pack for 10.136.200.79:45430 2021/09/22 07:11:56 Started POST /api/internal/hook/pre-receive/server/bb3de708d2a5480eae05881b596b8db5 for [::1]:55642 2021/09/22 07:11:56 Completed POST /api/internal/hook/pre-receive/server/bb3de708d2a5480eae05881b596b8db5 200 OK in 3.116331ms 2021/09/22 07:11:56 Started POST /api/internal/hook/post-receive/server/bb3de708d2a5480eae05881b596b8db5 for [::1]:55650 2021/09/22 07:11:56 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 1 (for push_update-channel) has zero workers - adding 1 temporary workers for 5m0s 2021/09/22 07:11:56 Completed POST /api/internal/hook/post-receive/server/bb3de708d2a5480eae05881b596b8db5 200 OK in 2.523509ms 2021/09/22 07:11:56 Started POST /api/internal/hook/set-default-branch/server/bb3de708d2a5480eae05881b596b8db5/master for [::1]:55654 2021/09/22 07:11:56 Completed POST /api/internal/hook/set-default-branch/server/bb3de708d2a5480eae05881b596b8db5/master 200 OK in 52.226075ms 2021/09/22 07:11:57 Completed POST /server/bb3de708d2a5480eae05881b596b8db5.git/git-receive-pack 200 OK in 1.777624302s 2021/09/22 07:11:57 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 11 (for repo_stats_update-channel) has zero workers - adding 1 temporary workers for 5m0s ``` Failure: ``` 2021/09/22 07:05:22 Started POST /api/v1/repos/server/template-common/generate for 10.136.200.79:40448 2021/09/22 07:05:22 models/repo.go:1159:CreateRepository() [E] Failed to create /data/git/repositories/server/7f1536ca958841bab9054f424240ad43.git/git-daemon-export-ok: open /data/git/repositories/server/7f1536ca958841bab9054f424240ad43.git/git-daemon-export-ok: no such file or directory 2021/09/22 07:05:22 Completed POST /api/v1/repos/server/template-common/generate 201 Created in 609.753909ms 2021/09/22 07:05:22 Started PUT /api/v1/repos/server/7f1536ca958841bab9054f424240ad43/collaborators/collaborator for 10.136.200.79:40448 2021/09/22 07:05:22 Completed PUT /api/v1/repos/server/7f1536ca958841bab9054f424240ad43/collaborators/collaborator 204 No Content in 21.357616ms 2021/09/22 07:05:22 Started GET /server/7f1536ca958841bab9054f424240ad43.git/info/refs?service=git-receive-pack for 10.136.200.79:40448 2021/09/22 07:05:23 Completed GET /server/7f1536ca958841bab9054f424240ad43.git/info/refs?service=git-receive-pack 200 OK in 64.134938ms 2021/09/22 07:05:23 Started POST /server/7f1536ca958841bab9054f424240ad43.git/git-receive-pack for 10.136.200.79:40448 2021/09/22 07:05:23 Started POST /api/internal/hook/pre-receive/server/7f1536ca958841bab9054f424240ad43 for [::1]:36206 2021/09/22 07:05:23 Completed POST /api/internal/hook/pre-receive/server/7f1536ca958841bab9054f424240ad43 200 OK in 3.097823ms 2021/09/22 07:05:24 Started POST /api/internal/hook/post-receive/server/7f1536ca958841bab9054f424240ad43 for [::1]:36238 2021/09/22 07:05:24 Completed POST /api/internal/hook/post-receive/server/7f1536ca958841bab9054f424240ad43 200 OK in 2.161102ms 2021/09/22 07:05:24 Started POST /api/internal/hook/set-default-branch/server/7f1536ca958841bab9054f424240ad43/master for [::1]:36240 2021/09/22 07:05:24 Completed POST /api/internal/hook/set-default-branch/server/7f1536ca958841bab9054f424240ad43/master 200 OK in 52.543197ms 2021/09/22 07:05:24 Completed POST /server/7f1536ca958841bab9054f424240ad43.git/git-receive-pack 200 OK in 1.881910978s ``` I notice that the `repo_path` of the stuck process mentioned before isn't a existent repo. Is it casued by deleting a repo when pushing to remote?
Author
Owner

@cweijden commented on GitHub (Oct 27, 2021):

I started a fresh docker instance (v1.15.5 + MySQL:8) and also ran into the first posters problem. Do I understand correctly that when using Redis, this problem does not occur? Could it be database (version) specific?

@cweijden commented on GitHub (Oct 27, 2021): I started a fresh docker instance (v1.15.5 + MySQL:8) and also ran into the first posters problem. Do I understand correctly that when using Redis, this problem does not occur? Could it be database (version) specific?
Author
Owner

@zeripath commented on GitHub (Oct 27, 2021):

I installed redis yesterday for performance. By trial and error, adding these settings to the config seem to solve the problem somehow. Something does not work with queue on simple files maybe

[queue]
TYPE = redis
CONN_STR = redis://127.0.0.1:6379/0

Sorry I missed this.

I think this might be hinting as to what's going on.

There are two queues that could be getting full here:

  1. The Mirror Queue (although this is not technically a "Queue" in 1.15 - it will be in 1.16 - but it can still get full in 1.15)
  2. The Patch Checker Queue.

Both of these queues can block if they are full and you would need to adjust their lengths - for which there are settings.

Now if you set the default Queue type to be "redis" or "level" then for the patch checking queue (and the mirror queue on 1.16) you will avoid this limit.

So... I think the question is how many PRs do you have in your instance?

@zeripath commented on GitHub (Oct 27, 2021): > I installed redis yesterday for performance. By trial and error, adding these settings to the config seem to solve the problem somehow. Something does not work with queue on simple files maybe > > ```ini > [queue] > TYPE = redis > CONN_STR = redis://127.0.0.1:6379/0 > ``` Sorry I missed this. I think this might be hinting as to what's going on. There are two queues that could be getting full here: 1. The Mirror Queue (although this is not technically a "Queue" in 1.15 - it will be in 1.16 - but it can still get full in 1.15) 2. The Patch Checker Queue. Both of these queues can block if they are full and you would need to adjust their lengths - for which there are settings. Now if you set the default Queue type to be "redis" or "level" then for the patch checking queue (and the mirror queue on 1.16) you will avoid this limit. So... I think the question is how many PRs do you have in your instance?
Author
Owner

@frnmst commented on GitHub (Oct 27, 2021):

@cweijden in my case it works perfectly

@zeripath At the moment I'm still on Gitea 1.15.2. I have these queue settings in the app.ini file:

; Mirror sync queue length, increase if mirror syncing starts hanging
MIRROR_QUEUE_LENGTH       = 4096
; Patch test queue length, increase if pull request patch testing starts hanging
PULL_REQUEST_QUEUE_LENGTH = 1000

I just read in the config cheatsheet that these configs are deprecated although their defaults are still 1000.

I have ~= 300 repository mirrors but practically no PRs. Just for reference here is the full report of what happened.

@frnmst commented on GitHub (Oct 27, 2021): @cweijden in my case it works perfectly @zeripath At the moment I'm still on Gitea 1.15.2. I have these queue settings in the app.ini file: ```ini ; Mirror sync queue length, increase if mirror syncing starts hanging MIRROR_QUEUE_LENGTH = 4096 ; Patch test queue length, increase if pull request patch testing starts hanging PULL_REQUEST_QUEUE_LENGTH = 1000 ``` I just read in the config cheatsheet that these configs are deprecated although their defaults are still 1000. I have ~= 300 repository mirrors but practically no PRs. Just for reference [here](https://blog.franco.net.eu.org/notes/problems-and-solutions-with-gitea-1-15.html) is the full report of what happened.
Author
Owner

@zeripath commented on GitHub (Oct 27, 2021):

Sorry incorrect PR linking above!


The cheatsheet you are looking at is referring to 1.16 not 1.15. Both of those values are still used in 1.15


@zeripath commented on GitHub (Oct 27, 2021): Sorry incorrect PR linking above! --- The cheatsheet you are looking at is referring to 1.16 not 1.15. Both of those values are still used in 1.15 ---
Author
Owner

@frnmst commented on GitHub (Oct 28, 2021):

Ah ok. Is 1.16 still in dev? If so maybe it would be useful to put a version identifier for https://docs.gitea.io something like the one used in the Sphinx RTD theme.

@frnmst commented on GitHub (Oct 28, 2021): Ah ok. Is 1.16 still in dev? If so maybe it would be useful to put a version identifier for https://docs.gitea.io something like the one used in the Sphinx RTD theme.
Author
Owner

@zeripath commented on GitHub (Oct 28, 2021):

There are other issues to discuss online documentation.


I think there's another queue that could actually be getting full - the repo_stats_update queue.

It would be useful to know if you have changed your queue settings from default at all.

If not why don't you try setting your base queue type to level or redis to see if that solves the issue.

@zeripath commented on GitHub (Oct 28, 2021): There are other issues to discuss online documentation. --- I think there's another queue that could actually be getting full - the repo_stats_update queue. It would be useful to know if you have changed your queue settings from default at all. If not why don't you try setting your base queue type to level or redis to see if that solves the issue.
Author
Owner

@frnmst commented on GitHub (Oct 28, 2021):

I found a solution soon after opening this issue. At the moment I have these queue settings and everything is working fine:

[queue]
TYPE = redis
CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[queue.task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

The problem is that this does not answer the question of what went wrong in the first place with the default queue type.

I don't remember changing the queue values before updating to 1.15.2 because of this issue.

@frnmst commented on GitHub (Oct 28, 2021): I found a solution soon after opening this issue. At the moment I have these queue settings and everything is working fine: ```ini [queue] TYPE = redis CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s [queue.task] QUEUE_TYPE = redis QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s [task] QUEUE_TYPE = redis QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s ``` The problem is that this does not answer the question of what went wrong in the first place with the default queue type. I don't remember changing the queue values before updating to 1.15.2 because of this issue.
Author
Owner

@cweijden commented on GitHub (Oct 30, 2021):

For me there seems to be some environmental factor in play. I have started a fresh docker-compose instance on a physical amd64 machine (running Ubuntu 21.04), for which the first personal local repository can be pushed to Gitea, though without proceeding past the Quick Guide page. When starting the same docker-compose.yml in a Ubuntu 21.10 vm (same physical machine) Gitea gets past the Quick Guide after pushing that same local repository.
I guess one of the other Docker containers on the physical machine interferes somehow. I will be sifting through the networking configuration when I have some more time.

@cweijden commented on GitHub (Oct 30, 2021): For me there seems to be some environmental factor in play. I have started a fresh docker-compose instance on a physical amd64 machine (running Ubuntu 21.04), for which the first personal local repository can be pushed to Gitea, though without proceeding past the Quick Guide page. When starting the same docker-compose.yml in a Ubuntu 21.10 vm (same physical machine) Gitea gets past the Quick Guide after pushing that same local repository. I guess one of the other Docker containers on the physical machine interferes somehow. I will be sifting through the networking configuration when I have some more time.
Author
Owner

@lunny commented on GitHub (Nov 4, 2021):

Could you help to run ./gitea doctor?

@lunny commented on GitHub (Nov 4, 2021): Could you help to run `./gitea doctor`?
Author
Owner

@wxiaoguang commented on GitHub (Apr 6, 2022):

This problem should have been fixed after adding the IsEmpty check in repo view.

@wxiaoguang commented on GitHub (Apr 6, 2022): This problem should have been fixed after adding the `IsEmpty` check in repo view.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#7819