mirror of
https://github.com/go-gitea/gitea.git
synced 2026-03-15 20:52:52 -05:00
Merge conflict checking in progress. Try again in a few minutes #9002
Closed
opened 2025-11-02 08:25:23 -06:00 by GiteaMirror
·
29 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#9002
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @stu1811 on GitHub (May 27, 2022).
Description
For the past several version we've run into many PRs that get stuck "Merge conflict checking in progress. Try again in a few minutes". The only work around I've found is to close the PR and open a new one with the same branch. Is there a way to force it to try again?
Gitea Version
1.16.8
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
2.36.0
Operating System
RHEL 7
How are you running Gitea?
Self hosted behind an apache proxy. I download the amd64 linux binary from the releases page. Binary is started by service at boot.
Database
MySQL
@silentcodeg commented on GitHub (May 27, 2022):
Do you remember at which version of Gitea the problem started to show? That would help narrow down the cause.
@stu1811 commented on GitHub (May 27, 2022):
I want to say sometime in the 1.15.X series but it's worse in the past few 1.16.X. Side note I tried updating our git from 2.24.1 to 2.36.0 and it didn't help.
@silentcodeg commented on GitHub (May 27, 2022):
Is there a way for you to reproduce the problem and capture the gitea logs when you do? Or does it happen randomly?
@stu1811 commented on GitHub (May 27, 2022):
Which logging should I enable? The current logs aren't very helpful.
Edit: It seems to happen after our main branch is updated. Normally the Update Branch button would be available at this point.
Edit 2: Not sure if it matters but we're using LFS.
@stu1811 commented on GitHub (May 27, 2022):
The server is on separate network so I'm manually copying things over.
@silentcodeg commented on GitHub (May 29, 2022):
This is unfortunately too little information to figure out the cause of the problem. If someone could take a look at the full logs and run some tests with the live instance I'm pretty sure there is a simple explanation.
@99rgosse commented on GitHub (May 31, 2022):
Hello

We ran into the same issue lately, and I think this is linked to (the zombie processes here)[https://github.com/go-gitea/gitea/issues/19077] with docker
All those 5 zombie processes are git child of Gitea, running since 22 days for the older ones. (I am still with v1.16.5 under docker & PGsql)
Here is what you get when you restart the container :

I will try to check this with a test instance
@silentcodeg commented on GitHub (May 31, 2022):
This indeed is an issue that seems to be eluding everyone for years. A few bug fixes were applied in this area but there never was a proper bug report on the matter and the root cause is still here, evidently.
Yet, there is a simple way to figure it out if someone was able to get access to a live instance where it happens and the corresponding logs. In your case @99rgosse, the next time it happens, would you be able to investigate further? Looking at the logs from when the zombie git process was launched, trying to figure out which zombie process relates to which operation in Gitea, trying to correlate which task is pending for a long time and why etc. It is tedious but it will lead to a bug report that will be a solid basis for a definitive bug fix.
@stu1811 commented on GitHub (May 31, 2022):
@silentcodeg I'm not sure if this is related or a separate issue. If I a edit a file from the gitea website, create a PR, and then make another edit to the same file on the branch, I get "This pull request is broken due to missing fork information." I reproduced it on the demo site.
https://try.gitea.io/developers/foobar/pulls/6
@silentcodeg commented on GitHub (May 31, 2022):
@zeripath just posted a comment here:
https://github.com/go-gitea/gitea/issues/19077#issuecomment-1141930825
that brings some clarity on this topic. In a nutshell, with 1.17 a significant amount of code was modified to:
@silentcodeg commented on GitHub (May 31, 2022):
This is unrelated and would be worth a separate bug report: the reproducer your provide will allow for a quick fix 👍
@lunny commented on GitHub (May 31, 2022):
Please upgrade to v1.16.8
@99rgosse commented on GitHub (May 31, 2022):
I will :)
I did not have time to try yet the upgrade on my test server... I'm always cautious !
@stu1811 commented on GitHub (May 31, 2022):
I build 1.17 dev and my PR that was previously stuck is no longer stuck 🥳
EDIT: There is one error in the log.
patch_unmerged.go:131:unmergedFIles() [E] Unable to run ls-files -u -z! Error: git ls-files -u -z: signal: interrupt@stu1811 commented on GitHub (May 31, 2022):
Sorry for the double post. One of my PRs is stuck with 1.17.0+dev-634-ge31c6166e.
Here is the stack trace
Another dead process
@zeripath commented on GitHub (May 31, 2022):
@stu1811 Let me deal with these last two comments first:
EDIT: There is one error in the log.
https://github.com/go-gitea/gitea/issues/19818#issuecomment-1142083599
signal: interruptimplies that something has killed the os process runninggit ls-files -u -z- either the parent context has been killed - e.g. you killed the process in the table or you cancelled a cherry-pick request - or... possibly it timed out.There would be more interesting logs above/around this if there was a timeout. But a single logline isn't really enough to help us to help you.
Go-routines labelled with
(dead process)https://github.com/go-gitea/gitea/issues/19818#issuecomment-1142110259
Goroutines labelled
(dead process)are not necessarily a problem. They do need examination but this label does not mean what you think it means.We label each goroutine created by Gitea with a marker of where it was created.
When requests come in we label its goroutine with a pprof label to trace any goroutines it creates and the goroutine itself with that request.
Now that in these cases the request has caused a new longlived DB connection to occur and because that code/library does not relabel its goroutines it is still associated with the request.
In the meantime the request has been dealt with and its process closed but the DB connection and its goroutines remain - hence their association with the "dead" process. This is annoying but not a problem.
Stacktraces
database.sql.(*DB).connectionCleaner /usr/local/go/src/database/sql/sql.go:1068This is simply related to the DB connection pool cleaner as you would see if you looked at the code. This is not a problem.
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 /home/stu/go/pkg/mod/github.com/go-sql-driver@v1.6.0/connection.go:614Again an internal goroutine related to DB connection management.
Hanging Git Processes (or How to use the stacktraces tab correctly to identify a hanging git processes)
If there are "hanging" git OS processes these will be attached to very likely non-dead Gitea processes & requests. The associated process will have a goroutine labelled os/exec.Start() or something like that.
To truly identify a hanging git os process the stacktrace should be there for a while AND the process should not die after clicking on the 🗑️ icon.
If you are concerned that there are hanging processes - give us the whole trace unless you really know what you're looking at.
The webpage output is fine but
gitea manager processes --stacktraceswill emit a text output of this data.How to help us with " One of my PRs is stuck with 1.17.0+dev-634-ge31c6166e."
I need logs.
I'm gonna need to read the rest of this thread as I've been very busy recently and I've missed it however you've provided us with just too little information to help you.
I need logs.
I don't understand why we have to repeatedly paste this: https://docs.gitea.io/en-us/logging-configuration/#debugging-problems
and copy and paste this:
When you open a bug report the template very clearly asks you to do this. What can we do to make this clearer?
Do we need to put this in huge letters?
Can you please give me some logs after you've pushed a change and the pr has got stuck?
@zeripath commented on GitHub (May 31, 2022):
This was due a misconfiguration on the try.gitea.io docker setup - see below.
OLD STUFF
I agree that this is unrelated but - I'm not going to pretend that this is a quick fix...
The problem is interesting because I cannot replicate it locally and so I think it's related to another problem on try that I do not completely understand.
If I add trace logging in I can't see an update message.
OH I BET THIS HAS SOMETHING TO DO WITH PROC-RECEIVE - nope...
@zeripath commented on GitHub (May 31, 2022):
PushToBaseRepo is only in proc-receive which is not definitely going to always be run.-- nope that's not it.The post-receive hook does not appear to be being run... Actually no hooks appear to be being run.
The problem is that the script is failing at:
I bet this another seccomp issue on docker...
Yup... the docker version running try.gitea.io is old.
FIXED.
@zeripath commented on GitHub (May 31, 2022):
OK @stu1811 if you're getting the same problem on your local instance your problem is that your docker version is too old.
See:
https://github.com/go-gitea/gitea/pull/18050
See: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0 & https://about.gitlab.com/blog/2021/08/26/its-time-to-upgrade-docker-engine/
@stu1811 commented on GitHub (May 31, 2022):
I actually don't see it locally since I run it bare metal. I ran into this issue while trying to reproduce the issue in the OP on the demo site.
@singuliere commented on GitHub (May 31, 2022):
You are looking at the problem in the wrong way.
If someone has a service contract that guarantees the confidentiality of the data won't be exposed publicly, they will be in a position to diagnose the problem. You could be that person. Otherwise every person reporting a bug is compelled to edit what they think are the relevant parts of the logs and the database content to not expose confidential information to third parties that are not trusted. And the selection they make is bound to hide the relevant information most of the time.
@zeripath commented on GitHub (May 31, 2022):
@singuliere I'm talking about repeated comments like: https://github.com/go-gitea/gitea/issues/19818#issuecomment-1139794607 - this happens all the time.
We can't help if we don't see any logs and it's crazy the number of times we have to post the same app.ini section from the linked document. If the issue template is not clear we need to make it clearer.
I can suggest a slight improvement on the app.ini suggestion above:
Then at least you'd see some messages like:
which could be used to trace the patch testing process.
@singuliere commented on GitHub (May 31, 2022):
Oh, I see. In that case maybe moving this from point 6. in the list to point 1. in the template would help ?
This is off topic though, sorry for the noise.
@stu1811 commented on GitHub (Jul 26, 2022):
We've run 1.16.8 for a while now and the issues seems much better. Sometimes PRs do get stuck for about 30 minutes but eventually do recover and the update branch/merge buttons become available. One thing I noticed is while stuck some PRs have a red merge arrow and others have a yellow merge arrow. I'm not sure if that's of any significance.
@songchenghao commented on GitHub (Feb 2, 2023):
@zeripath when there is commit status check, this appears.
Is there any works number limit to do commit status check?Our system can have multiple PRs concurrently, and The commit status check of these merge requests will last about ten minutes...
sometimes there is the log below.
2023/01/30 15:24:31 .../queue/workerpool.go:157:pushBoost() [W] WorkerPool: 8 (for repo_stats_update-channel) Channel blocked for 1s - adding 1 temporary workers for 5m0s, block timeout now 2s2023/01/31 21:59:23 .../queue/workerpool.go:112:zeroBoost() [D] WorkerPool: 14 (for pr_patch_checker-channel) has zero workers - adding 1 temporary workers for 5m0s@MoutonSanglant commented on GitHub (Mar 1, 2023):
We have the same issue since a few days on Gitea v1.18.3. Some issues get stuck with "Merge conflict checking in progress. Try again in a few minutes" and the only way we found to unlock it is to close the PR and open a new one.
As you can see in our logs, there is this line which comes repeatedely many times every seconds. The only way we found to make this error disapear is to remove the
/data/gitea/tmp/local-repo/pull.git*folder. I don't think it's a good idea and I would prefer to have an official solution.Logs
I never dived into Gitea codebase and my asumption may be all wrong, but I suspect that some data got corrupted in the database because the docker instance got killed while doing a big operation, and then it tries to start the operation over and over, but fails to do it.
The issue started last week after someone tried to open a very big PR ( ~12,000 files changed, ~150,000 deletions).
We really need to fix this issue because merge conflitct checking takes a very long time (from 5 minutes to half an hour) everytime we open a new PR.
Configuration of logs are set accordingly to the recommandations
@brechtvl commented on GitHub (Mar 4, 2023):
@MoutonSanglant #23154 and #23219 are likely to help, they are very recent and not in a release yet.
@MoutonSanglant commented on GitHub (Mar 6, 2023):
@brechtvl Thank you! I'll make a report as soon as we can test it
@wxiaoguang commented on GitHub (Apr 25, 2023):
Is there any new problem?