SyncRepositoryHooks failing due to EINTR #5456

See:
https://github.com/golang/go/issues/39026#event-3329295983
https://github.com/golang/go/issues/38033
https://github.com/golang/go/issues/20400

@zeripath commented on GitHub (May 25, 2020):

So this has to do with changes in go 1.14 that mean that CIFS has problems.

It appears that GODEBUG=asyncpreemptoff=1 will reduce this from happening.

Rather disappointingly:

Sorry, we aren't going to backport the EINTR change to 1.14. That is much too invasive and risky.

1.14 did not introduce this problem, it merely made it occur more often. The workaround of setting GODEBUG will reduce the number of times the problems occurs back to 1.13 levels.

It looks like the only way to fix this to either:

You need to set GODEBUG=asyncpreemptoff=1 in the environment (!)(!)
Wait till go 1.15 (!)(!)
Rewrite every call that could return an EINTR to retry (!)(!)

this is highly irritating.

@zeripath commented on GitHub (May 25, 2020): So this has to do with changes in go 1.14 that mean that CIFS has problems. See: https://github.com/golang/go/issues/39026#event-3329295983 https://github.com/golang/go/issues/38033 https://github.com/golang/go/issues/20400 It appears that `GODEBUG=asyncpreemptoff=1` will reduce this from happening. Rather disappointingly: > Sorry, we aren't going to backport the EINTR change to 1.14. That is much too invasive and risky. > > 1.14 did not introduce this problem, it merely made it occur more often. The workaround of setting GODEBUG will reduce the number of times the problems occurs back to 1.13 levels. It looks like the only way to fix this to either: * You need to set `GODEBUG=asyncpreemptoff=1` in the environment (!)(!) * Wait till go 1.15 (!)(!) * Rewrite every call that could return an EINTR to retry (!)(!) this is highly irritating.

GiteaMirror commented

https://github.com/go-gitea/gitea/issues/11606

@sshaikh commented on GitHub (May 25, 2020):

Ill have a play - this may be why I'm seeing another issue:

@sshaikh commented on GitHub (May 25, 2020): Ill have a play - this may be why I'm seeing another issue: https://github.com/go-gitea/gitea/issues/11606

GiteaMirror commented

@zeripath commented on GitHub (May 25, 2020):

I suspect it's the same underlying issue.

@zeripath commented on GitHub (May 25, 2020): I suspect it's the same underlying issue.

GiteaMirror commented

@sshaikh commented on GitHub (May 25, 2020):

I can confirm that setting GODEBUG=asyncpreemptoff=1 fixed at least this issue.

Will monitor the other behaviours I noticed going forward.

@sshaikh commented on GitHub (May 25, 2020): I can confirm that setting GODEBUG=asyncpreemptoff=1 fixed at least this issue. Will monitor the other behaviours I noticed going forward.

GiteaMirror commented

@zeripath commented on GitHub (May 25, 2020):

I'm sorry it's a crap "solution" hopefully go 1.15 will fix it properly otherwise we're gonna have to wrap every call that could return an EINTR which will be beyond tiresome!

@zeripath commented on GitHub (May 25, 2020): I'm sorry it's a crap "solution" hopefully go 1.15 will fix it properly otherwise we're gonna have to wrap every call that could return an EINTR which will be beyond tiresome!

GiteaMirror commented

@stale[bot] commented on GitHub (Jul 25, 2020):

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

@stale[bot] commented on GitHub (Jul 25, 2020): This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

GiteaMirror commented

@zeripath commented on GitHub (Oct 11, 2020):

It would be useful to know if this is still happening as we've done a number of things to avoid this.

@zeripath commented on GitHub (Oct 11, 2020): It would be useful to know if this is still happening as we've done a number of things to avoid this.

GiteaMirror commented

@sshaikh commented on GitHub (Oct 11, 2020):

So, update to latest version and remove GODEBUG=asyncpreemptoff=1?

If so I can give it a go over the next few days.

@sshaikh commented on GitHub (Oct 11, 2020): So, update to latest version and remove GODEBUG=asyncpreemptoff=1? If so I can give it a go over the next few days.

GiteaMirror commented

@zeripath commented on GitHub (Oct 11, 2020):

yeah that would be helpful

@zeripath commented on GitHub (Oct 11, 2020): yeah that would be helpful

GiteaMirror commented

@sshaikh commented on GitHub (Oct 12, 2020):

I removed the GODEBUG and reproduced the issue with 1.12.

I then updated to 1.12.5 and did a repo sync, but unfortunately got the same "interrupted system call" error. I'll keep it as is for the time being to see if I get those random 500s too.

@sshaikh commented on GitHub (Oct 12, 2020): I removed the GODEBUG and reproduced the issue with 1.12. I then updated to 1.12.5 and did a repo sync, but unfortunately got the same "interrupted system call" error. I'll keep it as is for the time being to see if I get those random 500s too.

GiteaMirror commented

@zeripath commented on GitHub (Oct 12, 2020):

are you building with Go 1.15.x

@zeripath commented on GitHub (Oct 12, 2020): are you building with Go 1.15.x

GiteaMirror commented

@sshaikh commented on GitHub (Oct 12, 2020):

I used the binaries from the downloads page.

@sshaikh commented on GitHub (Oct 12, 2020): I used the binaries from the downloads page.

GiteaMirror commented

@sshaikh commented on GitHub (Oct 13, 2020):

Oh and I have seen the random 500s since too, so I've readded the env variable (which has done the tick again).

@sshaikh commented on GitHub (Oct 13, 2020): Oh and I have seen the random 500s since too, so I've readded the env variable (which has done the tick again).

GiteaMirror commented

@sshaikh commented on GitHub (Jan 5, 2021):

I've removed GODEBUG=asyncpreemptoff=1 with 1.13.1 and can't see any immediate signs of the interruptions. I'll monitor and update if I see any further issues.

@sshaikh commented on GitHub (Jan 5, 2021): I've removed `GODEBUG=asyncpreemptoff=1` with 1.13.1 and can't see any immediate signs of the interruptions. I'll monitor and update if I see any further issues.

GiteaMirror commented

@zeripath commented on GitHub (Jan 6, 2021):

OK I'll remove the confirmed label so this can go stale and we can close it if so

@zeripath commented on GitHub (Jan 6, 2021): OK I'll remove the confirmed label so this can go stale and we can close it if so

GiteaMirror commented

@sshaikh commented on GitHub (Jan 9, 2021):

Confirmed I'm afraid:

2021/01/09 18:13:31 routers/repo/repo.go:183:handleCreateError() [E] CreatePost: initRepository: createDelegateHooks: write new hook file '/appdata/repositories/user/repo.git/hooks/update.d/gitea': close /appdata/repositories/user/repo.git/hooks/update.d/gitea: interrupted system call

@sshaikh commented on GitHub (Jan 9, 2021): Confirmed I'm afraid: `2021/01/09 18:13:31 routers/repo/repo.go:183:handleCreateError() [E] CreatePost: initRepository: createDelegateHooks: write new hook file '/appdata/repositories/user/repo.git/hooks/update.d/gitea': close /appdata/repositories/user/repo.git/hooks/update.d/gitea: interrupted system call `

GiteaMirror commented

@zeripath commented on GitHub (Jan 9, 2021):

if you confirm you have to tell us what version you are running.

@zeripath commented on GitHub (Jan 9, 2021): if you confirm you have to tell us what version you are running.

GiteaMirror commented

@sshaikh commented on GitHub (Jan 9, 2021):

Version unchanged from my last report at 1.13.1.

@sshaikh commented on GitHub (Jan 9, 2021): Version unchanged from my last report at 1.13.1.

GiteaMirror commented

@zeripath commented on GitHub (Jan 9, 2021):

damn.

@zeripath commented on GitHub (Jan 9, 2021): damn.

GiteaMirror commented