github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-07-15 20:09:18 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Crashes/502 error when act runner is running #14212

New Issue
Closed
opened 2025-11-02 11:06:33 -06:00 by GiteaMirror · 16 comments
GiteaMirror commented 2025-11-02 11:06:33 -06:00
Owner
Copy Link

Originally created by @NadeemSadiq on GitHub (Mar 3, 2025).

Description

If i have a act runner connected to my gitea, the gitea (or cloudflare, not 100% sure tbh) returns a 502 error. I am unable to pull any packages while act runner is connected and the webpage randomly crashes a few seconds. Below is the full list of all the info i can think of.

Logs from Gitea Monitor/stack page:
gitea-diagnosis-20250303-112721.zip

Logs from console output:
console-log-output.txt

View on resources on kubernetes:
Image
Image
Image

Gitea Version

1.23.4

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

1.23.4

Operating System

Arm64-Kubernetes (k3s)-Linux

How are you running Gitea?

Environment:

  • Kubernetes Version:k3s Kubernetes
  • Arm64 CPU Arch

Version:

  • Gitea-1.23.4
  • Act runner-v0.2.11
  • DockerInDocker-docker:20.10-dind

Deployment YAML for Gita:
deployment-yaml.txt

Database

SQLite

Originally created by @NadeemSadiq on GitHub (Mar 3, 2025). ### Description If i have a act runner connected to my gitea, the gitea (or cloudflare, not 100% sure tbh) returns a 502 error. I am unable to pull any packages while act runner is connected and the webpage randomly crashes a few seconds. Below is the full list of all the info i can think of. Logs from Gitea Monitor/stack page: [gitea-diagnosis-20250303-112721.zip](https://github.com/user-attachments/files/19052475/gitea-diagnosis-20250303-112721.zip) Logs from console output: [console-log-output.txt](https://github.com/user-attachments/files/19052499/console-log-output.txt) View on resources on kubernetes: ![Image](https://github.com/user-attachments/assets/254cab86-abb3-4dc2-8d66-ee130872702d) ![Image](https://github.com/user-attachments/assets/ab4dba9f-468f-4aa0-9124-4bad70773f86) ![Image](https://github.com/user-attachments/assets/5398f90e-cfe5-49e5-8d90-204ffd0cde19) ### Gitea Version 1.23.4 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version 1.23.4 ### Operating System Arm64-Kubernetes (k3s)-Linux ### How are you running Gitea? Environment: - Kubernetes Version:k3s Kubernetes - Arm64 CPU Arch Version: - Gitea-1.23.4 - Act runner-v0.2.11 - DockerInDocker-docker:20.10-dind Deployment YAML for Gita: [deployment-yaml.txt](https://github.com/user-attachments/files/19052453/deployment-yaml.txt) ### Database SQLite
GiteaMirror added the issue/needs-feedbacktype/bug labels 2025-11-02 11:06:33 -06:00
GiteaMirror commented 2025-11-02 11:06:34 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 3, 2025):

Is it a public instance?

@wxiaoguang commented on GitHub (Mar 3, 2025): Is it a public instance?
GiteaMirror commented 2025-11-02 11:06:34 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 3, 2025):

Its technically private but It is exposed. You can view it here:
https://git.areaq.xyz/

You can just refresh the page couple times to see it 502. Don't need to even sign up to see the problem

@NadeemSadiq commented on GitHub (Mar 3, 2025): Its technically private but It is exposed. You can view it here: https://git.areaq.xyz/ You can just refresh the page couple times to see it 502. Don't need to even sign up to see the problem
GiteaMirror commented 2025-11-02 11:06:34 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 3, 2025):

From your log, I didn't see any "ERROR" log. And Gitea itself doesn't respond 502 (it only responds 500 if internal error occurs)

Could you try to curl your server directly without cloudflare to see whether there would be still any errors?

@wxiaoguang commented on GitHub (Mar 3, 2025): From your log, I didn't see any "ERROR" log. And Gitea itself doesn't respond 502 (it only responds 500 if internal error occurs) Could you try to `curl` your server directly without cloudflare to see whether there would be still any errors?
GiteaMirror commented 2025-11-02 11:06:34 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 3, 2025):

And, is it 100% related to "when act runner is running"? For example: when no runner, no 502, when runner starts, then randomly 502?

@wxiaoguang commented on GitHub (Mar 3, 2025): And, is it 100% related to "when act runner is running"? For example: when no runner, no 502, when runner starts, then randomly 502?
GiteaMirror commented 2025-11-02 11:06:34 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 3, 2025):

Yes, when i stop the git runner, it works fine. Been using it for a long while now and wanted to include act runner now which is why i'm getting this issue. I did get this issue on an older version, upgrade since i assumed it was a version issue which was not the case. I can disable it now if you want to view the site with the 502?

@NadeemSadiq commented on GitHub (Mar 3, 2025): Yes, when i stop the git runner, it works fine. Been using it for a long while now and wanted to include act runner now which is why i'm getting this issue. I did get this issue on an older version, upgrade since i assumed it was a version issue which was not the case. I can disable it now if you want to view the site with the 502?
GiteaMirror commented 2025-11-02 11:06:35 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 3, 2025):

Hmm, that's quite strange. If it is surely related to the runner, then maybe we need to figure more details. So let's keep the runner running:

  • Did you see any ERROR logs in Gitea's log when the 502 error occurs? Or other reverse proxies?
  • Could you try to curl your server directly without cloudflare to see whether there would be still any errors?

I can see that even some simple static files are also responded as 502 randomly (for example: https://git.areaq.xyz/assets/img/favicon.png, it only serves a static file, no git/db operation but it could also be 502), I guess it's somewhat related to the HTTP reverse proxy or the Gitea's builtin HTTP server?

@wxiaoguang commented on GitHub (Mar 3, 2025): Hmm, that's quite strange. If it is surely related to the runner, then maybe we need to figure more details. So let's keep the runner running: * Did you see any ERROR logs in Gitea's log when the 502 error occurs? Or other reverse proxies? * Could you try to `curl` your server directly without cloudflare to see whether there would be still any errors? I can see that even some simple static files are also responded as 502 randomly (for example: `https://git.areaq.xyz/assets/img/favicon.png`, it only serves a static file, no git/db operation but it could also be 502), I guess it's somewhat related to the HTTP reverse proxy or the Gitea's builtin HTTP server?
GiteaMirror commented 2025-11-02 11:06:35 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 3, 2025):

Odd, i redirected the webpage in Kubernetes to expose it directly as well and it seems to not be crashing at all now only when talking directly with it. I am a bit confused since i do see events show up in Gitea when it crashes when going through the cloudflare link. I think the issue may be on my ingress possibly but not 100% sure. Will investigate a bit further to make sure the act runner is still connected while i changed the service since this may not be connected correctly (hence the lack of crashing)
.

@NadeemSadiq commented on GitHub (Mar 3, 2025): Odd, i redirected the webpage in Kubernetes to expose it directly as well and it seems to not be crashing at all now only when talking directly with it. I am a bit confused since i do see events show up in Gitea when it crashes when going through the cloudflare link. I think the issue may be on my ingress possibly but not 100% sure. Will investigate a bit further to make sure the act runner is still connected while i changed the service since this may not be connected correctly (hence the lack of crashing) .
GiteaMirror commented 2025-11-02 11:06:35 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 6, 2025):

Sorry, been a little while. Had other issues that was self inflicted that forced me to reset up k3s cluster. Sadly the issue is still there.

When trying, when i get a 504 error, i do see an event show up when crashing but i don't see the page that is trying to load when it gets a 504:

2025/03/06 21:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.25:59068, 200 OK in 41.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/03/06 21:41:35 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 10.42.0.12:33454, 200 OK in 150059.1ms @ events/events.go:18(events.Events) 2025/03/06 21:41:35 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.25:59068, 200 OK in 42.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)

This happens when i go through the traefik ingress. I looked into the traefik logs, but i get nothing. When stopping the act runner, it doesn't crash any more. Not sure why the act runner would have such an effect.

Any ideas? I can't think of anything else to check on kuberenetes, and the fact i see the even show up during a 503 crash suggest gitea is receiving the message but not sure how to verify this further

@NadeemSadiq commented on GitHub (Mar 6, 2025): Sorry, been a little while. Had other issues that was self inflicted that forced me to reset up k3s cluster. Sadly the issue is still there. When trying, when i get a 504 error, i do see an event show up when crashing but i don't see the page that is trying to load when it gets a 504: `2025/03/06 21:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.25:59068, 200 OK in 41.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/03/06 21:41:35 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 10.42.0.12:33454, 200 OK in 150059.1ms @ events/events.go:18(events.Events) 2025/03/06 21:41:35 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.25:59068, 200 OK in 42.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)` This happens when i go through the traefik ingress. I looked into the traefik logs, but i get nothing. When stopping the act runner, it doesn't crash any more. Not sure why the act runner would have such an effect. Any ideas? I can't think of anything else to check on kuberenetes, and the fact i see the even show up during a 503 crash suggest gitea is receiving the message but not sure how to verify this further
GiteaMirror commented 2025-11-02 11:06:35 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 6, 2025):

It's highly likely related to traefik ingress or the k3s.

If cloudflare reports 504 but you could connect to your local k3s/gitea, it means the cloudflare is not able to create TCP connection to your ingress's port. There could be various reasons, for example: exhausted system resource (connection number hard-limit), firewall rules, etc.

Still, that's just a guess. And still since only cloudflare could reproduce the problem, I think it's not really related to Gitea

@wxiaoguang commented on GitHub (Mar 6, 2025): It's highly likely related to traefik ingress or the k3s. If cloudflare reports 504 but you could connect to your local k3s/gitea, it means the cloudflare is not able to create TCP connection to your ingress's port. There could be various reasons, for example: exhausted system resource (connection number hard-limit), firewall rules, etc. Still, that's just a guess. And still since only cloudflare could reproduce the problem, I think it's not really related to Gitea
GiteaMirror commented 2025-11-02 11:06:35 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 7, 2025):

Yeah, the issue could be traefik related but unable to find any errors or anything to show it. The only thing that makes me think it is a gitea issue is when it does 504, there is an event created in gitea logs (not sure how I can access this event though) as seen in my previous post. If it was traefik issue, I would imagine gitea not showing this event log at all. Very odd that this happens only when act runner is running as well.

@NadeemSadiq commented on GitHub (Mar 7, 2025): Yeah, the issue could be traefik related but unable to find any errors or anything to show it. The only thing that makes me think it is a gitea issue is when it does 504, there is an event created in gitea logs (not sure how I can access this event though) as seen in my previous post. If it was traefik issue, I would imagine gitea not showing this event log at all. Very odd that this happens only when act runner is running as well.
GiteaMirror commented 2025-11-02 11:06:36 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 7, 2025):

The only thing that makes me think it is a gitea issue is when it does 504, there is an event created in gitea logs

I do not think it is related, because these 50x errors happen randomly.

Suppose there are 3 requests to your k3s, then 2 succeed and one fails, you still see 2 logs in Gitea's log.

The key point is that the 50x error happens "randomly". And: Gitea never responds any 50x errors other than 500. Every server failure error responded by Gitea is "500 Internal Server Error".

502 "Bad Gateway" / 504 "Gateway Timeout" errors are from your "reverse proxies".

Very odd that this happens only when act runner is running as well.

There could be various other reasons causing 502/504 problem, unstable network, exhausted system resource (connection number hard-limit), firewall rules, etc.

@wxiaoguang commented on GitHub (Mar 7, 2025): > The only thing that makes me think it is a gitea issue is when it does 504, there is an event created in gitea logs I do not think it is related, because these 50x errors happen randomly. Suppose there are 3 requests to your k3s, then 2 succeed and one fails, you still see 2 logs in Gitea's log. The key point is that the 50x error happens "randomly". And: Gitea never responds any 50x errors other than 500. Every server failure error responded by Gitea is "500 Internal Server Error". 502 "Bad Gateway" / 504 "Gateway Timeout" errors are from your "reverse proxies". ---- > Very odd that this happens only when act runner is running as well. There could be various other reasons causing 502/504 problem, unstable network, exhausted system resource (connection number hard-limit), firewall rules, etc.
GiteaMirror commented 2025-11-02 11:06:36 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 10, 2025):

Hi, I showed an example of the event popping up right after i get the gateway error. Wouldn't this mean it is hitting gitea? Not sure if there is any more/other data i can get to validate if its actually hitting gitea or not.

https://github.com/user-attachments/assets/31951698-1f38-4e66-91fa-eb1bcc4efa5e

@NadeemSadiq commented on GitHub (Mar 10, 2025): Hi, I showed an example of the event popping up right after i get the gateway error. Wouldn't this mean it is hitting gitea? Not sure if there is any more/other data i can get to validate if its actually hitting gitea or not. https://github.com/user-attachments/assets/31951698-1f38-4e66-91fa-eb1bcc4efa5e
GiteaMirror commented 2025-11-02 11:06:36 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 10, 2025):

No, it doesn't hit Gitea. the /user/events endpoint is for long-polling, but you are accessing /-/admin/xxx. There is no /-/admin/xxx request to Gitea in your screenshot.

Maybe you could try to disable the /user/events endpoint (related: Slow browsing on http2 enabled reverse proxy (apache2), long-polling /user/events blocks other requests #19265)

;; This setting determines how often the db is queried to get the latest notification counts.
;; If the browser client supports EventSource and SharedWorker, a SharedWorker will be used in preference to polling notification. Set to -1 to disable the EventSource
;EVENT_SOURCE_UPDATE_TIME = 10s
@wxiaoguang commented on GitHub (Mar 10, 2025): No, it doesn't hit Gitea. the `/user/events` endpoint is for long-polling, but you are accessing `/-/admin/xxx`. There is no `/-/admin/xxx` request to Gitea in your screenshot. Maybe you could try to disable the `/user/events` endpoint (related: Slow browsing on http2 enabled reverse proxy (apache2), long-polling /user/events blocks other requests #19265) ``` ;; This setting determines how often the db is queried to get the latest notification counts. ;; If the browser client supports EventSource and SharedWorker, a SharedWorker will be used in preference to polling notification. Set to -1 to disable the EventSource ;EVENT_SOURCE_UPDATE_TIME = 10s ```
GiteaMirror commented 2025-11-02 11:06:37 -06:00
Author
Owner
Copy Link

@NadeemSadiq commented on GitHub (Mar 10, 2025):

https://github.com/user-attachments/assets/1921268d-3640-4029-893d-facc3f83eada

I removed the ingress (traefik) and I am now talking directly to the gitea server. Web pages seem fine and I am unable to see any errors on the page itself.

However, when pulling, i randomly get errors. See the video for an example of when i stop the act runner service and when i start it and how the git pull reacts to that.

@NadeemSadiq commented on GitHub (Mar 10, 2025): https://github.com/user-attachments/assets/1921268d-3640-4029-893d-facc3f83eada I removed the ingress (traefik) and I am now talking directly to the gitea server. Web pages seem fine and I am unable to see any errors on the page itself. However, when pulling, i randomly get errors. See the video for an example of when i stop the act runner service and when i start it and how the git pull reacts to that.
GiteaMirror commented 2025-11-02 11:06:37 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Mar 10, 2025):

"Failed to connect to ..." means a network error. Not Gitea's error. Gitea didn't receive that request.

Again, suggest a third time: check your network and system.

@wxiaoguang commented on GitHub (Mar 10, 2025): "Failed to connect to ..." means a network error. Not Gitea's error. Gitea didn't receive that request. Again, suggest a third time: check your network and system.
GiteaMirror commented 2025-11-02 11:06:37 -06:00
Author
Owner
Copy Link

@GiteaBot commented on GitHub (Apr 9, 2025):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Apr 9, 2025): We close issues that need feedback from the author if there were no new comments for a month. :tea:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label issue/needs-feedback type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#14212
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?