mirror of
https://github.com/go-gitea/gitea.git
synced 2026-03-09 21:10:00 -05:00
Long gitea-actions log stops output at ~3hours and job failure #13776
Closed
opened 2025-11-02 10:52:51 -06:00 by GiteaMirror
·
31 comments
No Branch/Tag Specified
main
release/v1.25
release/v1.24
release/v1.23
release/v1.22
release/v1.21
release/v1.20
release/v1.19
release/v1.18
release/v1.17
release/v1.16
release/v1.15
release/v1.14
release/v1.13
release/v1.12
release/v1.11
release/v1.10
release/v1.9
release/v1.8
v1.25.3
v1.25.2
v1.25.1
v1.25.0
v1.24.7
v1.25.0-rc0
v1.26.0-dev
v1.24.6
v1.24.5
v1.24.4
v1.24.3
v1.24.2
v1.24.1
v1.24.0
v1.23.8
v1.24.0-rc0
v1.25.0-dev
v1.23.7
v1.23.6
v1.23.5
v1.23.4
v1.23.3
v1.23.2
v1.23.1
v1.23.0
v1.23.0-rc0
v1.24.0-dev
v1.22.6
v1.22.5
v1.22.4
v1.22.3
v1.22.2
v1.22.1
v1.22.0
v1.23.0-dev
v1.22.0-rc1
v1.21.11
v1.22.0-rc0
v1.21.10
v1.21.9
v1.21.8
v1.21.7
v1.21.6
v1.21.5
v1.21.4
v1.21.3
v1.21.2
v1.20.6
v1.21.1
v1.21.0
v1.21.0-rc2
v1.21.0-rc1
v1.20.5
v1.22.0-dev
v1.21.0-rc0
v1.20.4
v1.20.3
v1.20.2
v1.20.1
v1.20.0
v1.19.4
v1.21.0-dev
v1.20.0-rc2
v1.20.0-rc1
v1.20.0-rc0
v1.19.3
v1.19.2
v1.19.1
v1.19.0
v1.19.0-rc1
v1.20.0-dev
v1.19.0-rc0
v1.18.5
v1.18.4
v1.18.3
v1.18.2
v1.18.1
v1.18.0
v1.17.4
v1.18.0-rc1
v1.19.0-dev
v1.18.0-rc0
v1.17.3
v1.17.2
v1.17.1
v1.17.0
v1.17.0-rc2
v1.16.9
v1.17.0-rc1
v1.18.0-dev
v1.16.8
v1.16.7
v1.16.6
v1.16.5
v1.16.4
v1.16.3
v1.16.2
v1.16.1
v1.16.0
v1.15.11
v1.17.0-dev
v1.16.0-rc1
v1.15.10
v1.15.9
v1.15.8
v1.15.7
v1.15.6
v1.15.5
v1.15.4
v1.15.3
v1.15.2
v1.15.1
v1.14.7
v1.15.0
v1.15.0-rc3
v1.14.6
v1.15.0-rc2
v1.14.5
v1.16.0-dev
v1.15.0-rc1
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.7
v1.14.0-rc2
v1.13.6
v1.13.5
v1.14.0-rc1
v1.15.0-dev
v1.13.4
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.6
v1.13.0-rc2
v1.14.0-dev
v1.13.0-rc1
v1.12.5
v1.12.4
v1.12.3
v1.12.2
v1.12.1
v1.11.8
v1.12.0
v1.11.7
v1.12.0-rc2
v1.11.6
v1.12.0-rc1
v1.13.0-dev
v1.11.5
v1.11.4
v1.11.3
v1.10.6
v1.12.0-dev
v1.11.2
v1.10.5
v1.11.1
v1.10.4
v1.11.0
v1.11.0-rc2
v1.10.3
v1.11.0-rc1
v1.10.2
v1.10.1
v1.10.0
v1.9.6
v1.9.5
v1.10.0-rc2
v1.11.0-dev
v1.10.0-rc1
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.9.0-rc2
v1.10.0-dev
v1.9.0-rc1
v1.8.3
v1.8.2
v1.8.1
v1.8.0
v1.8.0-rc3
v1.7.6
v1.8.0-rc2
v1.7.5
v1.8.0-rc1
v1.9.0-dev
v1.7.4
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.7.0-rc3
v1.6.4
v1.7.0-rc2
v1.6.3
v1.7.0-rc1
v1.7.0-dev
v1.6.2
v1.6.1
v1.6.0
v1.6.0-rc2
v1.5.3
v1.6.0-rc1
v1.6.0-dev
v1.5.2
v1.5.1
v1.5.0
v1.5.0-rc2
v1.5.0-rc1
v1.5.0-dev
v1.4.3
v1.4.2
v1.4.1
v1.4.0
v1.4.0-rc3
v1.4.0-rc2
v1.3.3
v1.4.0-rc1
v1.3.2
v1.3.1
v1.3.0
v1.3.0-rc2
v1.3.0-rc1
v1.2.3
v1.2.2
v1.2.1
v1.2.0
v1.2.0-rc3
v1.2.0-rc2
v1.1.4
v1.2.0-rc1
v1.1.3
v1.1.2
v1.1.1
v1.1.0
v1.0.2
v1.0.1
v1.0.0
v0.9.99
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
Mirrored from GitHub Pull Request
Milestone
No items
No Milestone
Projects
Clear projects
No project
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/gitea#13776
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @RobertBerger on GitHub (Dec 2, 2024).
Description
This[1], which calls this[2] keeps on running
... but call-use-uninative-core-image-minimal-local-source-mirror stops around 3 hrs with logs and I can not see the error.
[1] https://github.com/RobertBerger/gitea-yocto-build/blob/master/.gitea/workflows/00.01-release.yml
[2] https://github.com/RobertBerger/gitea-yocto-build/blob/master/.gitea/workflows/03-use-uninative-core-image-minimal-local-source-mirror.yml
I already changed ENDLESS_TASK_TIMEOUT = 6h
and added timeout: 6h to my workflow.
gitea/gitea:1.22.4
gitea/act_runner:0.2.11
both are containers
Gitea Version
gitea/gitea:1.22.4
Can you reproduce the bug on the Gitea demo site?
Yes
Log Gist
https://drive.google.com/file/d/1-bJ2tCk3zqCteGW8dh_TuGUh2vqGoLZn/view?usp=sharing
Screenshots
No response
Git Version
whatever came with the docker container
Operating System
Linux/docker container
How are you running Gitea?
Database
None
@RobertBerger commented on GitHub (Dec 2, 2024):
Note: I did not try on the Gitea demo site
@Zettat123 commented on GitHub (Dec 3, 2024):
Thank you for reporting this issue.
I noticed that you added
timeout: 6hto your workflow. Buttimeoutis a configuration item of act_runner and should not be added to the workflow file. Also, Gitea does not support the timeout for jobs (see this doc). So I think setting a timeout will not help with this issue.From the log, I found that the log ended at
Compress Artifacts after buildstep without errors. I want to know when the log stops, is the task status "Running" or has it turned into "Failure"? If it has turned into "Failure", has it been running for longer thanENDLESS_TASK_TIMEOUT?@RobertBerger commented on GitHub (Dec 3, 2024):
As you said, I also see no error. The
Compress Artifacts after buildstep just takes the build artefacts and makes tarballs out of them. I tried splitting it into more tarballs, but this does not help either.As you can see from the yellow (spinning) clock one might think it still runs after 10+ hours.
The job failed:
ENDLESS_TASK_TIMEOUT = 6h
... but it fails pretty much exactly at 3h.
Is there any way to see if it read ENDLESS_TASK_TIMEOUT = 6h?
Here is also my app.ini
app.ini.txt
Where should I set the timeout of act_runner?
@RobertBerger commented on GitHub (Dec 3, 2024):
Maybe let me add. If I don't build the 3 things (
uninative, core-image-minimal, core-image-sato-sdk) but onlyuninative, core-image-sato-sdkit works, since it takes significantly less time.@RobertBerger commented on GitHub (Dec 3, 2024):
It looks like it does what it's supposed to do, but gets stuck after 3 hours somehow. Let me do further experiments.
@RobertBerger commented on GitHub (Dec 3, 2024):
The next build step stops at 3 hours with a failure, although there is no obvious failure visible.
@Zettat123 commented on GitHub (Dec 4, 2024):
You could configure your act_runner with a configuration file by following this instruction.
@Zettat123 commented on GitHub (Dec 4, 2024):
I see. Building
core-image-sato-sdktook more than 2 hours, but eventually succeeded. How long does theCompress Artifacts after buildstep usually take?And I also found the
ENDLESS_TASK_TIMEOUTconfiguration seems to not work. Several tasks took significantly longer thanENDLESS_TASK_TIMEOUT. I'm going to take a look at this issue.@Zettat123 commented on GitHub (Dec 4, 2024):
One more thing I would like to know, are the logs downloaded from the UI or excerpted from act_runner's log file?
@RobertBerger commented on GitHub (Dec 4, 2024):
From the UI - download log.
@Zettat123 commented on GitHub (Dec 5, 2024):
Could you get some logs from the act_runner? Maybe we can find some error messages from them.
@RobertBerger commented on GitHub (Dec 5, 2024):
This is my config.yaml
config.yaml.txt
Do you want me to change the log level from
infoto something else?Do you want me modify it to get more logs?
@RobertBerger commented on GitHub (Dec 5, 2024):
At the moment I suspect https://github.com/appleboy/scp-action to explode in some mysterious way, but I don't see the exact problem anywhere:
So I am tring with https://github.com/marketplace/actions/copy-via-ssh instead.
Let's see.
https://github.com/appleboy/scp-action tars my tarballs again, which is not needed and they are some tens of Gigs in size.
@RobertBerger commented on GitHub (Dec 5, 2024):
If you think it helps, I guess I could also give you remote access to my machine, or we can do a remote live session.
@Akshu-305 commented on GitHub (Dec 6, 2024):
Even I'm facing the similar issue(pipeline stopped after 3hrs) after changing the ENDLESS_TASK_TIMEOUT under [actions] in app.ini of gitea and changed runner.timeout in the configuration file of gitea act runner. Is there anything else I should change/do?
@RobertBerger commented on GitHub (Dec 6, 2024):
I am still stuck there. I suspected one of my steps to be the problem, but I am still investigating. At the moment I try to break everything down to something smaller than 3 hours, but that is not easy, since I build a complete Linux OS for an Embedded target from sources and if I want to build everything only from sources without any special tricks on 16 cores this takes time.
@RobertBerger commented on GitHub (Dec 6, 2024):
In my case, I'm not entirely sure who to blame. It might be the scp action ???
What happens is that my job runs 3h12m13s and fails.
At the moment it fails, it tries to compress/scp this file (which is already compressed)
Please note that it is 9.8G !!!
On the server I still see some scp process
I might be able to find a workaround by making these huge releases only on this specific server and instead of scp-ing stuff from the container to the server to copy things around.
@RobertBerger commented on GitHub (Dec 8, 2024):
Looking again at the logs, I see this
ENV ==> [RUNNER_TOOL_CACHE=/opt/hostedtoolcache RUNNER_OS=Linux RUNNER_ARCH=X64 RUNNER_TEMP=/tmp LANG=C.UTF-8] 🐳 docker run image=reliableembeddedsystems/poky-container:2024-03-11-master-local-icecc-ub***-doc-spdx platform= entrypoint=["/bin/sleep" "10800"] cmd=[] network="GITEA-ACTIONS-TASK-614_WORKFLOW-release-use-uninative-core-image-minimal-local-source-mirror_JOB-release-use-uninative-core-image-minimal-local-source-mirror-job-release-use-uninative-core-image-minimal-local-source-mirror-job-network"entrypoint=["/bin/sleep" "10800"]would be 3 hours. I am not quite sure where this comes from, but this might be an issue.@RobertBerger commented on GitHub (Dec 10, 2024):
... and looking at this[1] it makes me think, that it comes from the gitea act runner. Can someone confirm and maybe fix this?
[1] https://forum.gitea.com/t/oci-runtime-exec-failed-unable-to-start-container-process/8299/3
just search for sleep and you will find:
@lunny commented on GitHub (Dec 10, 2024):
Have you tried to change the configuration file? ref https://docs.gitea.com/usage/actions/act-runner#configuration
You can try to adjust the
timeout.@RobertBerger commented on GitHub (Dec 10, 2024):
Here is my current setup -- please review and comment.
docker compose/portainersetup:Are those the correct act runners?
The other one is mine, which I need to build my stuff.
My app.ini contains
config.yaml:
On this webpage: https://docs.gitea.com/usage/actions/act-runner#configuration
I searched for
timeout, but can not find this:In my 'config.yaml' file, I have, as you can see above:
My top level job is this: https://github.com/RobertBerger/gitea-yocto-build/blob/master/.gitea/workflows/00.01-release.yml
It calls stuff from here:
https://github.com/RobertBerger/gitea-yocto-build/tree/master/.gitea/workflows
There I defined:
With this setup, I see in the logs:
or instead of
sleep 10799sleep 10800, which is 3hAm I missing something?
@Zettat123 commented on GitHub (Dec 11, 2024):
You can find the
timeoutconfiguration in config.example.yamlrunner-latest-182is not a valid configuration key. Could you please changerunner-latest-182torunner?@Akshu-305 commented on GitHub (Dec 11, 2024):
@Zettat123 Even I used runner.timeout in my configuration still it doesn't work for me. Should I add anything else?
@Zettat123 commented on GitHub (Dec 11, 2024):
Thanks for your feedback. I think just specifying the
timeoutis enough, no need to add any additional configuration. I will do some tests to check if there are any bugs related to thetimeoutconfiguration.@RobertBerger commented on GitHub (Dec 11, 2024):
Hi,
Good catch!
I did change this in my config.yaml, restarted everything, but I still get:
entrypoint=["/bin/sleep" "10800"]@Zettat123 commented on GitHub (Dec 13, 2024):
This is weird. It works on my act_runner.
@Zettat123 commented on GitHub (Dec 13, 2024):
Ah, I think I know where the issue is.
The
CONFIG_FILEenv of act_runner in yourcompose.yamlisIt should be
@RobertBerger commented on GitHub (Dec 13, 2024):
As far as I can see, this fixed it. Thanks a lot!
@Zettat123 commented on GitHub (Dec 16, 2024):
I'll close this issue since it has been resolved. Please reopen it if you need more help.
@Akshu-305 commented on GitHub (Dec 16, 2024):
@Zettat123 in my act runner Deployment yaml I use
What should I do?
@RobertBerger commented on GitHub (Dec 16, 2024):
@Akshu-305
What does your
entrypoint=["/bin/sleep" "10800"]return? Something like10800?Maybe your config file is wrong or not picked up?