github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-07-10 16:44:50 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Gitea action runner fails to start randomly on low-end hardware #14371

New Issue
Open
opened 2025-11-02 11:11:07 -06:00 by GiteaMirror · 15 comments
GiteaMirror commented 2025-11-02 11:11:07 -06:00
Owner
Copy Link

Originally created by @repo-ooo on GitHub (Apr 14, 2025).

Description

I'm running gitea and gitea runner in docker on low-end hardware (Intel J1900). The actions job is very hard to start. Mostly, the actions run page is stucked on "Set up job" without any output, and finally failed after 10~12 minutes.

I have to repeatedly click cancel and rerun all jobs on the actions run page, and quickly check if the job is successfully sent to runner via docker log gitea-runner. If I'm lucky, it might start successfully on the first try; if not, it could fail more than 20 times in a row.

I believe this problem is due to the low-end hardware. When I'm running git clone https://git.mydomain.com/a/b, It may also fail at the first time, and succeed at the second time. Unfortunately, it's easy to run git clone 2 times, but it's hard to cancel and rerun action jobs again and again.

Is it possible to add some retry policy when starting an action job?

2025/04/14 19:27:15 ...s/process/manager.go:188:Add() [T] Start 67fcf113: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:15 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 127.0.0.1:48944, 200 OK in 2495.5ms @ healthcheck/check.go:67(healthcheck.Check)
2025/04/14 19:27:16 ...s/process/manager.go:231:remove() [T] Done 67fcf111-5: GET: /api/healthz
2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /admini/repo1/actions/runs/26/rerun for 192.168.1.100:0, elapsed 3563.8ms @ actions/view.go:366(actions.Rerun)
2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3520.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:18 ...s/process/manager.go:188:Add() [T] Start 67fcf116: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:18 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:19 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, 500 Internal Server Error in 5548.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf111-4: POST: /api/actions/runner.v1.RunnerService/FetchTask
2025/04/14 19:27:19 ...dels/actions/task.go:246:CreateTaskForRunner() [T] runner labels: [linux_amd64 ubuntu-22.04]
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin (from 67fcf111) (normal)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-parse
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git] (from 67fcf111) (normal)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git]
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-3: /usr/bin/git cat-file --batch [repo_path: /data/git/repositories/admini/repo1.git] (modules/git/repo_base_nogogit.go:65) (from 67fcf111) (normal)
2025/04/14 19:27:20 ...git/commit_status.go:480:NewCommitStatus() [D] NewCommitStatus[/data/git/repositories/admini/repo1.git, 4315663afe45a375e15bb0c8ae0ea0379eabf401]: 3
2025/04/14 19:27:20 ...s/process/manager.go:188:Add() [T] Start 67fcf118: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:20 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:22 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:22 ...ons/commit_status.go:30:CreateCommitStatus() [E] Failed to create commit status for job 26: GetLatestCommitStatus: context canceled

Possible related issues:
#32348, #33492, #33497

Gitea Version

1.23.6

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Image

Git Version

No response

Operating System

Synology DSM 7.2.2, kernel version 4.4.302+

How are you running Gitea?

docker-compose up -d

services:
  server:
    image: gitea/gitea:1.23.6
    container_name: gitea
    environment:
      - USER=git
      - USER_UID=1027
      - USER_GID=100
      - GITEA_CUSTOM=/data/gitea
      - DOMAIN=git.mydomain.com
      - SSH_DOMAIN=git.mydomain.com
      - ROOT_URL=https://git.mydomain.com/
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "5030:3000"
      - "5022:22"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s

  runner:
    image: gitea/act_runner:nightly # v0.2.11+15-gb1ae30d
    container_name: gitea-runner
    depends_on:
      server:
        condition: service_healthy
    environment:
      CONFIG_FILE: /config.yaml
      GITEA_INSTANCE_URL: http://server:3000/
      GITEA_RUNNER_REGISTRATION_TOKEN: <some token>
      GITEA_RUNNER_NAME: host-docker-runner
      GITEA_RUNNER_LABELS: "linux_amd64:docker://node:16.20.2-slim,ubuntu-22.04:docker://gitea/runner-images:ubuntu-22.04-v25.04.01"
    volumes:
      - ./config.yaml:/config.yaml
      - ./runner:/data
      - /var/run/docker.sock:/var/run/docker.sock

The domain git.mydomain.com is resolved to 192.168.0.100, and reverse proxied by Synology DSM to port :5030.

Database

SQLite

Originally created by @repo-ooo on GitHub (Apr 14, 2025). ### Description I'm running gitea and gitea runner in docker on low-end hardware (Intel J1900). The actions job is very hard to start. Mostly, the actions run page is stucked on "Set up job" without any output, and finally failed after 10~12 minutes. I have to repeatedly click `cancel` and `rerun all jobs` on the actions run page, and quickly check if the job is successfully sent to runner via `docker log gitea-runner`. If I'm lucky, it might start successfully on the first try; if not, it could fail more than 20 times in a row. I believe this problem is due to the low-end hardware. When I'm running `git clone https://git.mydomain.com/a/b`, It may also fail at the first time, and succeed at the second time. Unfortunately, it's easy to run `git clone` 2 times, but it's hard to cancel and rerun action jobs again and again. Is it possible to add some retry policy when starting an action job? ```log 2025/04/14 19:27:15 ...s/process/manager.go:188:Add() [T] Start 67fcf113: POST: /api/actions/runner.v1.RunnerService/FetchTask (request) 2025/04/14 19:27:15 ...eb/routing/logger.go:47:func1() [T] router: started POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0 2025/04/14 19:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 127.0.0.1:48944, 200 OK in 2495.5ms @ healthcheck/check.go:67(healthcheck.Check) 2025/04/14 19:27:16 ...s/process/manager.go:231:remove() [T] Done 67fcf111-5: GET: /api/healthz 2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow POST /admini/repo1/actions/runs/26/rerun for 192.168.1.100:0, elapsed 3563.8ms @ actions/view.go:366(actions.Rerun) 2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3520.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/04/14 19:27:18 ...s/process/manager.go:188:Add() [T] Start 67fcf116: POST: /api/actions/runner.v1.RunnerService/FetchTask (request) 2025/04/14 19:27:18 ...eb/routing/logger.go:47:func1() [T] router: started POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0 2025/04/14 19:27:19 ...eb/routing/logger.go:68:func1() [W] router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/04/14 19:27:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, 500 Internal Server Error in 5548.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf111-4: POST: /api/actions/runner.v1.RunnerService/FetchTask 2025/04/14 19:27:19 ...dels/actions/task.go:246:CreateTaskForRunner() [T] runner labels: [linux_amd64 ubuntu-22.04] 2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin 2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin (from 67fcf111) (normal) 2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin 2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-parse 2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git] (from 67fcf111) (normal) 2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git] 2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch 2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-3: /usr/bin/git cat-file --batch [repo_path: /data/git/repositories/admini/repo1.git] (modules/git/repo_base_nogogit.go:65) (from 67fcf111) (normal) 2025/04/14 19:27:20 ...git/commit_status.go:480:NewCommitStatus() [D] NewCommitStatus[/data/git/repositories/admini/repo1.git, 4315663afe45a375e15bb0c8ae0ea0379eabf401]: 3 2025/04/14 19:27:20 ...s/process/manager.go:188:Add() [T] Start 67fcf118: POST: /api/actions/runner.v1.RunnerService/FetchTask (request) 2025/04/14 19:27:20 ...eb/routing/logger.go:47:func1() [T] router: started POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0 2025/04/14 19:27:22 ...eb/routing/logger.go:68:func1() [W] router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/04/14 19:27:22 ...ons/commit_status.go:30:CreateCommitStatus() [E] Failed to create commit status for job 26: GetLatestCommitStatus: context canceled ``` Possible related issues: #32348, #33492, #33497 ### Gitea Version 1.23.6 ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots ![Image](https://github.com/user-attachments/assets/fcbd4e61-a974-44cd-b6c0-5c8a8862d603) ### Git Version _No response_ ### Operating System Synology DSM 7.2.2, kernel version 4.4.302+ ### How are you running Gitea? `docker-compose up -d` ```yaml services: server: image: gitea/gitea:1.23.6 container_name: gitea environment: - USER=git - USER_UID=1027 - USER_GID=100 - GITEA_CUSTOM=/data/gitea - DOMAIN=git.mydomain.com - SSH_DOMAIN=git.mydomain.com - ROOT_URL=https://git.mydomain.com/ volumes: - ./gitea:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ports: - "5030:3000" - "5022:22" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"] interval: 30s timeout: 5s retries: 5 start_period: 30s runner: image: gitea/act_runner:nightly # v0.2.11+15-gb1ae30d container_name: gitea-runner depends_on: server: condition: service_healthy environment: CONFIG_FILE: /config.yaml GITEA_INSTANCE_URL: http://server:3000/ GITEA_RUNNER_REGISTRATION_TOKEN: <some token> GITEA_RUNNER_NAME: host-docker-runner GITEA_RUNNER_LABELS: "linux_amd64:docker://node:16.20.2-slim,ubuntu-22.04:docker://gitea/runner-images:ubuntu-22.04-v25.04.01" volumes: - ./config.yaml:/config.yaml - ./runner:/data - /var/run/docker.sock:/var/run/docker.sock ``` The domain `git.mydomain.com` is resolved to `192.168.0.100`, and reverse proxied by Synology DSM to port `:5030`. ### Database SQLite
GiteaMirror added the performance/speedtopic/gitea-actionsissue/workaround labels 2025-11-02 11:11:07 -06:00
GiteaMirror commented 2025-11-02 11:11:07 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (May 10, 2025):

Looks like it's a performance problem for low-end hardware rather than a bug.

@lunny commented on GitHub (May 10, 2025): Looks like it's a performance problem for low-end hardware rather than a bug.
GiteaMirror commented 2025-11-02 11:11:07 -06:00
Author
Owner
Copy Link

@blaizard commented on GitHub (Jul 1, 2025):

I have the same issue, running on Synology hardware as well but different CPU: INTEL Celeron J4125.
The issue started to show up 8 months ago (most likely after updating to v1.22.4), before that I never had any issues (note that my gitea containers are always up to date as I use watchtower).

@blaizard commented on GitHub (Jul 1, 2025): I have the same issue, running on Synology hardware as well but different CPU: INTEL Celeron J4125. The issue started to show up 8 months ago (most likely after updating to [v1.22.4](https://github.com/go-gitea/gitea/releases/tag/v1.22.4)), before that I never had any issues (note that my gitea containers are always up to date as I use watchtower).
GiteaMirror commented 2025-11-02 11:11:08 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (Jul 1, 2025):

Have you enabled code search in your instance? I guess it maybe related to bleve engine.

@lunny commented on GitHub (Jul 1, 2025): Have you enabled code search in your instance? I guess it maybe related to bleve engine.
GiteaMirror commented 2025-11-02 11:11:08 -06:00
Author
Owner
Copy Link

@blaizard commented on GitHub (Jul 1, 2025):

Have you enabled code search in your instance? I guess it maybe related to bleve engine.

It's not enabled, REPO_INDEXER_ENABLED defaults to false. I also tried implicitly setting it to false and I still have the issue.

@blaizard commented on GitHub (Jul 1, 2025): > Have you enabled code search in your instance? I guess it maybe related to bleve engine. It's not enabled, `REPO_INDEXER_ENABLED` defaults to false. I also tried implicitly setting it to false and I still have the issue.
GiteaMirror commented 2025-11-02 11:11:08 -06:00
Author
Owner
Copy Link

@ericLemanissier commented on GitHub (Jul 2, 2025):

I have the same problem on a synology with the same CPU, with indexing disabled

@ericLemanissier commented on GitHub (Jul 2, 2025): I have the same problem on a synology with the same CPU, with indexing disabled
GiteaMirror commented 2025-11-02 11:11:08 -06:00
Author
Owner
Copy Link

@abrain commented on GitHub (Jul 17, 2025):

We see this effect with not-so-low hardware. Also Gitea, the database and the runner are on separate hosts.

But as far as I can see, it started exactly with the update from 1.23.7 to 1.24.2.

Oddly enough, only one specific repository seems to be affected. Due to a previously used cron schedule, this repo has over 30000 runs. I wonder if there is a performance impact related to the number of runs, that somehow increases the chance of this bug happening. The Actions tab surely loads slower than those of other repos.

@abrain commented on GitHub (Jul 17, 2025): We see this effect with not-so-low hardware. Also Gitea, the database and the runner are on separate hosts. But as far as I can see, it started exactly with the update from 1.23.7 to 1.24.2. Oddly enough, only one specific repository seems to be affected. Due to a previously used cron schedule, this repo has over 30000 runs. I wonder if there is a performance impact related to the number of runs, that somehow increases the chance of this bug happening. The Actions tab surely loads slower than those of other repos.
GiteaMirror commented 2025-11-02 11:11:08 -06:00
Author
Owner
Copy Link

@ericLemanissier commented on GitHub (Jul 25, 2025):

I'm not sure if it helps, but this problem happens both with sqlite and postgres database, and on a repo which has only 350 runs.

at the time the job tried to start, with postgres db, gitea logs says :

2025/07/25 09:27:00 services/webhook/notifier.go:874:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: e3135e13b30ea342dafe4c711cebe2e10c3be6a6]: context canceled
2025/07/25 09:27:00 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 95623: GetBranchCommit[gitea]: context canceled
2025/07/25 09:27:00 services/webhook/notifier.go:961:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled

I'm not sure it related ?

@ericLemanissier commented on GitHub (Jul 25, 2025): I'm not sure if it helps, but this problem happens both with sqlite and postgres database, and on a repo which has only 350 runs. at the time the job tried to start, with postgres db, gitea logs says : ``` 2025/07/25 09:27:00 services/webhook/notifier.go:874:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: e3135e13b30ea342dafe4c711cebe2e10c3be6a6]: context canceled 2025/07/25 09:27:00 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 95623: GetBranchCommit[gitea]: context canceled 2025/07/25 09:27:00 services/webhook/notifier.go:961:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled ``` I'm not sure it related ?
GiteaMirror commented 2025-11-02 11:11:09 -06:00
Author
Owner
Copy Link

@techknowlogick commented on GitHub (Jul 26, 2025):

Any specific actions you are using in your workflows? I've seen specifically the Unity one cause apparent hangs due to its size and memory required for git to clone it.

@techknowlogick commented on GitHub (Jul 26, 2025): Any specific actions you are using in your workflows? I've seen specifically the Unity one cause apparent hangs due to its size and memory required for git to clone it.
GiteaMirror commented 2025-11-02 11:11:09 -06:00
Author
Owner
Copy Link

@ericLemanissier commented on GitHub (Jul 31, 2025):

Correction on my message above: only the first job failed (after boot). After a few days, all subensequent jobs succeeded since migration to postgresql.

@ericLemanissier commented on GitHub (Jul 31, 2025): Correction on my message above: only the first job failed (after boot). After a few days, all subensequent jobs succeeded since migration to postgresql.
GiteaMirror commented 2025-11-02 11:11:09 -06:00
Author
Owner
Copy Link

@ericLemanissier commented on GitHub (Aug 11, 2025):

after two weeks, I noticed at least two failures (with postgres db). As for the other failures, the workflow hangs at the very start, and does not even takes a spot in the runner, which is able to run another action while this one hangs. There is simply no log, so it seems to hang before checking out any action. Trying to download the raw log fails with DownloadActionsRunJobLogsWithIndex, OpenLogs: dbfs open "actions_log/el/windmill/a1/76449.log.zst": file does not exist
This workfow is using:

  • actions/setup-node@v4
  • actions/checkout@v4
  • oven-sh/setup-bun@v2
@ericLemanissier commented on GitHub (Aug 11, 2025): after two weeks, I noticed at least two failures (with postgres db). As for the other failures, the workflow hangs at the very start, and does not even takes a spot in the runner, which is able to run another action while this one hangs. There is simply no log, so it seems to hang before checking out any action. Trying to download the raw log fails with `DownloadActionsRunJobLogsWithIndex, OpenLogs: dbfs open "actions_log/el/windmill/a1/76449.log.zst": file does not exist` This workfow is using: - actions/setup-node@v4 - actions/checkout@v4 - oven-sh/setup-bun@v2
GiteaMirror commented 2025-11-02 11:11:10 -06:00
Author
Owner
Copy Link

@lichuanqi commented on GitHub (Aug 29, 2025):

same issue

@lichuanqi commented on GitHub (Aug 29, 2025): same issue
GiteaMirror commented 2025-11-02 11:11:10 -06:00
Author
Owner
Copy Link

@abrain commented on GitHub (Sep 8, 2025):

I adjusted the runner config a few days ago and did not witness this issue since. The config now reads:

  # The timeout for fetching the job from the Gitea instance.
  fetch_timeout: 10s
  # The interval for fetching the job from the Gitea instance.
  fetch_interval: 5s
@abrain commented on GitHub (Sep 8, 2025): I adjusted the runner config a few days ago and did not witness this issue since. The config now reads: ``` # The timeout for fetching the job from the Gitea instance. fetch_timeout: 10s # The interval for fetching the job from the Gitea instance. fetch_interval: 5s ```
GiteaMirror commented 2025-11-02 11:11:10 -06:00
Author
Owner
Copy Link

@abrain commented on GitHub (Sep 29, 2025):

TL;DR: A database query takes a long time and the runner cancels the request. Adding a database index significantly reduces the performance impact. The fact that certain tasks never run might be a manifestation of #33492, if the request is cancelled after picking a task, but before sending the response.

You can see in the reverse proxy logs (in my case nginx), that the runner sometimes closes the connection before getting a response.

[29/Sep/2025:02:34:35 +0200] "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/2.0" 499 0 "-" "connect-go/1.16.2 (go1.24.6)"
[29/Sep/2025:02:34:48 +0200] "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/2.0" 499 0 "-" "connect-go/1.16.2 (go1.24.6)"

On the Gitea side you see slow request and database query warnings:

2025/09/29 02:34:27 HTTPRequest [W] router: slow      POST /api/v1/repos/REDACTED/ansible-demo-deployment/pulls for REDACTED:0, elapsed 3663.8ms @ repo/pull.go:366(repo.CreatePullRequest)
2025/09/29 02:34:30 .../actions/notifier_helper.go:356:handleWorkflows() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 61 49 277395 277395 sha1 340 340 success failure cancelled skipped 340] - 5.383375477s
2025/09/29 02:34:34 HTTPRequest [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for REDACTED:0, elapsed 3367.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/09/29 02:34:35 models/actions/run_job.go:160:UpdateRunJob() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 61 49 277395 277395 sha1 340 340 success failure cancelled skipped 340] - 5.006670551s
2025/09/29 02:34:35 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 34939: load run: find primary languages: context canceled
2025/09/29 02:34:35 models/system/setting.go:134:(*dbConfigCachedGetter).GetRevision() [E] Unable to get all settings: context canceled
2025/09/29 02:34:35 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled
2025/09/29 02:34:36 HTTPRequest [W] router: slow      POST /api/v1/repos/REDACTED/ansible-demo-deployment/pulls for REDACTED:0, elapsed 3377.1ms @ repo/pull.go:366(repo.CreatePullRequest)
2025/09/29 02:34:42 .../actions/notifier_helper.go:356:handleWorkflows() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 62 49 280287 280287 sha1 340 340 success failure cancelled skipped 340] - 5.058891808s
2025/09/29 02:34:47 HTTPRequest [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for REDACTED:0, elapsed 3361.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/09/29 02:34:48 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 34940: load run: find primary languages: context canceled
2025/09/29 02:34:48 models/system/setting.go:134:(*dbConfigCachedGetter).GetRevision() [E] Unable to get all settings: context canceled
2025/09/29 02:34:48 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled

The slow query log on the MariaDB server shows those queries as well. Executing the queries directly on the server lead me to the issue. The problematic part is the sub-query for num_closed_action_runs.

MariaDB [gitea]> SELECT count(*) FROM action_run WHERE repo_id=340;
+----------+
| count(*) |
+----------+
|    30908 |
+----------+
1 row in set (0.014 sec)

MariaDB [gitea]> SELECT count(*) FROM action_run WHERE repo_id=340 AND status IN (1,2,3,4);
+----------+
| count(*) |
+----------+
|    30908 |
+----------+
1 row in set (3.236 sec)

You see the steep increase in query time, as soon as the status condition is added. There is no index yet that would speed up this query:

MariaDB [gitea]> ANALYZE SELECT count(*) FROM action_run WHERE repo_id=340 AND status IN (1,2,3,4);
+------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+
| id   | select_type | table      | type | possible_keys                                                          | key                    | key_len | ref   | rows | r_rows   | filtered | r_filtered | Extra       |
+------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+
|    1 | SIMPLE      | action_run | ref  | UQE_action_run_repo_index,IDX_action_run_repo_id,IDX_action_run_status | IDX_action_run_repo_id | 9       | const | 9862 | 30908.00 |    30.75 |     100.00 | Using where |
+------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+
1 row in set (2.735 sec)

After adding

CREATE INDEX `IDX_action_run_repo_status` ON action_run (`repo_id`, `status`);

this query now executes in 30 ms.

I tested again with more aggressive polling from the runner and with shorter timeouts, which seems fine now.

So I'd like to suggest to add this index. Maybe one of the other affected users here can confirm the effectiveness of the solution.

@abrain commented on GitHub (Sep 29, 2025): TL;DR: A database query takes a long time and the runner cancels the request. Adding a database index significantly reduces the performance impact. The fact that certain tasks never run might be a manifestation of #33492, if the request is cancelled after picking a task, but before sending the response. You can see in the reverse proxy logs (in my case nginx), that the runner sometimes closes the connection before getting a response. ``` [29/Sep/2025:02:34:35 +0200] "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/2.0" 499 0 "-" "connect-go/1.16.2 (go1.24.6)" [29/Sep/2025:02:34:48 +0200] "POST /api/actions/runner.v1.RunnerService/FetchTask HTTP/2.0" 499 0 "-" "connect-go/1.16.2 (go1.24.6)" ``` On the Gitea side you see slow request and database query warnings: ``` 2025/09/29 02:34:27 HTTPRequest [W] router: slow POST /api/v1/repos/REDACTED/ansible-demo-deployment/pulls for REDACTED:0, elapsed 3663.8ms @ repo/pull.go:366(repo.CreatePullRequest) 2025/09/29 02:34:30 .../actions/notifier_helper.go:356:handleWorkflows() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 61 49 277395 277395 sha1 340 340 success failure cancelled skipped 340] - 5.383375477s 2025/09/29 02:34:34 HTTPRequest [W] router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for REDACTED:0, elapsed 3367.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/09/29 02:34:35 models/actions/run_job.go:160:UpdateRunJob() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 61 49 277395 277395 sha1 340 340 success failure cancelled skipped 340] - 5.006670551s 2025/09/29 02:34:35 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 34939: load run: find primary languages: context canceled 2025/09/29 02:34:35 models/system/setting.go:134:(*dbConfigCachedGetter).GetRevision() [E] Unable to get all settings: context canceled 2025/09/29 02:34:35 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled 2025/09/29 02:34:36 HTTPRequest [W] router: slow POST /api/v1/repos/REDACTED/ansible-demo-deployment/pulls for REDACTED:0, elapsed 3377.1ms @ repo/pull.go:366(repo.CreatePullRequest) 2025/09/29 02:34:42 .../actions/notifier_helper.go:356:handleWorkflows() [W] [Slow SQL Query] UPDATE `repository` SET `owner_id` = ?, `owner_name` = ?, `lower_name` = ?, `name` = ?, `default_branch` = ?, `default_wiki_branch` = ?, `num_watches` = ?, `num_issues` = ?, `num_pulls` = ?, `num_closed_pulls` = ?, `size` = ?, `git_size` = ?, `object_format_name` = ?, `num_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=?), `num_closed_action_runs` = (SELECT count(*) FROM action_run WHERE repo_id=? AND status IN (?,?,?,?)) WHERE `id`=? [279 REDACTED ansible-demo-deployment ansible-demo-deployment main main 2 3 62 49 280287 280287 sha1 340 340 success failure cancelled skipped 340] - 5.058891808s 2025/09/29 02:34:47 HTTPRequest [W] router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for REDACTED:0, elapsed 3361.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2025/09/29 02:34:48 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 34940: load run: find primary languages: context canceled 2025/09/29 02:34:48 models/system/setting.go:134:(*dbConfigCachedGetter).GetRevision() [E] Unable to get all settings: context canceled 2025/09/29 02:34:48 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled ``` The slow query log on the MariaDB server shows those queries as well. Executing the queries directly on the server lead me to the issue. The problematic part is the sub-query for `num_closed_action_runs`. ``` MariaDB [gitea]> SELECT count(*) FROM action_run WHERE repo_id=340; +----------+ | count(*) | +----------+ | 30908 | +----------+ 1 row in set (0.014 sec) MariaDB [gitea]> SELECT count(*) FROM action_run WHERE repo_id=340 AND status IN (1,2,3,4); +----------+ | count(*) | +----------+ | 30908 | +----------+ 1 row in set (3.236 sec) ``` You see the steep increase in query time, as soon as the `status` condition is added. There is no index yet that would speed up this query: ``` MariaDB [gitea]> ANALYZE SELECT count(*) FROM action_run WHERE repo_id=340 AND status IN (1,2,3,4); +------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra | +------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+ | 1 | SIMPLE | action_run | ref | UQE_action_run_repo_index,IDX_action_run_repo_id,IDX_action_run_status | IDX_action_run_repo_id | 9 | const | 9862 | 30908.00 | 30.75 | 100.00 | Using where | +------+-------------+------------+------+------------------------------------------------------------------------+------------------------+---------+-------+------+----------+----------+------------+-------------+ 1 row in set (2.735 sec) ``` After adding ``` CREATE INDEX `IDX_action_run_repo_status` ON action_run (`repo_id`, `status`); ``` this query now executes in 30 ms. I tested again with more aggressive polling from the runner and with shorter timeouts, which seems fine now. So I'd like to suggest to add this index. Maybe one of the other affected users here can confirm the effectiveness of the solution.
GiteaMirror commented 2025-11-02 11:11:10 -06:00
Author
Owner
Copy Link

@ericLemanissier commented on GitHub (Sep 30, 2025):

For the last failure of this kind, here is my log:

2025/09/29 18:12:14 .../actions/clear_tasks.go:84:stopTasks() [W] Cannot transfer logs of task 81209: dbfs open "actions_log/el/windmill/39/81209.log.zst": file does not exist
2025/09/29 18:02:27 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168942 1759168942 14 0] - 5.080915329s
2025/09/29 18:00:26 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled
2025/09/29 18:00:26 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 101053: GetBranchCommit[main]: context deadline exceeded
2025/09/29 18:00:26 services/webhook/notifier.go:875:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: a8ea77d37248bd4e343153863eed67a16b9c18e5]: context deadline exceeded
2025/09/29 18:00:20 .../actions/schedule_tasks.go:25:StartScheduleTasks() [W] [Slow SQL Query] COMMIT [] - 7.149237895s
2025/09/29 18:00:20 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168812 1759168812 15 0] - 7.201982944s
2025/09/29 18:00:20 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168813 1759168813 11 0] - 6.390584744s

Unfortunately, nothing signals for the action_run table.
In my case, the [Slow SQL Query] on action_runner are probably not the cause, because they are also logged even when the jobs start properly, and with duration even longer than the ones above.

It leaves us with these 3 lines:

services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled
.../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 101053: GetBranchCommit[main]: context deadline exceeded
services/webhook/notifier.go:875:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: a8ea77d37248bd4e343153863eed67a16b9c18e5]: context deadline exceeded

These are the same 3 lines I had in July and @abrain's log also has WorkflowJobStatusUpdate and CreateCommitStatus. Does it ring a bell ?

@ericLemanissier commented on GitHub (Sep 30, 2025): For the last failure of this kind, here is my log: ``` 2025/09/29 18:12:14 .../actions/clear_tasks.go:84:stopTasks() [W] Cannot transfer logs of task 81209: dbfs open "actions_log/el/windmill/39/81209.log.zst": file does not exist 2025/09/29 18:02:27 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168942 1759168942 14 0] - 5.080915329s 2025/09/29 18:00:26 services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled 2025/09/29 18:00:26 .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 101053: GetBranchCommit[main]: context deadline exceeded 2025/09/29 18:00:26 services/webhook/notifier.go:875:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: a8ea77d37248bd4e343153863eed67a16b9c18e5]: context deadline exceeded 2025/09/29 18:00:20 .../actions/schedule_tasks.go:25:StartScheduleTasks() [W] [Slow SQL Query] COMMIT [] - 7.149237895s 2025/09/29 18:00:20 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168812 1759168812 15 0] - 7.201982944s 2025/09/29 18:00:20 .../connect@v1.18.1/handler.go:69:NewUnaryHandler.2() [W] [Slow SQL Query] UPDATE "action_runner" SET "last_online" = $1, "updated" = $2 WHERE "id"=$3 AND ("deleted"=$4 OR "deleted" IS NULL) [1759168813 1759168813 11 0] - 6.390584744s ``` Unfortunately, nothing signals for the `action_run` table. In my case, the `[Slow SQL Query]` on `action_runner` are probably not the cause, because they are also logged even when the jobs start properly, and with duration even longer than the ones above. It leaves us with these 3 lines: ``` services/webhook/notifier.go:962:(*webhookNotifier).WorkflowJobStatusUpdate() [E] Error loading job attributes: find primary languages: context canceled .../actions/commit_status.go:31:CreateCommitStatus() [E] Failed to create commit status for job 101053: GetBranchCommit[main]: context deadline exceeded services/webhook/notifier.go:875:(*webhookNotifier).CreateCommitStatus() [E] commits.ToAPIPayloadCommits failed: FileStatus [commit_sha1: a8ea77d37248bd4e343153863eed67a16b9c18e5]: context deadline exceeded ``` These are the same 3 lines I had [in July](https://github.com/go-gitea/gitea/issues/34198#issuecomment-3117272160) and @abrain's log also has `WorkflowJobStatusUpdate` and `CreateCommitStatus`. Does it ring a bell ?
GiteaMirror commented 2025-11-02 11:11:10 -06:00
Author
Owner
Copy Link

@abrain commented on GitHub (Sep 30, 2025):

I did not see the slow query on UPDATE action_runner yesterday, but today it showed up for me as well. Around that time, the database server's CPU was spiking.

In the database's slow query log I found additional queries happening at the same time:

# Query_time: 2.862758  Lock_time: 0.000000  Rows_sent: 35003  Rows_examined: 35003
# Rows_affected: 0  Bytes_sent: 108522314
SET timestamp=1759203512;
SELECT /*!40001 SQL_NO_CACHE */ `id`, `job_id`, `attempt`, `runner_id`, `status`, `started`, `stopped`, `repo_id`, `owner_id`, `commit_sha`, `is_fork_pull_request`, `token_hash`, `token_salt`, `token_last_eight`, `log_filename`, `log_in_storage`, `log_length`, `log_size`, `log_indexes`, `log_expired`, `created`, `updated` FROM `action_task`;

# Query_time: 6.318850  Lock_time: 0.000000  Rows_sent: 167316  Rows_examined: 167316
# Rows_affected: 0  Bytes_sent: 91351066
SET timestamp=1759210547;
SELECT /*!40001 SQL_NO_CACHE */ `id`, `user_id`, `op_type`, `act_user_id`, `repo_id`, `comment_id`, `is_deleted`, `ref_name`, `is_private`, `content`, `created_unix` FROM `action`;

# Query_time: 5.577170  Lock_time: 0.000000  Rows_sent: 34964  Rows_examined: 34964
# Rows_affected: 0  Bytes_sent: 210914107
SET timestamp=1759210553;
SELECT /*!40001 SQL_NO_CACHE */ `id`, `title`, `repo_id`, `owner_id`, `workflow_id`, `index`, `trigger_user_id`, `ref`, `commit_sha`, `event`, `is_fork_pull_request`, `event_payload`, `status`, `started`, `stopped`, `created`, `updated`, `need_approval`, `approved_by`, `trigger_event`, `version`, `schedule_id`, `previous_duration` FROM `action_run`;

The result sets were 90 to 210 MB in size. I am not sure why the Gitea server would request entire tables. But I think the slowness of the very simple UPDATE action_runner query is a side effect of the DB server being quite busy.

@ericLemanissier Regarding the action_run (or rather UPDATE repository) query not showing up in your log: The default SLOW_QUERY_THRESHOLD for Gitea is 5 seconds. If there are two queries taking 3 seconds each, they could cause the same issue, but you would not see them in the Gitea log.

From my understanding, the WorkflowJobStatusUpdate and CreateCommitStatus lines are the result of the runner cancelling the connection because of the timeout (also 5 seconds by default), but I could be wrong.

@abrain commented on GitHub (Sep 30, 2025): I did not see the slow query on `UPDATE action_runner` yesterday, but today it showed up for me as well. Around that time, the database server's CPU was spiking. In the database's slow query log I found additional queries happening at the same time: ``` # Query_time: 2.862758 Lock_time: 0.000000 Rows_sent: 35003 Rows_examined: 35003 # Rows_affected: 0 Bytes_sent: 108522314 SET timestamp=1759203512; SELECT /*!40001 SQL_NO_CACHE */ `id`, `job_id`, `attempt`, `runner_id`, `status`, `started`, `stopped`, `repo_id`, `owner_id`, `commit_sha`, `is_fork_pull_request`, `token_hash`, `token_salt`, `token_last_eight`, `log_filename`, `log_in_storage`, `log_length`, `log_size`, `log_indexes`, `log_expired`, `created`, `updated` FROM `action_task`; # Query_time: 6.318850 Lock_time: 0.000000 Rows_sent: 167316 Rows_examined: 167316 # Rows_affected: 0 Bytes_sent: 91351066 SET timestamp=1759210547; SELECT /*!40001 SQL_NO_CACHE */ `id`, `user_id`, `op_type`, `act_user_id`, `repo_id`, `comment_id`, `is_deleted`, `ref_name`, `is_private`, `content`, `created_unix` FROM `action`; # Query_time: 5.577170 Lock_time: 0.000000 Rows_sent: 34964 Rows_examined: 34964 # Rows_affected: 0 Bytes_sent: 210914107 SET timestamp=1759210553; SELECT /*!40001 SQL_NO_CACHE */ `id`, `title`, `repo_id`, `owner_id`, `workflow_id`, `index`, `trigger_user_id`, `ref`, `commit_sha`, `event`, `is_fork_pull_request`, `event_payload`, `status`, `started`, `stopped`, `created`, `updated`, `need_approval`, `approved_by`, `trigger_event`, `version`, `schedule_id`, `previous_duration` FROM `action_run`; ``` The result sets were 90 to 210 MB in size. I am not sure why the Gitea server would request entire tables. But I think the slowness of the very simple `UPDATE action_runner` query is a side effect of the DB server being quite busy. @ericLemanissier Regarding the `action_run` (or rather `UPDATE repository`) query not showing up in your log: The default `SLOW_QUERY_THRESHOLD` for Gitea is 5 seconds. If there are two queries taking 3 seconds each, they could cause the same issue, but you would not see them in the Gitea log. From my understanding, the `WorkflowJobStatusUpdate` and `CreateCommitStatus` lines are the result of the runner cancelling the connection because of the timeout (also 5 seconds by default), but I could be wrong.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label issue/workaround performance/speed topic/gitea-actions
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#14371
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?