github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-10 05:32:52 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Repeated Queue "code_indexer" failed to handle batch of 1 items error logs #13408

New Issue
Closed
opened 2025-11-02 10:41:27 -06:00 by GiteaMirror · 3 comments
GiteaMirror commented 2025-11-02 10:41:27 -06:00
Owner
Copy Link

Originally created by @MattSG on GitHub (Aug 20, 2024).

Description

Error logs appear every couple seconds constantly as app is running, appears to come with a memory leak

Setting REPO_INDEXER_ENABLED = false stops the errors

Unaware of RAM usage at the time on the machine but after thousands of these logs, the service appeared to stop itself without reason. The shutdown may not be related to this

Edit: Found these logs in the event viewer that align with the time gitea stopped

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: gitea.exe (8476) consumed 11716718592 bytes, MsMpEng.exe (8008) consumed 459538432 bytes, and MsSense.exe (3024) consumed 256311296 bytes.

The gitea service terminated unexpectedly.  It has done this 1 time(s).

Maybe this is related to https://github.com/go-gitea/gitea/issues/31565?

2024/08/20 10:52:43 cmd/web.go:355:listen() [I] HTTP Listener: 0.0.0.0:3000 Closed
2024/08/20 10:52:43 .../graceful/manager.go:168:doHammerTime() [W] Setting Hammer condition
2024/08/20 10:52:44 .../graceful/manager.go:184:doTerminate() [W] Terminating
2024/08/20 10:52:44 ...er/issues/indexer.go:120:2() [I] PID: 8940 Issue Indexer closed
2024/08/20 10:52:44 ...exer/code/indexer.go:112:func1() [I] PID: 8940 Repository Indexer closed
2024/08/20 10:52:44 cmd/web.go:214:serveInstalled() [I] PID: 8940 Gitea Web Finished

Indexer config

[indexer]
; ...
REPO_INDEXER_ENABLED = true
REPO_INDEXER_PATH = indexers/repos.bleve
MAX_FILE_SIZE = 1048576
REPO_INDEXER_INCLUDE = 
REPO_INDEXER_EXCLUDE = resources/bin/**

Error log:

2024/08/20 00:00:01 ...s/actions/cleanup.go:60:cleanNeedDeleteArtifacts() [I] Found 0 artifacts pending deletion
2024/08/20 00:00:03 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 75.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:03 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.
2024/08/20 00:00:03 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/08/20 00:00:05 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 30.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:07 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 28.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:09 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 28.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:10 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.
2024/08/20 00:00:10 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/08/20 00:00:11 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 36.6ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:13 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 62.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:15 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 84.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:17 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 58.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:17 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.
2024/08/20 00:00:17 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/08/20 00:00:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 312.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:21 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 85.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:23 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 72.6ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:25 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.
2024/08/20 00:00:25 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/08/20 00:00:25 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 299.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 60.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:30 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 1177.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:31 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 122.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:32 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.
2024/08/20 00:00:32 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds
2024/08/20 00:00:33 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 110.7ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:35 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 639.7ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:37 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 364.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:39 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 277.2ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2024/08/20 00:00:39 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.

Gitea Version

1.22.1 go-git

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.45.2.windows.1

Operating System

Windows Server 2016

How are you running Gitea?

Windows gogit exe on prem running a service

Database

SQLite

Originally created by @MattSG on GitHub (Aug 20, 2024). ### Description Error logs appear every couple seconds constantly as app is running, appears to come with a memory leak Setting `REPO_INDEXER_ENABLED = false` stops the errors ~~Unaware of RAM usage at the time on the machine but after thousands of these logs, the service appeared to stop itself without reason. The shutdown may not be related to this~~ Edit: Found these logs in the event viewer that align with the time gitea stopped ``` Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: gitea.exe (8476) consumed 11716718592 bytes, MsMpEng.exe (8008) consumed 459538432 bytes, and MsSense.exe (3024) consumed 256311296 bytes. ``` ``` The gitea service terminated unexpectedly. It has done this 1 time(s). ``` Maybe this is related to https://github.com/go-gitea/gitea/issues/31565? ``` 2024/08/20 10:52:43 cmd/web.go:355:listen() [I] HTTP Listener: 0.0.0.0:3000 Closed 2024/08/20 10:52:43 .../graceful/manager.go:168:doHammerTime() [W] Setting Hammer condition 2024/08/20 10:52:44 .../graceful/manager.go:184:doTerminate() [W] Terminating 2024/08/20 10:52:44 ...er/issues/indexer.go:120:2() [I] PID: 8940 Issue Indexer closed 2024/08/20 10:52:44 ...exer/code/indexer.go:112:func1() [I] PID: 8940 Repository Indexer closed 2024/08/20 10:52:44 cmd/web.go:214:serveInstalled() [I] PID: 8940 Gitea Web Finished ``` Indexer config ``` [indexer] ; ... REPO_INDEXER_ENABLED = true REPO_INDEXER_PATH = indexers/repos.bleve MAX_FILE_SIZE = 1048576 REPO_INDEXER_INCLUDE = REPO_INDEXER_EXCLUDE = resources/bin/** ``` Error log: ``` 2024/08/20 00:00:01 ...s/actions/cleanup.go:60:cleanNeedDeleteArtifacts() [I] Found 0 artifacts pending deletion 2024/08/20 00:00:03 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 75.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:03 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. 2024/08/20 00:00:03 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds 2024/08/20 00:00:05 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 30.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:07 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 28.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:09 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 28.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:10 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. 2024/08/20 00:00:10 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds 2024/08/20 00:00:11 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 36.6ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:13 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 62.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:15 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 84.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:17 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 58.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:17 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. 2024/08/20 00:00:17 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds 2024/08/20 00:00:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 312.0ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:21 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 85.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:23 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 72.6ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:25 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. 2024/08/20 00:00:25 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds 2024/08/20 00:00:25 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 299.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:27 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 60.5ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:30 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 1177.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:31 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 122.9ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:32 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. 2024/08/20 00:00:32 ...queue/workergroup.go:107:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds 2024/08/20 00:00:33 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 110.7ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:35 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 639.7ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:37 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 364.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:39 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 127.0.0.1:51052, 200 OK in 277.2ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm) 2024/08/20 00:00:39 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. ``` ### Gitea Version 1.22.1 go-git ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist _No response_ ### Screenshots _No response_ ### Git Version 2.45.2.windows.1 ### Operating System Windows Server 2016 ### How are you running Gitea? Windows gogit exe on prem running a service ### Database SQLite
GiteaMirror added the type/bug label 2025-11-02 10:41:27 -06:00
GiteaMirror commented 2025-11-02 10:41:28 -06:00
Author
Owner
Copy Link

@yp05327 commented on GitHub (Aug 22, 2024):

[E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long.

This error maybe similar to https://github.com/go-gitea/gitea/issues/29907#issuecomment-2014402614
Windows has maximum path length limitation which is too short in sometimes.

@yp05327 commented on GitHub (Aug 22, 2024): > [E] Codes indexer handler: index error for repo 78: fork/exec C:\Program Files\Git\cmd\git.exe: The filename or extension is too long. This error maybe similar to https://github.com/go-gitea/gitea/issues/29907#issuecomment-2014402614 Windows has maximum path length limitation which is too short in sometimes.
GiteaMirror commented 2025-11-02 10:41:28 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Aug 22, 2024):

Nope, the root problem is this:

-> search indexer broken: ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 29: fork/exec /usr/bin/git: argument list too long #28584

See my comment https://github.com/go-gitea/gitea/issues/28584#issuecomment-1870130145

Gitea reads the full file list into memory.

Another related problem is the builtin bleve indexer, it sometimes consumes a lot of memory.

@wxiaoguang commented on GitHub (Aug 22, 2024): Nope, the root problem is this: -> search indexer broken: ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 29: fork/exec /usr/bin/git: argument list too long #28584 See my comment https://github.com/go-gitea/gitea/issues/28584#issuecomment-1870130145 Gitea reads the full file list into memory. Another related problem is the builtin bleve indexer, it sometimes consumes a lot of memory.
GiteaMirror commented 2025-11-02 10:41:28 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (Aug 22, 2024):

I sent a PR to try to fix #28584

@lunny commented on GitHub (Aug 22, 2024): I sent a PR to try to fix #28584
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#13408
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?