github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-12 02:24:21 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

High disk usage on repetitive logging #4386

New Issue
Closed
opened 2025-11-02 05:48:58 -06:00 by GiteaMirror · 9 comments
GiteaMirror commented 2025-11-02 05:48:58 -06:00
Owner
Copy Link

Originally created by @brandonkal on GitHub (Nov 25, 2019).

  • Gitea version (or commit ref): 1.9.4
  • Git version: latest
  • Operating system: Ubuntu 18 LTS
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

Description

I noticed my kubernetes node is continually writing at at 5MiB/s over my network. The k8s PV is on an NFS share and sqlite is used.
The issue may just be the loop of repetitive and high-frequency logging Gitea is performing:

 2019/11/25 00:53:16 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log                                                                                   │
│ 2019/11/25 00:53:17 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled                                                    │
│ 2019/11/25 00:53:17 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea                                                                                   │
│ 2019/11/25 00:53:17 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea                                                                                     │
│ 2019/11/25 00:53:17 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea                                                                                    │
│ 2019/11/25 00:53:17 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log                                                                                   │
│ 2019/11/25 00:53:18 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled                                                    │
│ 2019/11/25 00:53:18 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea                                                                                   │
│ 2019/11/25 00:53:18 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea                                                                                     │
│ 2019/11/25 00:53:18 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea                                                                                    │
│ 2019/11/25 00:53:18 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log                                                                                   │
│ 2019/11/25 00:53:18 ...dules/setting/log.go:226:newLogService() [I] Gitea v1.9.4 built with GNU Make 4.2.1, go1.12.10 : bindata, sqlite, sqlite_unlock_notify       │
│ 2019/11/25 00:53:19 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled                                                    │
│ 2019/11/25 00:53:19 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea                                                                                   │
│ 2019/11/25 00:53:19 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea                                                                                     │
│ 2019/11/25 00:53:19 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea                                                                                    │
│ 2019/11/25 00:53:19 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log                                                                                   │
│ 2019/11/25 00:53:20 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled                                                    │
│ 2019/11/25 00:53:20 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea                                                                                   │
│ 2019/11/25 00:53:20 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea                                                                                     │
│ 2019/11/25 00:53:20 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea                                                                                    │
│ 2019/11/25 00:53:20 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log                                                                                   │
│ 2019/11/25 00:53:20 ...dules/setting/log.go:226:newLogService() [I] Gitea v1.9.4 built with GNU Make 4.2.1, go1.12.10 : bindata, sqlite, sqlite_unlock_notify       │
│ 2019/11/25 00:53:21 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled                                                    │
│ 2019/11/25 00:53:21 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea                                                                                   │
│ 2019/11/25 00:53:21 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea                                                                                     │
│ 2019/11/25 00:53:21 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea                                                                                    │
│ 2019/11/25 00:53:21 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log

Whatever is causing this loop likely needs fixing.

That goes on forever without end.

On another note, I would much rather have logging occur to stdout rather than to disk.

Screenshots

Originally created by @brandonkal on GitHub (Nov 25, 2019). <!-- NOTE: If your issue is a security concern, please send an email to security@gitea.io instead of opening a public issue --> <!-- 1. Please speak English, this is the language all maintainers can speak and write. 2. Please ask questions or configuration/deploy problems on our Discord server (https://discord.gg/gitea) or forum (https://discourse.gitea.io). 3. Please take a moment to check that your issue doesn't already exist. 4. Please give all relevant information below for bug reports, because incomplete details will be handled as an invalid report. --> - Gitea version (or commit ref): 1.9.4 - Git version: latest - Operating system: Ubuntu 18 LTS - Database (use `[x]`): - [ ] PostgreSQL - [ ] MySQL - [ ] MSSQL - [x] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: ## Description I noticed my kubernetes node is continually writing at at 5MiB/s over my network. The k8s PV is on an NFS share and sqlite is used. The issue may just be the loop of repetitive and high-frequency logging Gitea is performing: ``` 2019/11/25 00:53:16 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log │ │ 2019/11/25 00:53:17 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled │ │ 2019/11/25 00:53:17 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea │ │ 2019/11/25 00:53:17 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea │ │ 2019/11/25 00:53:17 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea │ │ 2019/11/25 00:53:17 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log │ │ 2019/11/25 00:53:18 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled │ │ 2019/11/25 00:53:18 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea │ │ 2019/11/25 00:53:18 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea │ │ 2019/11/25 00:53:18 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea │ │ 2019/11/25 00:53:18 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log │ │ 2019/11/25 00:53:18 ...dules/setting/log.go:226:newLogService() [I] Gitea v1.9.4 built with GNU Make 4.2.1, go1.12.10 : bindata, sqlite, sqlite_unlock_notify │ │ 2019/11/25 00:53:19 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled │ │ 2019/11/25 00:53:19 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea │ │ 2019/11/25 00:53:19 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea │ │ 2019/11/25 00:53:19 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea │ │ 2019/11/25 00:53:19 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log │ │ 2019/11/25 00:53:20 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled │ │ 2019/11/25 00:53:20 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea │ │ 2019/11/25 00:53:20 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea │ │ 2019/11/25 00:53:20 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea │ │ 2019/11/25 00:53:20 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log │ │ 2019/11/25 00:53:20 ...dules/setting/log.go:226:newLogService() [I] Gitea v1.9.4 built with GNU Make 4.2.1, go1.12.10 : bindata, sqlite, sqlite_unlock_notify │ │ 2019/11/25 00:53:21 ...dules/setting/git.go:83:newGit() [I] Git Version: 2.22.0, Wire Protocol Version 2 Enabled │ │ 2019/11/25 00:53:21 routers/init.go:72:GlobalInit() [T] AppPath: /app/gitea/gitea │ │ 2019/11/25 00:53:21 routers/init.go:73:GlobalInit() [T] AppWorkPath: /app/gitea │ │ 2019/11/25 00:53:21 routers/init.go:74:GlobalInit() [T] Custom path: /data/gitea │ │ 2019/11/25 00:53:21 routers/init.go:75:GlobalInit() [T] Log path: /data/gitea/log ``` Whatever is causing this loop likely needs fixing. That goes on forever without end. On another note, I would much rather have logging occur to stdout rather than to disk. ## Screenshots <!-- **If this issue involves the Web Interface, please include a screenshot** -->
GiteaMirror added the type/question label 2025-11-02 05:48:58 -06:00
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@lunny commented on GitHub (Nov 25, 2019):

So this pod start failed?

@lunny commented on GitHub (Nov 25, 2019): So this pod start failed?
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@brandonkal commented on GitHub (Nov 25, 2019):

No. The pod runs fine.

@brandonkal commented on GitHub (Nov 25, 2019): No. The pod runs fine.
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@guillep2k commented on GitHub (Nov 25, 2019):

@brandonkal You can set up the log as you need (e.g. to console or nowhere if you'd like) in app.ini: https://docs.gitea.io/en-us/logging-configuration/

BTW, use of SQLite is actually not recommended. It's only "supported" because it simplifies our testing procedures, but it's a one-transaction-at-a-time database. You will encounter issues sooner than later in the form of deadlocks.

@guillep2k commented on GitHub (Nov 25, 2019): @brandonkal You can set up the log as you need (e.g. to console or nowhere if you'd like) in `app.ini`: https://docs.gitea.io/en-us/logging-configuration/ BTW, use of SQLite is actually not recommended. It's only "supported" because it simplifies our testing procedures, but it's a one-transaction-at-a-time database. You will encounter issues sooner than later in the form of deadlocks.
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Nov 25, 2019):

The pod is not starting fine.

Gitea is constantly starting and restarting.

Presumably there is a log.Fatal missing here - not sure why it's not being logged. It might be that macaron is killing everything because of its dumb init but I would have thought that would kill things earlier.

Please check your config.

If you don't want file logs, Gitea's logging is highly configurable and you are able to change this very easily.

If you can't work out how to change this feel free to post a copy of your suitably a sanitised app.ini here.

@zeripath commented on GitHub (Nov 25, 2019): The pod is not starting fine. Gitea is constantly starting and restarting. Presumably there is a log.Fatal missing here - not sure why it's not being logged. It might be that macaron is killing everything because of its dumb init but I would have thought that would kill things earlier. Please check your config. If you don't want file logs, Gitea's logging is highly configurable and you are able to change this very easily. If you can't work out how to change this feel free to post a copy of your suitably a sanitised app.ini here.
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@brandonkal commented on GitHub (Nov 25, 2019):

@zeripath okay.

There are some issues with the current approach. As far as Kubernetes is concerned, everything is running fine. In fact, kubernetes cannot really manage gitea properly because it has no way of knowing the status of the several processes inside the container.

I see that s6 is used inside the container. That is okay, but if that is done, kubernetes needs a way of understanding that something is not right.

I will redeploy with the latest version and see how things go.

@brandonkal commented on GitHub (Nov 25, 2019): @zeripath okay. There are some issues with the current approach. As far as Kubernetes is concerned, everything is running fine. In fact, kubernetes cannot really manage gitea properly because it has no way of knowing the status of the several processes inside the container. I see that s6 is used inside the container. That is okay, but if that is done, kubernetes needs a way of understanding that something is not right. I will redeploy with the latest version and see how things go.
GiteaMirror commented 2025-11-02 05:48:59 -06:00
Author
Owner
Copy Link

@zeripath commented on GitHub (Nov 25, 2019):

I don't think just redeploying will solve the issue. There is very likely to be a problem with your configuration - I'm not sure what without looking at your app.ini.

In terms of improving the docker - do you have any suggestions for a better architecture? We're always open to PRs.

@zeripath commented on GitHub (Nov 25, 2019): I don't think just redeploying will solve the issue. There is very likely to be a problem with your configuration - I'm not sure what without looking at your app.ini. In terms of improving the docker - do you have any suggestions for a better architecture? We're always open to PRs.
GiteaMirror commented 2025-11-02 05:49:00 -06:00
Author
Owner
Copy Link

@brandonkal commented on GitHub (Nov 26, 2019):

I don't expect redeploying will solve the issue. I also don't expect it is anything specific

For the docker:

  1. There should be a /_/health endpoint
  2. The container should exit when a service fails.
  3. If there must be a long running service managed outside of the container orchestrator, it should have exponential back-off so it doesn't thrash the network/disk.
  4. sshd/openssh should be split into its own container
@brandonkal commented on GitHub (Nov 26, 2019): I don't expect redeploying will solve the issue. I also don't expect it is anything specific For the docker: 1. There should be a `/_/health` endpoint 2. The container should exit when a service fails. 3. If there must be a long running service managed outside of the container orchestrator, it should have exponential back-off so it doesn't thrash the network/disk. 4. sshd/openssh should be split into its own container
GiteaMirror commented 2025-11-02 05:49:00 -06:00
Author
Owner
Copy Link

@lafriks commented on GitHub (Nov 27, 2019):

  1. There is plan to move docker image away from OpenSSH to built in ssh server. Than we wont need service manager in container etc
@lafriks commented on GitHub (Nov 27, 2019): 4. There is plan to move docker image away from OpenSSH to built in ssh server. Than we wont need service manager in container etc
GiteaMirror commented 2025-11-02 05:49:00 -06:00
Author
Owner
Copy Link

@techknowlogick commented on GitHub (Dec 2, 2019):

Closing this as documentation on logging has been provided, and a PR to stop container when binary fails has been merged. As @lafriks mentioned we will be removing OpenSSH from container (can't be moved to a different container by itself), and built-in SSH server will be used (as 1.10 had a major overhaul of it improving reliability and speed of it).

@techknowlogick commented on GitHub (Dec 2, 2019): Closing this as documentation on logging has been provided, and a PR to stop container when binary fails has been merged. As @lafriks mentioned we will be removing OpenSSH from container (can't be moved to a different container by itself), and built-in SSH server will be used (as 1.10 had a major overhaul of it improving reliability and speed of it).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label type/question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#4386
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?