Log OOM errors, if possible #12324

New Issue

GiteaMirror · 2025-11-02T10:05:38-06:00

GiteaMirror commented

2025-11-02 10:05:38 -06:00

Originally created by @lonix1 on GitHub (Jan 11, 2024).

Feature Description

I was running gitea in docker, in a small VPS with limited memory. It worked for years, but suddenly I could not clone, and the log showed:

Received signal 15; terminating.

That is SIGTERM. It doesn't say what happened, so it's very hard to diagnose. I had debug logging enabled.

When I increased the memory, it worked again. So the issue was actually an Out Of Memory error.

I assume the SIGTERM is sent by docker; does gitea receive any information other than the signal?

If yes:

it should include "OOM" in the log message to make it easy to diagnose such problems

If no:

just close this issue
OR, maybe include extra info in the log message? Something like Received signal 15; terminating. If you did not stop the container then this could be due to an OOM error.

Screenshots

No response

Originally created by @lonix1 on GitHub (Jan 11, 2024). ### Feature Description I was running gitea in docker, in a small VPS with limited memory. It worked for years, but suddenly I could not clone, and the log showed: > Received signal 15; terminating. That is `SIGTERM`. It doesn't say what happened, so it's very hard to diagnose. I had debug logging enabled. When I increased the memory, it worked again. So the issue was actually an *Out Of Memory* error. I assume the SIGTERM is sent by docker; does gitea receive any information other than the signal? If yes: - it should include "OOM" in the log message to make it easy to diagnose such problems If no: - just close this issue - OR, maybe include extra info in the log message? Something like `Received signal 15; terminating. If you did not stop the container then this could be due to an OOM error.` ### Screenshots _No response_

GiteaMirror added the type/proposal issue/needs-feedback labels 2025-11-02 10:05:38 -06:00

GiteaMirror closed this issue

2025-11-02 10:05:38 -06:00

GiteaMirror commented

2025-11-02 10:05:39 -06:00

@lunny commented on GitHub (Jan 11, 2024):

Is there a log file? Especially the last lines before exit.

@lunny commented on GitHub (Jan 11, 2024): Is there a log file? Especially the last lines before exit.

GiteaMirror commented

2025-11-02 10:05:39 -06:00

@lonix1 commented on GitHub (Jan 11, 2024):

I was logging on debug level. Everything was "normal" (like always), except for the last line:

Received signal 15; terminating.

I monitored the container using docker stats and I could see that was when the container was terminated, and then restarted. Docker engine sends the SIGTERM when it runs out of memory.

@lonix1 commented on GitHub (Jan 11, 2024): I was logging on debug level. Everything was "normal" (like always), except for the last line: > Received signal 15; terminating. I monitored the container using `docker stats` and I could see that was when the container was terminated, and then restarted. Docker engine sends the SIGTERM when it runs out of memory.

GiteaMirror commented

2025-11-02 10:05:39 -06:00

@lonix1 commented on GitHub (Jan 12, 2024):

You can simulate the memory issue like this:

gitea:
  # ...
  deploy:
    resources:
      limits:
        cpus: '0.25'
        memory: 50m
  # ...

Just use a small value. Maybe 50m.

@lonix1 commented on GitHub (Jan 12, 2024): You can simulate the memory issue like this: ```yml gitea: # ... deploy: resources: limits: cpus: '0.25' memory: 50m # ... ``` Just use a small value. Maybe `50m`.

GiteaMirror commented

2025-11-02 10:05:40 -06:00

@lunny commented on GitHub (Jan 12, 2024):

50M looks too small for Gitea, I think it should at least 1GB as the limitation.

@lunny commented on GitHub (Jan 12, 2024): 50M looks too small for Gitea, I think it should at least 1GB as the limitation.

GiteaMirror commented

2025-11-02 10:05:40 -06:00

@lonix1 commented on GitHub (Jan 12, 2024):

Agreed it's too small.

But what I mean is that you can force the OOM error that way. So you can see the log that I see.

(By the way, I have used gitea with 150MiB for years... Amazing app 😄. Today I am using 300MiB without problems.)

@lonix1 commented on GitHub (Jan 12, 2024): Agreed it's too small. But what I mean is that you can force the OOM error that way. So you can see the log that I see. (By the way, I have used gitea with 150MiB for years... Amazing app :smile:. Today I am using 300MiB without problems.)

GiteaMirror commented

2025-11-02 10:05:40 -06:00

@wxiaoguang commented on GitHub (Jan 15, 2024):

Golang is not Java. IIRC it's impossible to catch the OOM inside a Go app ....

If there is such an approach, feel free to tell me and I'd like to learn from it .....

https://github.com/golang/go/issues/14162
https://github.com/golang/go/issues/16843
https://github.com/golang/go/issues/29696
https://www.reddit.com/r/golang/comments/9dpxwt/how_to_catch_out_of_memory_panic_in_go/

@wxiaoguang commented on GitHub (Jan 15, 2024): Golang is not Java. IIRC it's impossible to catch the OOM inside a Go app .... If there is such an approach, feel free to tell me and I'd like to learn from it ..... ``` https://github.com/golang/go/issues/14162 https://github.com/golang/go/issues/16843 https://github.com/golang/go/issues/29696 https://www.reddit.com/r/golang/comments/9dpxwt/how_to_catch_out_of_memory_panic_in_go/ ```

GiteaMirror commented

2025-11-02 10:05:41 -06:00

@lonix1 commented on GitHub (Jan 15, 2024):

Sorry, I'm not a go dev so I don't know.

So is my guess correct? If one can't detect the OOM error, then what happens is:

server runs out of memory
docker daemon starts killing containers
gitea receives SIGTERM without explanation
gitea logs Received signal 15; terminating. and terminates
docker daemon restarts gitea

If that's the case, what about adding a more useful log, like I mentioned above:

Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)

@lonix1 commented on GitHub (Jan 15, 2024): Sorry, I'm not a go dev so I don't know. So is my guess correct? If one can't detect the OOM error, then what happens is: - server runs out of memory - docker daemon starts killing containers - gitea receives `SIGTERM` without explanation - gitea logs `Received signal 15; terminating.` and terminates - docker daemon restarts gitea If that's the case, what about adding a more useful log, like I mentioned above: ``` Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.) ```

GiteaMirror commented

2025-11-02 10:05:41 -06:00

@wxiaoguang commented on GitHub (Jan 15, 2024):

OOM could appear anywhere, in kernel (seldom nowadays), in docker container program, in docker inside app.

If the Go app itself triggers OOM, then the app itself gets killed. Since Gitea's docker images uses s6 to run gitea web, so if the gitea web gets skilled, the docker container still runs because it still sees s6 service is running.

Maybe it could tell the s6 to run gitea web "always" (just like docker's "always"), even if it exits with error. (just a guess)

@wxiaoguang commented on GitHub (Jan 15, 2024): OOM could appear anywhere, in kernel (seldom nowadays), in docker container program, in docker inside app. If the Go app itself triggers OOM, then the app itself gets killed. Since Gitea's docker images uses s6 to run `gitea web`, so if the `gitea web` gets skilled, the docker container still runs because it still sees s6 service is running. Maybe it could tell the s6 to run `gitea web` "always" (just like docker's "always"), even if it exits with error. (just a guess)

GiteaMirror commented

2025-11-02 10:05:41 -06:00

@GiteaBot commented on GitHub (Feb 14, 2024):

We close issues that need feedback from the author if there were no new comments for a month. 🍵

@GiteaBot commented on GitHub (Feb 14, 2024): We close issues that need feedback from the author if there were no new comments for a month. :tea:

GiteaMirror commented

2025-11-02 10:05:41 -06:00

@lonix1 commented on GitHub (Feb 14, 2024):

Not sure what more feedback I must send. Maybe we can revisit this in the future.

@lonix1 commented on GitHub (Feb 14, 2024): Not sure what more feedback I must send. Maybe we can revisit this in the future.

GiteaMirror commented

2025-11-02 10:05:42 -06:00

@wxiaoguang commented on GitHub (Feb 15, 2024):

Hmm yup, at the moment I have no idea either. IMO "it is not possible for Golang to log OOM"

@wxiaoguang commented on GitHub (Feb 15, 2024): Hmm yup, at the moment I have no idea either. IMO "it is not possible for Golang to log OOM"

GiteaMirror commented

2025-11-02 10:05:42 -06:00

@lonix1 commented on GitHub (Feb 15, 2024):

The only thing that can be done is to log a better error like above:

Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)

@lonix1 commented on GitHub (Feb 15, 2024): The only thing that can be done is to log a better error like above: Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)

GiteaMirror commented

2025-11-02 10:05:42 -06:00

@wxiaoguang commented on GitHub (Feb 16, 2024):

The only thing that can be done is to log a better error like above:

Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)

Sorry I still couldn't catch your point. Could you propose a prototype to show how to "log a better error"?

@wxiaoguang commented on GitHub (Feb 16, 2024): > The only thing that can be done is to log a better error like above: > > Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.) Sorry I still couldn't catch your point. Could you propose a prototype to show how to "log a better error"?

GiteaMirror commented

2025-11-02 10:05:42 -06:00

@lonix1 commented on GitHub (Feb 16, 2024):

It logs this:

"Received signal 15; terminating."

So I suggested:

"Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)"

Or something like that.

@lonix1 commented on GitHub (Feb 16, 2024): It logs this: > "Received signal 15; terminating." So I suggested: > "Received signal 15; terminating. (If you did not stop the container then this could be due to an OOM error.)" Or something like that.

GiteaMirror commented

2025-11-02 10:05:42 -06:00

@wxiaoguang commented on GitHub (Feb 16, 2024):

Some more details:

Received signal 15 is not only caused by OOM, it could also be caused by some other problems. Only mentioning "OOM" in the message could be misleading.
Received signal 15; terminating. is printed by kernel, so you can't append anything to it.
When there is something wrong, sysop could/should take a look at dmesg, there are more details (also for OOM).
If you want to "catch" the exit code, you need to rewrite the dockerfile by removing the "exec", because Golang doesn't have the ability to catch OOM early.

If you have a feasible approach, feel free to propose a PR.

@wxiaoguang commented on GitHub (Feb 16, 2024): Some more details: * `Received signal 15` is not only caused by OOM, it could also be caused by some other problems. Only mentioning "OOM" in the message could be misleading. * `Received signal 15; terminating.` is printed by kernel, so you can't append anything to it. * When there is something wrong, sysop could/should take a look at dmesg, there are more details (also for OOM). * If you want to "catch" the exit code, you need to rewrite the dockerfile by removing the "exec", because Golang doesn't have the ability to catch OOM early. If you have a feasible approach, feel free to propose a PR.

GiteaMirror commented

2025-11-02 10:05:43 -06:00

@lonix1 commented on GitHub (Feb 16, 2024):

Received signal 15 is not only caused by OOM, it could also be caused by some other problems

Aha... I didn't know that. So you are right - we cannot change that log message!

It's a tricky problem. Thanks for looking into it.

@lonix1 commented on GitHub (Feb 16, 2024): > Received signal 15 is not only caused by OOM, it could also be caused by some other problems Aha... I didn't know that. So you are right - we cannot change that log message! It's a tricky problem. Thanks for looking into it.

GiteaMirror commented

2025-11-02 10:05:43 -06:00

@github-actions[bot] commented on GitHub (Feb 28, 2024):

Automatically locked because of our CONTRIBUTING guidelines

@github-actions[bot] commented on GitHub (Feb 28, 2024): Automatically locked because of our [CONTRIBUTING guidelines](https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md#issue-locking)

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#12324