github-starred/pangolin
2
0
Fork 0
mirror of https://github.com/fosrl/pangolin.git synced 2026-05-08 21:59:09 -05:00
Code Issues 564 Packages Projects Releases 72 Wiki Activity

[GH-ISSUE #436] Uploading Files to Home Server Causing Pangolin VPS to Explode in Size #3391

New Issue
Closed
opened 2026-04-20 07:19:36 -05:00 by GiteaMirror · 24 comments
GiteaMirror commented 2026-04-20 07:19:36 -05:00
Owner
Copy Link

Originally created by @kylepyke on GitHub (Mar 30, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/436

I'm having an issue where uploading files to my homeserver (e.g. Immich) through my Pangolin VPS causes the VPS hard drive to fill up. It looks like the /var/lib/docker/overlay2 folder balloons?

On the VPS, I'm only hosting services related to Pangolin: Traefik, Gerbil, and Crowdsec.

Thanks!

Originally created by @kylepyke on GitHub (Mar 30, 2025). Original GitHub issue: https://github.com/fosrl/pangolin/issues/436 I'm having an issue where uploading files to my homeserver (e.g. Immich) through my Pangolin VPS causes the VPS hard drive to fill up. It looks like the `/var/lib/docker/overlay2` folder balloons? On the VPS, I'm only hosting services related to Pangolin: Traefik, Gerbil, and Crowdsec. Thanks!
GiteaMirror added the stale label 2026-04-20 07:19:36 -05:00
GiteaMirror commented 2026-04-20 07:19:37 -05:00
Author
Owner
Copy Link

@oschwartz10612 commented on GitHub (Apr 1, 2025):

Hello!

It sounds like one of the containers running on your Pangolin VPS (likely Traefik or potentially CrowdSec) might be logging excessive data or temporarily buffering the uploads within its Docker layer, causing the /var/lib/docker/overlay2 growth. You could try checking the logs for each container (docker logs <container_name>) during an upload to see if one is generating massive output, and also review the Traefik and CrowdSec configurations to ensure log levels aren't set to debug or that there isn't unexpected buffering enabled.

<!-- gh-comment-id:2767748144 --> @oschwartz10612 commented on GitHub (Apr 1, 2025): Hello! It sounds like one of the containers running on your Pangolin VPS (likely Traefik or potentially CrowdSec) might be logging excessive data or temporarily buffering the uploads within its Docker layer, causing the /var/lib/docker/overlay2 growth. You could try checking the logs for each container (docker logs <container_name>) during an upload to see if one is generating massive output, and also review the Traefik and CrowdSec configurations to ensure log levels aren't set to debug or that there isn't unexpected buffering enabled.
GiteaMirror commented 2026-04-20 07:19:38 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 1, 2025):

Thanks! I will check configs for log level, though 40gb+ of logs seems like a lot. I am intrigued by a buffering issue, though. Where would I check those settings in Traefik and Crowdsec?

<!-- gh-comment-id:2770435791 --> @kylepyke commented on GitHub (Apr 1, 2025): Thanks! I will check configs for log level, though 40gb+ of logs seems like a lot. I am intrigued by a buffering issue, though. Where would I check those settings in Traefik and Crowdsec?
GiteaMirror commented 2026-04-20 07:19:39 -05:00
Author
Owner
Copy Link

@leodr99 commented on GitHub (Apr 6, 2025):

had the same issue some time ago on an unrelated project.
check the current log usage of the containers sudo du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))

if like me you see some gigantic absurd sizes, it's best to either limit the logging by each container, or globally.
to keep the docker log files in check globally, just edit||create /etc/docker/daemon.json, and limit them by adding:
{ "log-opts": { "max-size": "10m", "max-file": "3" } }, to the file. (note: it's needed to restart the docker daemon to take effect)

if by the per-container route, then:

services:
   example:
     logging:
       driver: "json-file"
       options:
          max-size: "10m"
          max-file: "3"

on each service stanza. Then recreate the container/stack.

my 2cents...

<!-- gh-comment-id:2781477268 --> @leodr99 commented on GitHub (Apr 6, 2025): had the same issue some time ago on an unrelated project. check the current log usage of the containers `sudo du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))` if like me you see some gigantic absurd sizes, it's best to either limit the logging by each container, or globally. to keep the docker log files in check globally, just edit||create `/etc/docker/daemon.json`, and limit them by adding: `{ "log-opts": { "max-size": "10m", "max-file": "3" } }`, to the file. _(note: it's needed to restart the docker daemon to take effect)_ if by the per-container route, then: ``` services: example: logging: driver: "json-file" options: max-size: "10m" max-file: "3" ``` on each service stanza. Then recreate the container/stack. _my 2cents..._
GiteaMirror commented 2026-04-20 07:19:40 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 6, 2025):

Wow, that's super helpful, thanks!

<!-- gh-comment-id:2781538032 --> @kylepyke commented on GitHub (Apr 6, 2025): Wow, that's super helpful, thanks!
GiteaMirror commented 2026-04-20 07:19:41 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 6, 2025):

I have figured out the issue, but I don't have a solution!!!! @miloschwartz, maybe you have an answer? It's not logs causing the issue, it's Traefik over-using RAM, and memory swapping.

When I upload files from Immich, traefik fills up my 2GB of VPS ram pretty quickly– particularly with big video files, and starts memory swapping, then filling up my hard drive. Would limiting memory usage and swappiness solve this, or would it just crash Traefik?

<!-- gh-comment-id:2781568863 --> @kylepyke commented on GitHub (Apr 6, 2025): I have figured out the issue, but I don't have a solution!!!! @miloschwartz, maybe you have an answer? It's not logs causing the issue, it's Traefik over-using RAM, and memory swapping. When I upload files from Immich, traefik fills up my 2GB of VPS ram pretty quickly– particularly with big video files, and starts memory swapping, then filling up my hard drive. Would limiting memory usage and swappiness solve this, or would it just crash Traefik?
GiteaMirror commented 2026-04-20 07:19:42 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 6, 2025):

So this is an Immich issue. Since Immich doesn't support chunk uploads (yet), Traefik will eat all of your RAM and swap to eating your HDD. More info here: https://github.com/immich-app/immich/pull/2101 and here https://www.reddit.com/r/immich/comments/1fhvv1i/chunk_upload/

Leaving this here for the next person who finds out the hard way.

<!-- gh-comment-id:2781596210 --> @kylepyke commented on GitHub (Apr 6, 2025): So this is an Immich issue. Since Immich doesn't support chunk uploads (yet), Traefik will eat all of your RAM and swap to eating your HDD. More info here: https://github.com/immich-app/immich/pull/2101 and here https://www.reddit.com/r/immich/comments/1fhvv1i/chunk_upload/ Leaving this here for the next person who finds out the hard way.
GiteaMirror commented 2026-04-20 07:19:44 -05:00
Author
Owner
Copy Link

@miloschwartz commented on GitHub (Apr 6, 2025):

This is good to know. Thanks for the update / finding this.

<!-- gh-comment-id:2781636850 --> @miloschwartz commented on GitHub (Apr 6, 2025): This is good to know. Thanks for the update / finding this.
GiteaMirror commented 2026-04-20 07:19:44 -05:00
Author
Owner
Copy Link

@TuncTaylan commented on GitHub (Apr 10, 2025):

I think it is premature to close this issue, because other applications may also be affected by this.
For example Minio (web ui) is causing similar issues, when uploading bigger files.
There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory.

Edit: With enough RAM I'm getting following log:
crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec

The question is, where is that configuration?

<!-- gh-comment-id:2794386564 --> @TuncTaylan commented on GitHub (Apr 10, 2025): I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory. Edit: With enough RAM I'm getting following log: `crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec` The question is, where is that configuration?
GiteaMirror commented 2026-04-20 07:19:45 -05:00
Author
Owner
Copy Link

@miloschwartz commented on GitHub (Apr 10, 2025):

I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory.

Edit: With enough RAM I'm getting following log: crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec

The question is, where is that configuration?

I agree it would be good to get some more information on this. Might want to checkout Traefik and Crowdsec's documentation as it's going to be the best resource for these tools.

<!-- gh-comment-id:2795333104 --> @miloschwartz commented on GitHub (Apr 10, 2025): > I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory. > > Edit: With enough RAM I'm getting following log: `crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec` > > The question is, where is that configuration? I agree it would be good to get some more information on this. Might want to checkout Traefik and Crowdsec's documentation as it's going to be the best resource for these tools.
GiteaMirror commented 2026-04-20 07:19:46 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 10, 2025):

FYI, I upgraded the RAM on my VPS to 4GB, and still cannot upload a 2.5G video file through Immich. That should be plenty of memory... I feel like something else is going on.

<!-- gh-comment-id:2795398872 --> @kylepyke commented on GitHub (Apr 10, 2025): FYI, I upgraded the RAM on my VPS to 4GB, and still cannot upload a 2.5G video file through Immich. That should be plenty of memory... I feel like something else is going on.
GiteaMirror commented 2026-04-20 07:19:47 -05:00
Author
Owner
Copy Link

@TuncTaylan commented on GitHub (Apr 11, 2025):

I'm currently sampling more data around it, the warning message came from this line: ad40dcbb05/internal/corazawaf/transaction.go (L841)
, a component of coraza (WAF implementation, thats being used by the crowdsec-bouncer-traefik-plugin) and the required settings is this: ad40dcbb05/coraza.conf-recommended (L40)

I have to find out, how to set this within the setup in pangolin + crowdsec.

So far I've found these:

I have to test it though.

Edit: Ideal way should be writing the buffers directly to disc in order to stay in the RAM budget of 1GB (recommended VPS size!)

Edit2: - maxRequestBodyBytes or memRequestBodyBytes didn't do anything: https://doc.traefik.io/traefik/middlewares/http/buffering/

Edit3: the culprit is definitely crowdsec or some appsec plugin of it, because, if I disable crowdsecAppsecEnabled in the traefic->dynamic_settings.yaml I can upload files with 1gb and 10gb without any issues.
So I have to experiment with the setting crowdsecAppsecBodyLimit here: https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin

Edit4: Even with the adjustment to crowdsecAppsecBodyLimit, traefik gets oom'ed, so I give up and go into the weekend.

<!-- gh-comment-id:2796845054 --> @TuncTaylan commented on GitHub (Apr 11, 2025): I'm currently sampling more data around it, the warning message came from this line: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/internal/corazawaf/transaction.go#L841 , a component of coraza (WAF implementation, thats being used by the crowdsec-bouncer-traefik-plugin) and the required settings is this: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/coraza.conf-recommended#L40 I have to find out, how to set this within the setup in pangolin + crowdsec. So far I've found these: - maxRequestBodyBytes - https://doc.traefik.io/traefik/middlewares/http/buffering/ - disable_body_inspection (which makes it unsecure imho) - https://docs.crowdsec.net/docs/appsec/configuration/ - max_body_size - https://docs.crowdsec.net/docs/data_sources/http/#max_body_size I have to test it though. Edit: Ideal way should be writing the buffers directly to disc in order to stay in the RAM budget of 1GB (recommended VPS size!) Edit2: - maxRequestBodyBytes or memRequestBodyBytes didn't do anything: https://doc.traefik.io/traefik/middlewares/http/buffering/ Edit3: the culprit is definitely crowdsec or some appsec plugin of it, because, if I disable crowdsecAppsecEnabled in the traefic->dynamic_settings.yaml I can upload files with 1gb and 10gb without any issues. So I have to experiment with the setting crowdsecAppsecBodyLimit here: https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin Edit4: Even with the adjustment to crowdsecAppsecBodyLimit, traefik gets oom'ed, so I give up and go into the weekend.
GiteaMirror commented 2026-04-20 07:19:47 -05:00
Author
Owner
Copy Link

@TuncTaylan commented on GitHub (Apr 11, 2025):

I think I got this.
The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of crowdsec-bouncer-traefik-plugin. Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along).

Update the version of the plugin to currently the newest v1.4.2 in the file: config/traefik/traefik_config.yml

experimental:
  plugins:
    crowdsec:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2

After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: config/traefik/dynamic_config.yml

http:
  middlewares:
    crowdsec:
      plugin:
        crowdsec:
          crowdsecAppsecBodyLimit: 10485760

Current issue is, for every file greater than 10MB (default value or set above) there is an error message: level=error msg="Failed to process request body", which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server.

And then there is this note:
/!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x

I'll have the pull request soon.

Edit: Here is the PR https://github.com/fosrl/pangolin/pull/515#issue-2989648287

<!-- gh-comment-id:2797924344 --> @TuncTaylan commented on GitHub (Apr 11, 2025): I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of [crowdsec-bouncer-traefik-plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin). Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along). Update the version of the plugin to currently the newest v1.4.2 in the file: [config/traefik/traefik_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/traefik_config.yml) ```shell experimental: plugins: crowdsec: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.2 ``` After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: [config/traefik/dynamic_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/dynamic_config.yml) ```shell http: middlewares: crowdsec: plugin: crowdsec: crowdsecAppsecBodyLimit: 10485760 ``` Current issue is, for every file greater than 10MB (default value or set above) there is an error message: `level=error msg="Failed to process request body"`, which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server. And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x I'll have the pull request soon. Edit: Here is the PR https://github.com/fosrl/pangolin/pull/515#issue-2989648287
GiteaMirror commented 2026-04-20 07:19:48 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 14, 2025):

I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of crowdsec-bouncer-traefik-plugin. Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along).

Update the version of the plugin to currently the newest v1.4.2 in the file: config/traefik/traefik_config.yml

experimental:
plugins:
crowdsec:
moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
version: v1.4.2

After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: config/traefik/dynamic_config.yml

http:
middlewares:
crowdsec:
plugin:
crowdsec:
crowdsecAppsecBodyLimit: 10485760

Current issue is, for every file greater than 10MB (default value or set above) there is an error message: level=error msg="Failed to process request body", which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server.

And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x

I'll have the pull request soon.

Edit: Here is the PR #515 (comment)

Hmmm... This didn't work for me. I'm experiencing the same issues. Pangolin v1.2, Traefik v 3.3.5, CrowdSec v1.4.2

<!-- gh-comment-id:2802894128 --> @kylepyke commented on GitHub (Apr 14, 2025): > I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of [crowdsec-bouncer-traefik-plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin). Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along). > > Update the version of the plugin to currently the newest v1.4.2 in the file: [config/traefik/traefik_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/traefik_config.yml) > > experimental: > plugins: > crowdsec: > moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin > version: v1.4.2 > > After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: [config/traefik/dynamic_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/dynamic_config.yml) > > http: > middlewares: > crowdsec: > plugin: > crowdsec: > crowdsecAppsecBodyLimit: 10485760 > > Current issue is, for every file greater than 10MB (default value or set above) there is an error message: `level=error msg="Failed to process request body"`, which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server. > > And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x > > I'll have the pull request soon. > > Edit: Here is the PR [#515 (comment)](https://github.com/fosrl/pangolin/pull/515#issue-2989648287) Hmmm... This didn't work for me. I'm experiencing the same issues. Pangolin v1.2, Traefik v 3.3.5, CrowdSec v1.4.2
GiteaMirror commented 2026-04-20 07:19:49 -05:00
Author
Owner
Copy Link

@TuncTaylan commented on GitHub (Apr 14, 2025):

Can you share your configs (both of traefik configs)?

<!-- gh-comment-id:2802904354 --> @TuncTaylan commented on GitHub (Apr 14, 2025): Can you share your configs (both of traefik configs)?
GiteaMirror commented 2026-04-20 07:19:50 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 14, 2025):

Docker Compose:

name: pangolin
networks:
  default:
    driver: bridge
    name: pangolin
services:
  crowdsec:
    command: -t
    container_name: crowdsec
    environment:
      ACQUIRE_FILES: /var/log/traefik/*.log
      COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
      ENROLL_INSTANCE_NAME: pangolin-crowdsec
      ENROLL_TAGS: docker
      GID: "1000"
      PARSERS: crowdsecurity/whitelists
    expose:
      - 6060
    healthcheck:
      test:
        - CMD
        - cscli
        - capi
        - status
    image: crowdsecurity/crowdsec:latest
    labels:
      - traefik.enable=false
    ports:
      - 6060:6060
    restart: unless-stopped
    volumes:
      - ./config/crowdsec:/etc/crowdsec
      - ./config/crowdsec/db:/var/lib/crowdsec/data
      - ./config/crowdsec_logs/auth.log:/var/log/auth.log:ro
      - ./config/crowdsec_logs/syslog:/var/log/syslog:ro
      - ./config/crowdsec_logs:/var/log
      - ./config/traefik/logs:/var/log/traefik
  gerbil:
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    command:
      - --reachableAt=http://gerbil:3003
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config
      - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth
    container_name: gerbil
    depends_on:
      pangolin:
        condition: service_healthy
    image: fosrl/gerbil:1.0.0
    ports:
      - 51820:51820/udp
      - 443:443
      - 80:80
    restart: unless-stopped
    volumes:
      - ./config/:/var/config
pangolin:
    container_name: pangolin
    healthcheck:
      interval: 3s
      retries: 5
      test:
        - CMD
        - curl
        - -f
        - http://localhost:3001/api/v1/
      timeout: 3s
    image: fosrl/pangolin:1.2.0
    restart: unless-stopped
    volumes:
      - ./config:/app/config
  traefik:
    command:
      - --configFile=/etc/traefik/traefik_config.yml
    container_name: traefik
    depends_on:
      pangolin:
        condition: service_healthy
    image: traefik:v3.3.5
    network_mode: service:gerbil
    restart: unless-stopped
    volumes:
      - ./config/traefik:/etc/traefik:ro
      - ./config/letsencrypt:/letsencrypt
      - ./config/traefik/logs:/var/log/traefik
#    deploy:
#      resources:
#        limits:
#          memory: 1.5G
    environment:
      CLOUDFLARE_DNS_API_TOKEN: "TOKEN"

dynamic_config.yml:

http:
  middlewares:
    limit:
      buffering:
        maxRequestBodyBytes: 3000000000
        memRequestBodyBytes: 2000000000
        #maxResponseBodyBytes: 3000000000
        #memResponseBodyBytes: 2000000000
    pangolin-geoblock:
      plugin:
        geoblock:
          silentStartUp: false
          allowLocalRequests: true
          logLocalRequests: false
          logAllowedRequests: false
          logApiRequests: false
          api: "https://get.geojs.io/v1/ip/country/{ip}"
          apiTimeoutMs: 750                                 # optional
          cacheSize: 15
          forceMonthlyUpdate: false
          allowUnknownCountries: false
          unknownCountryApiResponse: "nil"
          blackListMode: false
          addCountryHeader: false
          countries:
            - US
    crowdsec:
      plugin:
        crowdsec:
          clientTrustedIPs:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 100.89.137.0/20
          crowdsecAppsecEnabled: true
          crowdsecAppsecFailureBlock: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecUnreachableBlock: true
          crowdsecAppsecBodyLimit: 5000000000
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiKey: /W2z5/YqmzHsSBydWM0xhC2FLkH+R5GZsf0dXKfdTpQ
          crowdsecLapiScheme: http
          crowdsecMode: live
          defaultDecisionSeconds: 15
          enabled: true
          #forwardedHeadersTrustedIPs:
          #  - 0.0.0.0/0
          httpTimeoutSeconds: 10
          logLevel: INFO
          updateIntervalSeconds: 15
          updateMaxFailure: 0
    default-whitelist:
      ipWhiteList:
        sourceRange:
          - 10.0.0.0/8
          - 192.168.0.0/16
          - 172.16.0.0/12
    redirect-to-https:
      redirectScheme:
        scheme: https
    security-headers:
      headers:
        contentTypeNosniff: true
        customFrameOptionsValue: SAMEORIGIN
        customResponseHeaders:
          Server: ""
          X-Forwarded-Proto: https
          X-Powered-By: ""
        forceSTSHeader: true
        hostsProxyHeaders:
          - X-Forwarded-Host
        referrerPolicy: strict-origin-when-cross-origin
        sslProxyHeaders:
          X-Forwarded-Proto: https
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 63072000
  routers:
    api-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`) && PathPrefix(`/api/v1`)
      service: api-service
      tls:
        certResolver: letsencrypt
    main-app-router-redirect:
      entryPoints:
        - web
      middlewares:
        - redirect-to-https
      rule: Host(`pangolin.mydomain.com`)
      service: next-service
    next-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`) && !PathPrefix(`/api/v1`)
      service: next-service
      tls:
        certResolver: letsencrypt
        domains:
          - main: "mydomain.com"
            sans:
              - "*.mydomain.com"
    ws-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`)
      service: api-service
      tls:
        certResolver: letsencrypt
  services:
    api-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3000
    next-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3002

traefik_config.yml:

accessLog:
  bufferingSize: 100
  fields:
    defaultMode: drop
    headers:
      defaultMode: drop
      names:
        Authorization: redact
        Content-Type: keep
        Cookie: redact
        User-Agent: keep
        X-Forwarded-For: keep
        X-Forwarded-Proto: keep
        X-Real-Ip: keep
    names:
      ClientAddr: keep
      ClientHost: keep
      DownstreamContentSize: keep
      DownstreamStatus: keep
      Duration: keep
      RequestMethod: keep
      RequestPath: keep
      RequestProtocol: keep
      RetryAttempts: keep
      ServiceName: keep
      StartUTC: keep
      TLSCipher: keep
      TLSVersion: keep
  filePath: /var/log/traefik/access.log
  filters:
    minDuration: 100ms
    retryAttempts: true
    statusCodes:
      - 200-299
      - 400-499
      - 500-599
  format: json
api:
  dashboard: true
  insecure: true
certificatesResolvers:
  letsencrypt:
    acme:
      dnsChallenge:
        provider: cloudflare
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: mydomain@domain.com
      httpChallenge:
        entryPoint: web
      storage: /letsencrypt/acme.json
entryPoints:
  web:
    address: ':80'
  websecure:
    address: ':443'
    http:
      middlewares:
        - pangolin-geoblock@file
        - crowdsec@file
      tls:
        certResolver: letsencrypt
    transport:
      respondingTimeouts:
        readTimeout: 30m
experimental:
  plugins:
    badger:
      moduleName: github.com/fosrl/badger
      version: v1.1.0
    crowdsec:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2
    geoblock:
      moduleName: github.com/PascalMinder/geoblock
      version: v0.3.2
log:
  format: json
  level: INFO
providers:
  file:
    filename: /etc/traefik/dynamic_config.yml
  http:
    endpoint: http://pangolin:3001/api/v1/traefik-config
    pollInterval: 5s
serversTransport:
  insecureSkipVerify: true
<!-- gh-comment-id:2803002200 --> @kylepyke commented on GitHub (Apr 14, 2025): Docker Compose: ``` name: pangolin networks: default: driver: bridge name: pangolin services: crowdsec: command: -t container_name: crowdsec environment: ACQUIRE_FILES: /var/log/traefik/*.log COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ENROLL_INSTANCE_NAME: pangolin-crowdsec ENROLL_TAGS: docker GID: "1000" PARSERS: crowdsecurity/whitelists expose: - 6060 healthcheck: test: - CMD - cscli - capi - status image: crowdsecurity/crowdsec:latest labels: - traefik.enable=false ports: - 6060:6060 restart: unless-stopped volumes: - ./config/crowdsec:/etc/crowdsec - ./config/crowdsec/db:/var/lib/crowdsec/data - ./config/crowdsec_logs/auth.log:/var/log/auth.log:ro - ./config/crowdsec_logs/syslog:/var/log/syslog:ro - ./config/crowdsec_logs:/var/log - ./config/traefik/logs:/var/log/traefik gerbil: cap_add: - NET_ADMIN - SYS_MODULE command: - --reachableAt=http://gerbil:3003 - --generateAndSaveKeyTo=/var/config/key - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth container_name: gerbil depends_on: pangolin: condition: service_healthy image: fosrl/gerbil:1.0.0 ports: - 51820:51820/udp - 443:443 - 80:80 restart: unless-stopped volumes: - ./config/:/var/config pangolin: container_name: pangolin healthcheck: interval: 3s retries: 5 test: - CMD - curl - -f - http://localhost:3001/api/v1/ timeout: 3s image: fosrl/pangolin:1.2.0 restart: unless-stopped volumes: - ./config:/app/config traefik: command: - --configFile=/etc/traefik/traefik_config.yml container_name: traefik depends_on: pangolin: condition: service_healthy image: traefik:v3.3.5 network_mode: service:gerbil restart: unless-stopped volumes: - ./config/traefik:/etc/traefik:ro - ./config/letsencrypt:/letsencrypt - ./config/traefik/logs:/var/log/traefik # deploy: # resources: # limits: # memory: 1.5G environment: CLOUDFLARE_DNS_API_TOKEN: "TOKEN" ``` dynamic_config.yml: ``` http: middlewares: limit: buffering: maxRequestBodyBytes: 3000000000 memRequestBodyBytes: 2000000000 #maxResponseBodyBytes: 3000000000 #memResponseBodyBytes: 2000000000 pangolin-geoblock: plugin: geoblock: silentStartUp: false allowLocalRequests: true logLocalRequests: false logAllowedRequests: false logApiRequests: false api: "https://get.geojs.io/v1/ip/country/{ip}" apiTimeoutMs: 750 # optional cacheSize: 15 forceMonthlyUpdate: false allowUnknownCountries: false unknownCountryApiResponse: "nil" blackListMode: false addCountryHeader: false countries: - US crowdsec: plugin: crowdsec: clientTrustedIPs: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 - 100.89.137.0/20 crowdsecAppsecEnabled: true crowdsecAppsecFailureBlock: true crowdsecAppsecHost: crowdsec:7422 crowdsecAppsecUnreachableBlock: true crowdsecAppsecBodyLimit: 5000000000 crowdsecLapiHost: crowdsec:8080 crowdsecLapiKey: /W2z5/YqmzHsSBydWM0xhC2FLkH+R5GZsf0dXKfdTpQ crowdsecLapiScheme: http crowdsecMode: live defaultDecisionSeconds: 15 enabled: true #forwardedHeadersTrustedIPs: # - 0.0.0.0/0 httpTimeoutSeconds: 10 logLevel: INFO updateIntervalSeconds: 15 updateMaxFailure: 0 default-whitelist: ipWhiteList: sourceRange: - 10.0.0.0/8 - 192.168.0.0/16 - 172.16.0.0/12 redirect-to-https: redirectScheme: scheme: https security-headers: headers: contentTypeNosniff: true customFrameOptionsValue: SAMEORIGIN customResponseHeaders: Server: "" X-Forwarded-Proto: https X-Powered-By: "" forceSTSHeader: true hostsProxyHeaders: - X-Forwarded-Host referrerPolicy: strict-origin-when-cross-origin sslProxyHeaders: X-Forwarded-Proto: https stsIncludeSubdomains: true stsPreload: true stsSeconds: 63072000 routers: api-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) && PathPrefix(`/api/v1`) service: api-service tls: certResolver: letsencrypt main-app-router-redirect: entryPoints: - web middlewares: - redirect-to-https rule: Host(`pangolin.mydomain.com`) service: next-service next-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) && !PathPrefix(`/api/v1`) service: next-service tls: certResolver: letsencrypt domains: - main: "mydomain.com" sans: - "*.mydomain.com" ws-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) service: api-service tls: certResolver: letsencrypt services: api-service: loadBalancer: servers: - url: http://pangolin:3000 next-service: loadBalancer: servers: - url: http://pangolin:3002 ``` traefik_config.yml: ``` accessLog: bufferingSize: 100 fields: defaultMode: drop headers: defaultMode: drop names: Authorization: redact Content-Type: keep Cookie: redact User-Agent: keep X-Forwarded-For: keep X-Forwarded-Proto: keep X-Real-Ip: keep names: ClientAddr: keep ClientHost: keep DownstreamContentSize: keep DownstreamStatus: keep Duration: keep RequestMethod: keep RequestPath: keep RequestProtocol: keep RetryAttempts: keep ServiceName: keep StartUTC: keep TLSCipher: keep TLSVersion: keep filePath: /var/log/traefik/access.log filters: minDuration: 100ms retryAttempts: true statusCodes: - 200-299 - 400-499 - 500-599 format: json api: dashboard: true insecure: true certificatesResolvers: letsencrypt: acme: dnsChallenge: provider: cloudflare caServer: https://acme-v02.api.letsencrypt.org/directory email: mydomain@domain.com httpChallenge: entryPoint: web storage: /letsencrypt/acme.json entryPoints: web: address: ':80' websecure: address: ':443' http: middlewares: - pangolin-geoblock@file - crowdsec@file tls: certResolver: letsencrypt transport: respondingTimeouts: readTimeout: 30m experimental: plugins: badger: moduleName: github.com/fosrl/badger version: v1.1.0 crowdsec: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.2 geoblock: moduleName: github.com/PascalMinder/geoblock version: v0.3.2 log: format: json level: INFO providers: file: filename: /etc/traefik/dynamic_config.yml http: endpoint: http://pangolin:3001/api/v1/traefik-config pollInterval: 5s serversTransport: insecureSkipVerify: true ```
GiteaMirror commented 2026-04-20 07:19:51 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 14, 2025):

Immich running on separate home server, docker-compose:

#
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    devices:
      - /dev/dri:/dev/dri
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # Note the `-cuda` at the end
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    # Note the lack of an `extends` section
    device_cgroup_rules:
      - 'c 189:* rmw'
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /dev/bus/usb:/dev/bus/usb

      - model-cache:/cache
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:XXXX
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:XXXX
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:
<!-- gh-comment-id:2803021247 --> @kylepyke commented on GitHub (Apr 14, 2025): Immich running on separate home server, docker-compose: ``` # # WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose # # Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} devices: - /dev/dri:/dev/dri volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro ports: - '2283:2283' depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # Note the `-cuda` at the end image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino # Note the lack of an `extends` section device_cgroup_rules: - 'c 189:* rmw' devices: - /dev/dri:/dev/dri volumes: - /dev/bus/usb:/dev/bus/usb - model-cache:/cache restart: always redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:XXXX healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:XXXX environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data healthcheck: test: >- pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1; Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m start_interval: 30s start_period: 5m command: >- postgres -c shared_preload_libraries=vectors.so -c 'search_path="$$user", public, vectors' -c logging_collector=on -c max_wal_size=2GB -c shared_buffers=512MB -c wal_compression=on restart: always volumes: model-cache: ```
GiteaMirror commented 2026-04-20 07:19:52 -05:00
Author
Owner
Copy Link

@TuncTaylan commented on GitHub (Apr 15, 2025):

Did you also have the same problem with the default value of crowdsecAppsecBodyLimit with 10MB?
You've set it to up to 5000000000 (approx. 4.5GB).

Just for this I've setup a resource to my immich at home with the default setting of crowdsecAppsecBodyLimit 10485760.

I could upload a 2GB test video without any issues.

Image

Image

At first I tried to set crowdsecAppsecBodyLimit value impulsively to the "highest" file size I would upload, that failed also.

So try the default setting (10MB), it works.

<!-- gh-comment-id:2803984706 --> @TuncTaylan commented on GitHub (Apr 15, 2025): Did you also have the same problem with the default value of crowdsecAppsecBodyLimit with 10MB? You've set it to up to 5000000000 (approx. 4.5GB). Just for this I've setup a resource to my immich at home with the default setting of crowdsecAppsecBodyLimit 10485760. I could upload a 2GB test video without any issues. ![Image](https://github.com/user-attachments/assets/4b9a6eae-fc5b-440c-a22a-5d98a118a930) ![Image](https://github.com/user-attachments/assets/eb2680ed-4407-4aaa-b194-766b1af058b9) At first I tried to set crowdsecAppsecBodyLimit value impulsively to the "highest" file size I would upload, that failed also. So try the default setting (10MB), it works.
GiteaMirror commented 2026-04-20 07:19:53 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Apr 15, 2025):

Oh wow, this worked! Thanks! @TuncTaylan @oschwartz10612 @miloschwartz

<!-- gh-comment-id:2806424739 --> @kylepyke commented on GitHub (Apr 15, 2025): Oh wow, this worked! Thanks! @TuncTaylan @oschwartz10612 @miloschwartz
GiteaMirror commented 2026-04-20 07:19:54 -05:00
Author
Owner
Copy Link

@muffn commented on GitHub (Oct 21, 2025):

is this issue truly resolved? I think I have the same problem, as soon as I upload a large file (no matter the service, opencloud or immich break my vps) the logs are flooded and my vps becomes unresponsive until I restart it.

It is definitely related to crowdsec, as commenting out the crowdsec middleware "gets rid of the issue" but that isn't really a solution. It also starts right at the 10_000 bytes mark just as referenced in this issue.

docker-compose.yml

name: pangolin
networks:
  default:
    driver: bridge
    enable_ipv6: true
    name: pangolin
services:
  crowdsec:
    command: -t
    container_name: crowdsec
    environment:
      COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
      ENROLL_INSTANCE_NAME: pangolin-crowdsec
      ENROLL_TAGS: docker
      GID: "1000"
      PARSERS: crowdsecurity/whitelists
    healthcheck:
      interval: 10s
      retries: 15
      test:
        - CMD
        - cscli
        - capi
        - status
      timeout: 10s
    image: docker.io/crowdsecurity/crowdsec:latest
    labels:
      - traefik.enable=false
    ports:
      - 6060:6060
    restart: unless-stopped
    volumes:
      - ./config/crowdsec:/etc/crowdsec
      - ./config/crowdsec/db:/var/lib/crowdsec/data
      - ./config/traefik/logs:/var/log/traefik
  gerbil:
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    command:
      - --reachableAt=http://gerbil:3003
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config
      - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth
    container_name: gerbil
    depends_on:
      pangolin:
        condition: service_healthy
    image: docker.io/fosrl/gerbil:1.2.2
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
      - 443:443
      - 80:80
    restart: unless-stopped
    volumes:
      - ./config/:/var/config
  pangolin:
    container_name: pangolin
    healthcheck:
      interval: 10s
      retries: 15
      test:
        - CMD
        - curl
        - -f
        - http://localhost:3001/api/v1/
      timeout: 10s
    image: docker.io/fosrl/pangolin:1.11.0
    restart: unless-stopped
    volumes:
      - ./config:/app/config
  traefik:
    command:
      - --configFile=/etc/traefik/traefik_config.yml
    container_name: traefik
    depends_on:
      crowdsec:
        condition: service_healthy
      pangolin:
        condition: service_healthy
    image: docker.io/traefik:v3.5.3
    network_mode: service:gerbil
    restart: unless-stopped
    volumes:
      - ./config/traefik:/etc/traefik:ro
      - ./config/letsencrypt:/letsencrypt
      - ./config/traefik/logs:/var/log/traefik

dynamic_config.yml

http:
  middlewares:
  # added this in attempt to fix the problem
    limit:
      buffering:
        maxRequestBodyBytes: 3000000000
        memRequestBodyBytes: 2000000000
    crowdsec:
      plugin:
        crowdsec:
          clientTrustedIPs:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 100.89.137.0/20
          crowdsecAppsecBodyLimit: 10485760
          crowdsecAppsecEnabled: true
          crowdsecAppsecFailureBlock: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecUnreachableBlock: true
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiKey: <apikey>
          crowdsecLapiScheme: http
          crowdsecMode: live
          defaultDecisionSeconds: 15
          enabled: true
          forwardedHeadersTrustedIPs:
            - 0.0.0.0/0
          httpTimeoutSeconds: 10
          logLevel: INFO
          updateIntervalSeconds: 15
          updateMaxFailure: 0
    default-whitelist:
      ipWhiteList:
        sourceRange:
          - 10.0.0.0/8
          - 192.168.0.0/16
          - 172.16.0.0/12
    redirect-to-https:
      redirectScheme:
        scheme: https
    security-headers:
      headers:
        contentTypeNosniff: true
        customFrameOptionsValue: SAMEORIGIN
        customResponseHeaders:
          Server: ""
          X-Forwarded-Proto: https
          X-Powered-By: ""
        forceSTSHeader: true
        hostsProxyHeaders:
          - X-Forwarded-Host
        referrerPolicy: strict-origin-when-cross-origin
        sslProxyHeaders:
          X-Forwarded-Proto: https
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 63072000
  routers:
    api-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.domain.com`) && PathPrefix(`/api/v1`)
      service: api-service
      tls:
        certResolver: letsencrypt
    main-app-router-redirect:
      entryPoints:
        - web
      middlewares:
        - redirect-to-https
      rule: Host(`pangolin.domain.com`)
      service: next-service
    next-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.domain.com`) && !PathPrefix(`/api/v1`)
      service: next-service
      tls:
        certResolver: letsencrypt
    ws-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.domain.com`)
      service: api-service
      tls:
        certResolver: letsencrypt
  services:
    api-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3000
    next-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3002

traefik_config.yml

accessLog:
  bufferingSize: 100
  fields:
    defaultMode: drop
    headers:
      defaultMode: drop
      names:
        Authorization: redact
        Content-Type: keep
        Cookie: redact
        User-Agent: keep
        X-Forwarded-For: keep
        X-Forwarded-Proto: keep
        X-Real-Ip: keep
    names:
      ClientAddr: keep
      ClientHost: keep
      DownstreamContentSize: keep
      DownstreamStatus: keep
      Duration: keep
      RequestMethod: keep
      RequestPath: keep
      RequestProtocol: keep
      RetryAttempts: keep
      ServiceName: keep
      StartUTC: keep
      TLSCipher: keep
      TLSVersion: keep
  filePath: /var/log/traefik/access.log
  filters:
    minDuration: 100ms
    retryAttempts: true
    statusCodes:
      - 200-299
      - 400-499
      - 500-599
  format: json
api:
  dashboard: true
  insecure: true
certificatesResolvers:
  letsencrypt:
    acme:
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: <email>
      httpChallenge:
        entryPoint: web
      storage: /letsencrypt/acme.json
entryPoints:
  web:
    address: :80
  websecure:
    address: :443
    http:
      middlewares:
        - crowdsec@file
      tls:
        certResolver: letsencrypt
    transport:
      respondingTimeouts:
        readTimeout: 30m
experimental:
  plugins:
    badger:
      moduleName: github.com/fosrl/badger
      version: v1.2.0
    crowdsec:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.5
log:
  compress: true
  format: json
  level: INFO
  maxAge: 3
  maxBackups: 3
  maxSize: 100
providers:
  file:
    filename: /etc/traefik/dynamic_config.yml
  http:
    endpoint: http://pangolin:3001/api/v1/traefik-config
    pollInterval: 5s
serversTransport:
  insecureSkipVerify: true
<!-- gh-comment-id:3428934206 --> @muffn commented on GitHub (Oct 21, 2025): is this issue truly resolved? I think I have the same problem, as soon as I upload a large file (no matter the service, opencloud or immich break my vps) the logs are flooded and my vps becomes unresponsive until I restart it. It is definitely related to crowdsec, as commenting out the crowdsec middleware "gets rid of the issue" but that isn't really a solution. It also starts right at the 10_000 bytes mark just as referenced in this issue. docker-compose.yml ```yml name: pangolin networks: default: driver: bridge enable_ipv6: true name: pangolin services: crowdsec: command: -t container_name: crowdsec environment: COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ENROLL_INSTANCE_NAME: pangolin-crowdsec ENROLL_TAGS: docker GID: "1000" PARSERS: crowdsecurity/whitelists healthcheck: interval: 10s retries: 15 test: - CMD - cscli - capi - status timeout: 10s image: docker.io/crowdsecurity/crowdsec:latest labels: - traefik.enable=false ports: - 6060:6060 restart: unless-stopped volumes: - ./config/crowdsec:/etc/crowdsec - ./config/crowdsec/db:/var/lib/crowdsec/data - ./config/traefik/logs:/var/log/traefik gerbil: cap_add: - NET_ADMIN - SYS_MODULE command: - --reachableAt=http://gerbil:3003 - --generateAndSaveKeyTo=/var/config/key - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth container_name: gerbil depends_on: pangolin: condition: service_healthy image: docker.io/fosrl/gerbil:1.2.2 ports: - 51820:51820/udp - 21820:21820/udp - 443:443 - 80:80 restart: unless-stopped volumes: - ./config/:/var/config pangolin: container_name: pangolin healthcheck: interval: 10s retries: 15 test: - CMD - curl - -f - http://localhost:3001/api/v1/ timeout: 10s image: docker.io/fosrl/pangolin:1.11.0 restart: unless-stopped volumes: - ./config:/app/config traefik: command: - --configFile=/etc/traefik/traefik_config.yml container_name: traefik depends_on: crowdsec: condition: service_healthy pangolin: condition: service_healthy image: docker.io/traefik:v3.5.3 network_mode: service:gerbil restart: unless-stopped volumes: - ./config/traefik:/etc/traefik:ro - ./config/letsencrypt:/letsencrypt - ./config/traefik/logs:/var/log/traefik ``` dynamic_config.yml ```yml http: middlewares: # added this in attempt to fix the problem limit: buffering: maxRequestBodyBytes: 3000000000 memRequestBodyBytes: 2000000000 crowdsec: plugin: crowdsec: clientTrustedIPs: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 - 100.89.137.0/20 crowdsecAppsecBodyLimit: 10485760 crowdsecAppsecEnabled: true crowdsecAppsecFailureBlock: true crowdsecAppsecHost: crowdsec:7422 crowdsecAppsecUnreachableBlock: true crowdsecLapiHost: crowdsec:8080 crowdsecLapiKey: <apikey> crowdsecLapiScheme: http crowdsecMode: live defaultDecisionSeconds: 15 enabled: true forwardedHeadersTrustedIPs: - 0.0.0.0/0 httpTimeoutSeconds: 10 logLevel: INFO updateIntervalSeconds: 15 updateMaxFailure: 0 default-whitelist: ipWhiteList: sourceRange: - 10.0.0.0/8 - 192.168.0.0/16 - 172.16.0.0/12 redirect-to-https: redirectScheme: scheme: https security-headers: headers: contentTypeNosniff: true customFrameOptionsValue: SAMEORIGIN customResponseHeaders: Server: "" X-Forwarded-Proto: https X-Powered-By: "" forceSTSHeader: true hostsProxyHeaders: - X-Forwarded-Host referrerPolicy: strict-origin-when-cross-origin sslProxyHeaders: X-Forwarded-Proto: https stsIncludeSubdomains: true stsPreload: true stsSeconds: 63072000 routers: api-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.domain.com`) && PathPrefix(`/api/v1`) service: api-service tls: certResolver: letsencrypt main-app-router-redirect: entryPoints: - web middlewares: - redirect-to-https rule: Host(`pangolin.domain.com`) service: next-service next-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.domain.com`) && !PathPrefix(`/api/v1`) service: next-service tls: certResolver: letsencrypt ws-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.domain.com`) service: api-service tls: certResolver: letsencrypt services: api-service: loadBalancer: servers: - url: http://pangolin:3000 next-service: loadBalancer: servers: - url: http://pangolin:3002 ``` traefik_config.yml ```yml accessLog: bufferingSize: 100 fields: defaultMode: drop headers: defaultMode: drop names: Authorization: redact Content-Type: keep Cookie: redact User-Agent: keep X-Forwarded-For: keep X-Forwarded-Proto: keep X-Real-Ip: keep names: ClientAddr: keep ClientHost: keep DownstreamContentSize: keep DownstreamStatus: keep Duration: keep RequestMethod: keep RequestPath: keep RequestProtocol: keep RetryAttempts: keep ServiceName: keep StartUTC: keep TLSCipher: keep TLSVersion: keep filePath: /var/log/traefik/access.log filters: minDuration: 100ms retryAttempts: true statusCodes: - 200-299 - 400-499 - 500-599 format: json api: dashboard: true insecure: true certificatesResolvers: letsencrypt: acme: caServer: https://acme-v02.api.letsencrypt.org/directory email: <email> httpChallenge: entryPoint: web storage: /letsencrypt/acme.json entryPoints: web: address: :80 websecure: address: :443 http: middlewares: - crowdsec@file tls: certResolver: letsencrypt transport: respondingTimeouts: readTimeout: 30m experimental: plugins: badger: moduleName: github.com/fosrl/badger version: v1.2.0 crowdsec: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.5 log: compress: true format: json level: INFO maxAge: 3 maxBackups: 3 maxSize: 100 providers: file: filename: /etc/traefik/dynamic_config.yml http: endpoint: http://pangolin:3001/api/v1/traefik-config pollInterval: 5s serversTransport: insecureSkipVerify: true ```
GiteaMirror commented 2026-04-20 07:19:55 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Jan 14, 2026):

Hmmm... I'm also still experiencing this issue with OpenCloud now. Seems my system bloats until it crashes.

<!-- gh-comment-id:3748042393 --> @kylepyke commented on GitHub (Jan 14, 2026): Hmmm... I'm also still experiencing this issue with OpenCloud now. Seems my system bloats until it crashes.
GiteaMirror commented 2026-04-20 07:19:56 -05:00
Author
Owner
Copy Link

@muffn commented on GitHub (Jan 14, 2026):

Hmmm... I'm also still experiencing this issue with OpenCloud now. Seems my system bloats until it crashes.

Probably this issue: https://github.com/fosrl/pangolin/issues/2120

I resolved the Instant crash by setting the appsec bodylimit

crowdsecAppsecBodyLimit: 1048576

But crowdsec is not usable on a 1GB vps right now due to the memory leak. I upgraded to 4GB RAM and all issues went away

<!-- gh-comment-id:3748350598 --> @muffn commented on GitHub (Jan 14, 2026): > Hmmm... I'm also still experiencing this issue with OpenCloud now. Seems my system bloats until it crashes. Probably this issue: https://github.com/fosrl/pangolin/issues/2120 I resolved the Instant crash by setting the appsec bodylimit ``` crowdsecAppsecBodyLimit: 1048576 ``` But crowdsec is not usable on a 1GB vps right now due to the memory leak. I upgraded to 4GB RAM and all issues went away
GiteaMirror commented 2026-04-20 07:19:57 -05:00
Author
Owner
Copy Link

@kylepyke commented on GitHub (Jan 14, 2026):

I've set the appsec bodylimit, and have 4gb on my VPS. Still have that issue.

<!-- gh-comment-id:3750465634 --> @kylepyke commented on GitHub (Jan 14, 2026): I've set the appsec bodylimit, and have 4gb on my VPS. Still have that issue.
GiteaMirror commented 2026-04-20 07:19:57 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jan 29, 2026):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

<!-- gh-comment-id:3814667245 --> @github-actions[bot] commented on GitHub (Jan 29, 2026): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.
GiteaMirror commented 2026-04-20 07:19:58 -05:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Feb 12, 2026):

This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.

<!-- gh-comment-id:3887993789 --> @github-actions[bot] commented on GitHub (Feb 12, 2026): This issue has been automatically closed due to inactivity. If you believe this is still relevant, please open a new issue with up-to-date information.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
api
authentication
bug
config
dependencies
docker
documentation
enhancement
good first issue
help wanted
Improvement
Look Into
needs investigating
networking
new feature
non-critical bug
potential bug
pull-request
Mirrored from GitHub Pull Request
 question
reverse proxy
Security
stale
ui
wontfix
No Label stale
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#3391
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?