Uploading Files to Home Server Causing Pangolin VPS to Explode in Size #196

@oschwartz10612 commented on GitHub (Apr 1, 2025):

Hello!

It sounds like one of the containers running on your Pangolin VPS (likely Traefik or potentially CrowdSec) might be logging excessive data or temporarily buffering the uploads within its Docker layer, causing the /var/lib/docker/overlay2 growth. You could try checking the logs for each container (docker logs <container_name>) during an upload to see if one is generating massive output, and also review the Traefik and CrowdSec configurations to ensure log levels aren't set to debug or that there isn't unexpected buffering enabled.

@oschwartz10612 commented on GitHub (Apr 1, 2025): Hello! It sounds like one of the containers running on your Pangolin VPS (likely Traefik or potentially CrowdSec) might be logging excessive data or temporarily buffering the uploads within its Docker layer, causing the /var/lib/docker/overlay2 growth. You could try checking the logs for each container (docker logs <container_name>) during an upload to see if one is generating massive output, and also review the Traefik and CrowdSec configurations to ensure log levels aren't set to debug or that there isn't unexpected buffering enabled.

GiteaMirror commented

@kylepyke commented on GitHub (Apr 1, 2025):

Thanks! I will check configs for log level, though 40gb+ of logs seems like a lot. I am intrigued by a buffering issue, though. Where would I check those settings in Traefik and Crowdsec?

@kylepyke commented on GitHub (Apr 1, 2025): Thanks! I will check configs for log level, though 40gb+ of logs seems like a lot. I am intrigued by a buffering issue, though. Where would I check those settings in Traefik and Crowdsec?

GiteaMirror commented

@leodr99 commented on GitHub (Apr 6, 2025):

had the same issue some time ago on an unrelated project.
check the current log usage of the containers sudo du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))

if like me you see some gigantic absurd sizes, it's best to either limit the logging by each container, or globally.
to keep the docker log files in check globally, just edit||create /etc/docker/daemon.json, and limit them by adding:
{ "log-opts": { "max-size": "10m", "max-file": "3" } }, to the file. (note: it's needed to restart the docker daemon to take effect)

if by the per-container route, then:

services:
   example:
     logging:
       driver: "json-file"
       options:
          max-size: "10m"
          max-file: "3"

on each service stanza. Then recreate the container/stack.

my 2cents...

@leodr99 commented on GitHub (Apr 6, 2025): had the same issue some time ago on an unrelated project. check the current log usage of the containers `sudo du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))` if like me you see some gigantic absurd sizes, it's best to either limit the logging by each container, or globally. to keep the docker log files in check globally, just edit||create `/etc/docker/daemon.json`, and limit them by adding: `{ "log-opts": { "max-size": "10m", "max-file": "3" } }`, to the file. _(note: it's needed to restart the docker daemon to take effect)_ if by the per-container route, then: ``` services: example: logging: driver: "json-file" options: max-size: "10m" max-file: "3" ``` on each service stanza. Then recreate the container/stack. _my 2cents..._

GiteaMirror commented

@kylepyke commented on GitHub (Apr 6, 2025):

Wow, that's super helpful, thanks!

@kylepyke commented on GitHub (Apr 6, 2025): Wow, that's super helpful, thanks!

GiteaMirror commented

@kylepyke commented on GitHub (Apr 6, 2025):

I have figured out the issue, but I don't have a solution!!!! @miloschwartz, maybe you have an answer? It's not logs causing the issue, it's Traefik over-using RAM, and memory swapping.

When I upload files from Immich, traefik fills up my 2GB of VPS ram pretty quickly– particularly with big video files, and starts memory swapping, then filling up my hard drive. Would limiting memory usage and swappiness solve this, or would it just crash Traefik?

@kylepyke commented on GitHub (Apr 6, 2025): I have figured out the issue, but I don't have a solution!!!! @miloschwartz, maybe you have an answer? It's not logs causing the issue, it's Traefik over-using RAM, and memory swapping. When I upload files from Immich, traefik fills up my 2GB of VPS ram pretty quickly– particularly with big video files, and starts memory swapping, then filling up my hard drive. Would limiting memory usage and swappiness solve this, or would it just crash Traefik?

GiteaMirror commented

@kylepyke commented on GitHub (Apr 6, 2025):

So this is an Immich issue. Since Immich doesn't support chunk uploads (yet), Traefik will eat all of your RAM and swap to eating your HDD. More info here: https://github.com/immich-app/immich/pull/2101 and here https://www.reddit.com/r/immich/comments/1fhvv1i/chunk_upload/

Leaving this here for the next person who finds out the hard way.

@kylepyke commented on GitHub (Apr 6, 2025): So this is an Immich issue. Since Immich doesn't support chunk uploads (yet), Traefik will eat all of your RAM and swap to eating your HDD. More info here: https://github.com/immich-app/immich/pull/2101 and here https://www.reddit.com/r/immich/comments/1fhvv1i/chunk_upload/ Leaving this here for the next person who finds out the hard way.

GiteaMirror commented

@miloschwartz commented on GitHub (Apr 6, 2025):

This is good to know. Thanks for the update / finding this.

@miloschwartz commented on GitHub (Apr 6, 2025): This is good to know. Thanks for the update / finding this.

GiteaMirror commented

@TuncTaylan commented on GitHub (Apr 10, 2025):

I think it is premature to close this issue, because other applications may also be affected by this.
For example Minio (web ui) is causing similar issues, when uploading bigger files.
There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory.

Edit: With enough RAM I'm getting following log:
crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec

The question is, where is that configuration?

@TuncTaylan commented on GitHub (Apr 10, 2025): I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory. Edit: With enough RAM I'm getting following log: `crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec` The question is, where is that configuration?

GiteaMirror commented

@miloschwartz commented on GitHub (Apr 10, 2025):

I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory.

Edit: With enough RAM I'm getting following log: crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec

The question is, where is that configuration?

I agree it would be good to get some more information on this. Might want to checkout Traefik and Crowdsec's documentation as it's going to be the best resource for these tools.

@miloschwartz commented on GitHub (Apr 10, 2025): > I think it is premature to close this issue, because other applications may also be affected by this. For example Minio (web ui) is causing similar issues, when uploading bigger files. There needs to be options or documentation about the configs around the buffering sizes or caches. It should be possible to use write to directly to disk instead of caching in the memory. > > Edit: With enough RAM I'm getting following log: `crowdsec | time="2025-04-10T16:28:59Z" level=warning msg="Disrupting transaction with body size above the configured limit (Action Reject)" band=inband chain_rule_id=3231508716 name=myAppSecComponent runner_uuid=1111e24a-bc7f-4862-9709-b69e66382a5a tx_id=ed578e15-f17c-4166-9172-75816b4dff66 type=appsec` > > The question is, where is that configuration? I agree it would be good to get some more information on this. Might want to checkout Traefik and Crowdsec's documentation as it's going to be the best resource for these tools.

GiteaMirror commented

@kylepyke commented on GitHub (Apr 10, 2025):

FYI, I upgraded the RAM on my VPS to 4GB, and still cannot upload a 2.5G video file through Immich. That should be plenty of memory... I feel like something else is going on.

@kylepyke commented on GitHub (Apr 10, 2025): FYI, I upgraded the RAM on my VPS to 4GB, and still cannot upload a 2.5G video file through Immich. That should be plenty of memory... I feel like something else is going on.

GiteaMirror commented

@TuncTaylan commented on GitHub (Apr 11, 2025):

I'm currently sampling more data around it, the warning message came from this line: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/internal/corazawaf/transaction.go#L841
, a component of coraza (WAF implementation, thats being used by the crowdsec-bouncer-traefik-plugin) and the required settings is this: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/coraza.conf-recommended#L40

I have to find out, how to set this within the setup in pangolin + crowdsec.

So far I've found these:

maxRequestBodyBytes - https://doc.traefik.io/traefik/middlewares/http/buffering/
disable_body_inspection (which makes it unsecure imho) - https://docs.crowdsec.net/docs/appsec/configuration/
max_body_size - https://docs.crowdsec.net/docs/data_sources/http/#max_body_size

I have to test it though.

Edit: Ideal way should be writing the buffers directly to disc in order to stay in the RAM budget of 1GB (recommended VPS size!)

Edit2: - maxRequestBodyBytes or memRequestBodyBytes didn't do anything: https://doc.traefik.io/traefik/middlewares/http/buffering/

Edit3: the culprit is definitely crowdsec or some appsec plugin of it, because, if I disable crowdsecAppsecEnabled in the traefic->dynamic_settings.yaml I can upload files with 1gb and 10gb without any issues.
So I have to experiment with the setting crowdsecAppsecBodyLimit here: https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin

Edit4: Even with the adjustment to crowdsecAppsecBodyLimit, traefik gets oom'ed, so I give up and go into the weekend.

@TuncTaylan commented on GitHub (Apr 11, 2025): I'm currently sampling more data around it, the warning message came from this line: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/internal/corazawaf/transaction.go#L841 , a component of coraza (WAF implementation, thats being used by the crowdsec-bouncer-traefik-plugin) and the required settings is this: https://github.com/corazawaf/coraza/blob/ad40dcbb0596dc6763799cc66e1ae9b43fce261d/coraza.conf-recommended#L40 I have to find out, how to set this within the setup in pangolin + crowdsec. So far I've found these: - maxRequestBodyBytes - https://doc.traefik.io/traefik/middlewares/http/buffering/ - disable_body_inspection (which makes it unsecure imho) - https://docs.crowdsec.net/docs/appsec/configuration/ - max_body_size - https://docs.crowdsec.net/docs/data_sources/http/#max_body_size I have to test it though. Edit: Ideal way should be writing the buffers directly to disc in order to stay in the RAM budget of 1GB (recommended VPS size!) Edit2: - maxRequestBodyBytes or memRequestBodyBytes didn't do anything: https://doc.traefik.io/traefik/middlewares/http/buffering/ Edit3: the culprit is definitely crowdsec or some appsec plugin of it, because, if I disable crowdsecAppsecEnabled in the traefic->dynamic_settings.yaml I can upload files with 1gb and 10gb without any issues. So I have to experiment with the setting crowdsecAppsecBodyLimit here: https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin Edit4: Even with the adjustment to crowdsecAppsecBodyLimit, traefik gets oom'ed, so I give up and go into the weekend.

GiteaMirror commented

@TuncTaylan commented on GitHub (Apr 11, 2025):

I think I got this.
The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of crowdsec-bouncer-traefik-plugin. Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along).

Update the version of the plugin to currently the newest v1.4.2 in the file: config/traefik/traefik_config.yml

experimental:
  plugins:
    crowdsec:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2

After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: config/traefik/dynamic_config.yml

http:
  middlewares:
    crowdsec:
      plugin:
        crowdsec:
          crowdsecAppsecBodyLimit: 10485760

Current issue is, for every file greater than 10MB (default value or set above) there is an error message: level=error msg="Failed to process request body", which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server.

And then there is this note:
/!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x

I'll have the pull request soon.

Edit: Here is the PR https://github.com/fosrl/pangolin/pull/515#issue-2989648287

@TuncTaylan commented on GitHub (Apr 11, 2025): I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of [crowdsec-bouncer-traefik-plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin). Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along). Update the version of the plugin to currently the newest v1.4.2 in the file: [config/traefik/traefik_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/traefik_config.yml) ```shell experimental: plugins: crowdsec: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.2 ``` After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: [config/traefik/dynamic_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/dynamic_config.yml) ```shell http: middlewares: crowdsec: plugin: crowdsec: crowdsecAppsecBodyLimit: 10485760 ``` Current issue is, for every file greater than 10MB (default value or set above) there is an error message: `level=error msg="Failed to process request body"`, which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server. And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x I'll have the pull request soon. Edit: Here is the PR https://github.com/fosrl/pangolin/pull/515#issue-2989648287

GiteaMirror commented

@kylepyke commented on GitHub (Apr 14, 2025):

I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of crowdsec-bouncer-traefik-plugin. Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along).

Update the version of the plugin to currently the newest v1.4.2 in the file: config/traefik/traefik_config.yml

experimental:
plugins:
crowdsec:
moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
version: v1.4.2

After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: config/traefik/dynamic_config.yml

http:
middlewares:
crowdsec:
plugin:
crowdsec:
crowdsecAppsecBodyLimit: 10485760

Current issue is, for every file greater than 10MB (default value or set above) there is an error message: level=error msg="Failed to process request body", which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server.

And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x

I'll have the pull request soon.

Edit: Here is the PR #515 (comment)

Hmmm... This didn't work for me. I'm experiencing the same issues. Pangolin v1.2, Traefik v 3.3.5, CrowdSec v1.4.2

@kylepyke commented on GitHub (Apr 14, 2025): > I think I got this. The configuration we need "crowdsecAppsecBodyLimit" was first implemented in the version v1.4.0 of [crowdsec-bouncer-traefik-plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin). Newest version is v1.4.2, so I had to update the version first, after that I could upload 1GB and even 10GB files without an issue (well read along). > > Update the version of the plugin to currently the newest v1.4.2 in the file: [config/traefik/traefik_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/traefik_config.yml) > > experimental: > plugins: > crowdsec: > moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin > version: v1.4.2 > > After that the default value of the needed parameter "crowdsecAppsecBodyLimit" is already at 10MB, but can be adjusted in the file: [config/traefik/dynamic_config.yml](https://github.com/fosrl/pangolin/blob/main/install/config/traefik/dynamic_config.yml) > > http: > middlewares: > crowdsec: > plugin: > crowdsec: > crowdsecAppsecBodyLimit: 10485760 > > Current issue is, for every file greater than 10MB (default value or set above) there is an error message: `level=error msg="Failed to process request body"`, which I'm guessing that it works as designed, since it doesn't transmit to Crowdsec Appsec Server. > > And then there is this note: /!\ Appsec maximum body limit is defaulted to 10MB By careful when you upgrade to >1.4.x > > I'll have the pull request soon. > > Edit: Here is the PR [#515 (comment)](https://github.com/fosrl/pangolin/pull/515#issue-2989648287) Hmmm... This didn't work for me. I'm experiencing the same issues. Pangolin v1.2, Traefik v 3.3.5, CrowdSec v1.4.2

GiteaMirror commented

@TuncTaylan commented on GitHub (Apr 14, 2025):

Can you share your configs (both of traefik configs)?

@TuncTaylan commented on GitHub (Apr 14, 2025): Can you share your configs (both of traefik configs)?

GiteaMirror commented

@kylepyke commented on GitHub (Apr 14, 2025):

Docker Compose:

name: pangolin
networks:
  default:
    driver: bridge
    name: pangolin
services:
  crowdsec:
    command: -t
    container_name: crowdsec
    environment:
      ACQUIRE_FILES: /var/log/traefik/*.log
      COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
      ENROLL_INSTANCE_NAME: pangolin-crowdsec
      ENROLL_TAGS: docker
      GID: "1000"
      PARSERS: crowdsecurity/whitelists
    expose:
      - 6060
    healthcheck:
      test:
        - CMD
        - cscli
        - capi
        - status
    image: crowdsecurity/crowdsec:latest
    labels:
      - traefik.enable=false
    ports:
      - 6060:6060
    restart: unless-stopped
    volumes:
      - ./config/crowdsec:/etc/crowdsec
      - ./config/crowdsec/db:/var/lib/crowdsec/data
      - ./config/crowdsec_logs/auth.log:/var/log/auth.log:ro
      - ./config/crowdsec_logs/syslog:/var/log/syslog:ro
      - ./config/crowdsec_logs:/var/log
      - ./config/traefik/logs:/var/log/traefik
  gerbil:
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    command:
      - --reachableAt=http://gerbil:3003
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config
      - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth
    container_name: gerbil
    depends_on:
      pangolin:
        condition: service_healthy
    image: fosrl/gerbil:1.0.0
    ports:
      - 51820:51820/udp
      - 443:443
      - 80:80
    restart: unless-stopped
    volumes:
      - ./config/:/var/config
pangolin:
    container_name: pangolin
    healthcheck:
      interval: 3s
      retries: 5
      test:
        - CMD
        - curl
        - -f
        - http://localhost:3001/api/v1/
      timeout: 3s
    image: fosrl/pangolin:1.2.0
    restart: unless-stopped
    volumes:
      - ./config:/app/config
  traefik:
    command:
      - --configFile=/etc/traefik/traefik_config.yml
    container_name: traefik
    depends_on:
      pangolin:
        condition: service_healthy
    image: traefik:v3.3.5
    network_mode: service:gerbil
    restart: unless-stopped
    volumes:
      - ./config/traefik:/etc/traefik:ro
      - ./config/letsencrypt:/letsencrypt
      - ./config/traefik/logs:/var/log/traefik
#    deploy:
#      resources:
#        limits:
#          memory: 1.5G
    environment:
      CLOUDFLARE_DNS_API_TOKEN: "TOKEN"

dynamic_config.yml:

http:
  middlewares:
    limit:
      buffering:
        maxRequestBodyBytes: 3000000000
        memRequestBodyBytes: 2000000000
        #maxResponseBodyBytes: 3000000000
        #memResponseBodyBytes: 2000000000
    pangolin-geoblock:
      plugin:
        geoblock:
          silentStartUp: false
          allowLocalRequests: true
          logLocalRequests: false
          logAllowedRequests: false
          logApiRequests: false
          api: "https://get.geojs.io/v1/ip/country/{ip}"
          apiTimeoutMs: 750                                 # optional
          cacheSize: 15
          forceMonthlyUpdate: false
          allowUnknownCountries: false
          unknownCountryApiResponse: "nil"
          blackListMode: false
          addCountryHeader: false
          countries:
            - US
    crowdsec:
      plugin:
        crowdsec:
          clientTrustedIPs:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 100.89.137.0/20
          crowdsecAppsecEnabled: true
          crowdsecAppsecFailureBlock: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecUnreachableBlock: true
          crowdsecAppsecBodyLimit: 5000000000
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiKey: /W2z5/YqmzHsSBydWM0xhC2FLkH+R5GZsf0dXKfdTpQ
          crowdsecLapiScheme: http
          crowdsecMode: live
          defaultDecisionSeconds: 15
          enabled: true
          #forwardedHeadersTrustedIPs:
          #  - 0.0.0.0/0
          httpTimeoutSeconds: 10
          logLevel: INFO
          updateIntervalSeconds: 15
          updateMaxFailure: 0
    default-whitelist:
      ipWhiteList:
        sourceRange:
          - 10.0.0.0/8
          - 192.168.0.0/16
          - 172.16.0.0/12
    redirect-to-https:
      redirectScheme:
        scheme: https
    security-headers:
      headers:
        contentTypeNosniff: true
        customFrameOptionsValue: SAMEORIGIN
        customResponseHeaders:
          Server: ""
          X-Forwarded-Proto: https
          X-Powered-By: ""
        forceSTSHeader: true
        hostsProxyHeaders:
          - X-Forwarded-Host
        referrerPolicy: strict-origin-when-cross-origin
        sslProxyHeaders:
          X-Forwarded-Proto: https
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 63072000
  routers:
    api-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`) && PathPrefix(`/api/v1`)
      service: api-service
      tls:
        certResolver: letsencrypt
    main-app-router-redirect:
      entryPoints:
        - web
      middlewares:
        - redirect-to-https
      rule: Host(`pangolin.mydomain.com`)
      service: next-service
    next-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`) && !PathPrefix(`/api/v1`)
      service: next-service
      tls:
        certResolver: letsencrypt
        domains:
          - main: "mydomain.com"
            sans:
              - "*.mydomain.com"
    ws-router:
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      rule: Host(`pangolin.mydomain.com`)
      service: api-service
      tls:
        certResolver: letsencrypt
  services:
    api-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3000
    next-service:
      loadBalancer:
        servers:
          - url: http://pangolin:3002

traefik_config.yml:

accessLog:
  bufferingSize: 100
  fields:
    defaultMode: drop
    headers:
      defaultMode: drop
      names:
        Authorization: redact
        Content-Type: keep
        Cookie: redact
        User-Agent: keep
        X-Forwarded-For: keep
        X-Forwarded-Proto: keep
        X-Real-Ip: keep
    names:
      ClientAddr: keep
      ClientHost: keep
      DownstreamContentSize: keep
      DownstreamStatus: keep
      Duration: keep
      RequestMethod: keep
      RequestPath: keep
      RequestProtocol: keep
      RetryAttempts: keep
      ServiceName: keep
      StartUTC: keep
      TLSCipher: keep
      TLSVersion: keep
  filePath: /var/log/traefik/access.log
  filters:
    minDuration: 100ms
    retryAttempts: true
    statusCodes:
      - 200-299
      - 400-499
      - 500-599
  format: json
api:
  dashboard: true
  insecure: true
certificatesResolvers:
  letsencrypt:
    acme:
      dnsChallenge:
        provider: cloudflare
      caServer: https://acme-v02.api.letsencrypt.org/directory
      email: mydomain@domain.com
      httpChallenge:
        entryPoint: web
      storage: /letsencrypt/acme.json
entryPoints:
  web:
    address: ':80'
  websecure:
    address: ':443'
    http:
      middlewares:
        - pangolin-geoblock@file
        - crowdsec@file
      tls:
        certResolver: letsencrypt
    transport:
      respondingTimeouts:
        readTimeout: 30m
experimental:
  plugins:
    badger:
      moduleName: github.com/fosrl/badger
      version: v1.1.0
    crowdsec:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2
    geoblock:
      moduleName: github.com/PascalMinder/geoblock
      version: v0.3.2
log:
  format: json
  level: INFO
providers:
  file:
    filename: /etc/traefik/dynamic_config.yml
  http:
    endpoint: http://pangolin:3001/api/v1/traefik-config
    pollInterval: 5s
serversTransport:
  insecureSkipVerify: true

@kylepyke commented on GitHub (Apr 14, 2025): Docker Compose: ``` name: pangolin networks: default: driver: bridge name: pangolin services: crowdsec: command: -t container_name: crowdsec environment: ACQUIRE_FILES: /var/log/traefik/*.log COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules ENROLL_INSTANCE_NAME: pangolin-crowdsec ENROLL_TAGS: docker GID: "1000" PARSERS: crowdsecurity/whitelists expose: - 6060 healthcheck: test: - CMD - cscli - capi - status image: crowdsecurity/crowdsec:latest labels: - traefik.enable=false ports: - 6060:6060 restart: unless-stopped volumes: - ./config/crowdsec:/etc/crowdsec - ./config/crowdsec/db:/var/lib/crowdsec/data - ./config/crowdsec_logs/auth.log:/var/log/auth.log:ro - ./config/crowdsec_logs/syslog:/var/log/syslog:ro - ./config/crowdsec_logs:/var/log - ./config/traefik/logs:/var/log/traefik gerbil: cap_add: - NET_ADMIN - SYS_MODULE command: - --reachableAt=http://gerbil:3003 - --generateAndSaveKeyTo=/var/config/key - --remoteConfig=http://pangolin:3001/api/v1/gerbil/get-config - --reportBandwidthTo=http://pangolin:3001/api/v1/gerbil/receive-bandwidth container_name: gerbil depends_on: pangolin: condition: service_healthy image: fosrl/gerbil:1.0.0 ports: - 51820:51820/udp - 443:443 - 80:80 restart: unless-stopped volumes: - ./config/:/var/config pangolin: container_name: pangolin healthcheck: interval: 3s retries: 5 test: - CMD - curl - -f - http://localhost:3001/api/v1/ timeout: 3s image: fosrl/pangolin:1.2.0 restart: unless-stopped volumes: - ./config:/app/config traefik: command: - --configFile=/etc/traefik/traefik_config.yml container_name: traefik depends_on: pangolin: condition: service_healthy image: traefik:v3.3.5 network_mode: service:gerbil restart: unless-stopped volumes: - ./config/traefik:/etc/traefik:ro - ./config/letsencrypt:/letsencrypt - ./config/traefik/logs:/var/log/traefik # deploy: # resources: # limits: # memory: 1.5G environment: CLOUDFLARE_DNS_API_TOKEN: "TOKEN" ``` dynamic_config.yml: ``` http: middlewares: limit: buffering: maxRequestBodyBytes: 3000000000 memRequestBodyBytes: 2000000000 #maxResponseBodyBytes: 3000000000 #memResponseBodyBytes: 2000000000 pangolin-geoblock: plugin: geoblock: silentStartUp: false allowLocalRequests: true logLocalRequests: false logAllowedRequests: false logApiRequests: false api: "https://get.geojs.io/v1/ip/country/{ip}" apiTimeoutMs: 750 # optional cacheSize: 15 forceMonthlyUpdate: false allowUnknownCountries: false unknownCountryApiResponse: "nil" blackListMode: false addCountryHeader: false countries: - US crowdsec: plugin: crowdsec: clientTrustedIPs: - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 - 100.89.137.0/20 crowdsecAppsecEnabled: true crowdsecAppsecFailureBlock: true crowdsecAppsecHost: crowdsec:7422 crowdsecAppsecUnreachableBlock: true crowdsecAppsecBodyLimit: 5000000000 crowdsecLapiHost: crowdsec:8080 crowdsecLapiKey: /W2z5/YqmzHsSBydWM0xhC2FLkH+R5GZsf0dXKfdTpQ crowdsecLapiScheme: http crowdsecMode: live defaultDecisionSeconds: 15 enabled: true #forwardedHeadersTrustedIPs: # - 0.0.0.0/0 httpTimeoutSeconds: 10 logLevel: INFO updateIntervalSeconds: 15 updateMaxFailure: 0 default-whitelist: ipWhiteList: sourceRange: - 10.0.0.0/8 - 192.168.0.0/16 - 172.16.0.0/12 redirect-to-https: redirectScheme: scheme: https security-headers: headers: contentTypeNosniff: true customFrameOptionsValue: SAMEORIGIN customResponseHeaders: Server: "" X-Forwarded-Proto: https X-Powered-By: "" forceSTSHeader: true hostsProxyHeaders: - X-Forwarded-Host referrerPolicy: strict-origin-when-cross-origin sslProxyHeaders: X-Forwarded-Proto: https stsIncludeSubdomains: true stsPreload: true stsSeconds: 63072000 routers: api-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) && PathPrefix(`/api/v1`) service: api-service tls: certResolver: letsencrypt main-app-router-redirect: entryPoints: - web middlewares: - redirect-to-https rule: Host(`pangolin.mydomain.com`) service: next-service next-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) && !PathPrefix(`/api/v1`) service: next-service tls: certResolver: letsencrypt domains: - main: "mydomain.com" sans: - "*.mydomain.com" ws-router: entryPoints: - websecure middlewares: - security-headers rule: Host(`pangolin.mydomain.com`) service: api-service tls: certResolver: letsencrypt services: api-service: loadBalancer: servers: - url: http://pangolin:3000 next-service: loadBalancer: servers: - url: http://pangolin:3002 ``` traefik_config.yml: ``` accessLog: bufferingSize: 100 fields: defaultMode: drop headers: defaultMode: drop names: Authorization: redact Content-Type: keep Cookie: redact User-Agent: keep X-Forwarded-For: keep X-Forwarded-Proto: keep X-Real-Ip: keep names: ClientAddr: keep ClientHost: keep DownstreamContentSize: keep DownstreamStatus: keep Duration: keep RequestMethod: keep RequestPath: keep RequestProtocol: keep RetryAttempts: keep ServiceName: keep StartUTC: keep TLSCipher: keep TLSVersion: keep filePath: /var/log/traefik/access.log filters: minDuration: 100ms retryAttempts: true statusCodes: - 200-299 - 400-499 - 500-599 format: json api: dashboard: true insecure: true certificatesResolvers: letsencrypt: acme: dnsChallenge: provider: cloudflare caServer: https://acme-v02.api.letsencrypt.org/directory email: mydomain@domain.com httpChallenge: entryPoint: web storage: /letsencrypt/acme.json entryPoints: web: address: ':80' websecure: address: ':443' http: middlewares: - pangolin-geoblock@file - crowdsec@file tls: certResolver: letsencrypt transport: respondingTimeouts: readTimeout: 30m experimental: plugins: badger: moduleName: github.com/fosrl/badger version: v1.1.0 crowdsec: moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin version: v1.4.2 geoblock: moduleName: github.com/PascalMinder/geoblock version: v0.3.2 log: format: json level: INFO providers: file: filename: /etc/traefik/dynamic_config.yml http: endpoint: http://pangolin:3001/api/v1/traefik-config pollInterval: 5s serversTransport: insecureSkipVerify: true ```

GiteaMirror commented

@kylepyke commented on GitHub (Apr 14, 2025):

Immich running on separate home server, docker-compose:

#
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    devices:
      - /dev/dri:/dev/dri
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # Note the `-cuda` at the end
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    # Note the lack of an `extends` section
    device_cgroup_rules:
      - 'c 189:* rmw'
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /dev/bus/usb:/dev/bus/usb

      - model-cache:/cache
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:XXXX
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:XXXX
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:

@kylepyke commented on GitHub (Apr 14, 2025): Immich running on separate home server, docker-compose: ``` # # WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose # # Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} devices: - /dev/dri:/dev/dri volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro ports: - '2283:2283' depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # Note the `-cuda` at the end image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino # Note the lack of an `extends` section device_cgroup_rules: - 'c 189:* rmw' devices: - /dev/dri:/dev/dri volumes: - /dev/bus/usb:/dev/bus/usb - model-cache:/cache restart: always redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:XXXX healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:XXXX environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data healthcheck: test: >- pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1; Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m start_interval: 30s start_period: 5m command: >- postgres -c shared_preload_libraries=vectors.so -c 'search_path="$$user", public, vectors' -c logging_collector=on -c max_wal_size=2GB -c shared_buffers=512MB -c wal_compression=on restart: always volumes: model-cache: ```

GiteaMirror commented

@TuncTaylan commented on GitHub (Apr 15, 2025):

Did you also have the same problem with the default value of crowdsecAppsecBodyLimit with 10MB?
You've set it to up to 5000000000 (approx. 4.5GB).

Just for this I've setup a resource to my immich at home with the default setting of crowdsecAppsecBodyLimit 10485760.

I could upload a 2GB test video without any issues.

At first I tried to set crowdsecAppsecBodyLimit value impulsively to the "highest" file size I would upload, that failed also.

So try the default setting (10MB), it works.

@TuncTaylan commented on GitHub (Apr 15, 2025): Did you also have the same problem with the default value of crowdsecAppsecBodyLimit with 10MB? You've set it to up to 5000000000 (approx. 4.5GB). Just for this I've setup a resource to my immich at home with the default setting of crowdsecAppsecBodyLimit 10485760. I could upload a 2GB test video without any issues. ![Image](https://github.com/user-attachments/assets/4b9a6eae-fc5b-440c-a22a-5d98a118a930) ![Image](https://github.com/user-attachments/assets/eb2680ed-4407-4aaa-b194-766b1af058b9) At first I tried to set crowdsecAppsecBodyLimit value impulsively to the "highest" file size I would upload, that failed also. So try the default setting (10MB), it works.

GiteaMirror commented

@kylepyke commented on GitHub (Apr 15, 2025):

Oh wow, this worked! Thanks! @TuncTaylan @oschwartz10612 @miloschwartz

@kylepyke commented on GitHub (Apr 15, 2025): Oh wow, this worked! Thanks! @TuncTaylan @oschwartz10612 @miloschwartz

GiteaMirror commented