Pangolin laggy with 0.12.0/0.12.1 (long page load times) #756

Open
opened 2025-11-13 12:10:08 -06:00 by GiteaMirror · 12 comments
Owner

Originally created by @v1rusnl on GitHub (Nov 3, 2025).

Describe the Bug

After upgrading to 1.12.0, I rolled back to 1.11.1 today on both my LAN Pangolin instance + my VPS instance with Gerbil/Newt connected to my Homelab. I'm using Traefik 3.5.4 on both. Tried also the new 1.12.1 update, but it's the same. Migration throws no error in logs.

Something is really off with 1.12.0 (at least both of my instances). The loading times of webpages (even in my LAN locally) multiplied by min. 5 times. After rollback, everything is blazing fast again.

Comparison with the exact same Service (Emby) using Pangolin/Traefik locally as my LAN Reverse Proxy without Gerbil/Newt over Gigabit:

1.12.0/1.12.1
Screenshot 2025-11-02 185457
Screenshot 2025-11-02 185506

1.11.1
Screenshot 2025-11-02 185847
Screenshot 2025-11-02 185853

Some of the same assets load up to 10 times slower than with 1.11.1. I've sorted the elements by loading time. In 1.11.1, the slowest element (the backdrop) is 300ms, in 1.12.0/1.12.1 it's 4 seconds. Man, you feel that when opening a webpage... and it's reproduceable. Tried to migrate 3 times from my 1.11.1 backup to 1.12.0 and then to 1.12.1, but it's the same everytime. 1.12.0/1.12.1=slow, back to 1.11.1=fast.

Environment

  • OS Type & Version: LAN Instance: Debian 12 / VPS Instance: Ubuntu 24.04
  • Pangolin Version: 0.12.0/0.12.1 (tried both)
  • Gerbil Version: 1.2.2
  • Badger Version: 1.2.1
  • Traefik Version: 3.5.4
  • Newt Version: 1.6.0
  • Olm Version: not used

LAN Instance just uses Pangolin+Traefik/Badger
VPS instance uses Pangolin+Gerbil+Traefik/Badger connected to 3 Newt in my Home LAN

To Reproduce

  1. Open served ressource with 1.11.1 -> fast load times
  2. Update to 1.12.0/1.12.1 -> long load time/laggy behaviour of ressource
  3. Downgrade to 1.11.1 -> fast load times again

Expected Behavior

Update to 1.12.0/1.12.1 -> page load times are as fast as in 1.11.1

Originally created by @v1rusnl on GitHub (Nov 3, 2025). ### Describe the Bug After upgrading to 1.12.0, I rolled back to 1.11.1 today on both my LAN Pangolin instance + my VPS instance with Gerbil/Newt connected to my Homelab. I'm using Traefik 3.5.4 on both. Tried also the new 1.12.1 update, but it's the same. Migration throws no error in logs. Something is really off with 1.12.0 (at least both of my instances). The loading times of webpages (even in my LAN locally) multiplied by min. 5 times. After rollback, everything is blazing fast again. Comparison with the exact same Service (Emby) using Pangolin/Traefik locally as my LAN Reverse Proxy without Gerbil/Newt over Gigabit: 1.12.0/1.12.1 <img width="1906" height="371" alt="Screenshot 2025-11-02 185457" src="https://github.com/user-attachments/assets/80565b4e-b0ca-4cf4-a5b3-eb5bd97cc20b" /> <img width="1581" height="227" alt="Screenshot 2025-11-02 185506" src="https://github.com/user-attachments/assets/23b9db38-6ecb-48e4-a546-94071530c5fa" /> 1.11.1 <img width="1906" height="345" alt="Screenshot 2025-11-02 185847" src="https://github.com/user-attachments/assets/caacde6c-6b5a-4acf-9966-3b3b7b698005" /> <img width="1563" height="202" alt="Screenshot 2025-11-02 185853" src="https://github.com/user-attachments/assets/96bbc399-ec63-4f18-92d0-ecc526a524fc" /> Some of the same assets load up to 10 times slower than with 1.11.1. I've sorted the elements by loading time. In 1.11.1, the slowest element (the backdrop) is 300ms, in 1.12.0/1.12.1 it's 4 seconds. Man, you feel that when opening a webpage... and it's reproduceable. Tried to migrate 3 times from my 1.11.1 backup to 1.12.0 and then to 1.12.1, but it's the same everytime. 1.12.0/1.12.1=slow, back to 1.11.1=fast. ### Environment - OS Type & Version: LAN Instance: Debian 12 / VPS Instance: Ubuntu 24.04 - Pangolin Version: 0.12.0/0.12.1 (tried both) - Gerbil Version: 1.2.2 - Badger Version: 1.2.1 - Traefik Version: 3.5.4 - Newt Version: 1.6.0 - Olm Version: not used LAN Instance just uses Pangolin+Traefik/Badger VPS instance uses Pangolin+Gerbil+Traefik/Badger connected to 3 Newt in my Home LAN ### To Reproduce 1. Open served ressource with 1.11.1 -> fast load times 2. Update to 1.12.0/1.12.1 -> long load time/laggy behaviour of ressource 3. Downgrade to 1.11.1 -> fast load times again ### Expected Behavior Update to 1.12.0/1.12.1 -> page load times are as fast as in 1.11.1
Author
Owner

@Re4mstr commented on GitHub (Nov 3, 2025):

I am also experiencing this, but I would say almost 10x slowdown.

@Re4mstr commented on GitHub (Nov 3, 2025): I am also experiencing this, but I would say almost 10x slowdown.
Author
Owner

@oschwartz10612 commented on GitHub (Nov 3, 2025):

Hi I am sorry you are experiencing this! @v1rusnl do you see anything in the pangolin logs when you load the resource?

It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up?

@oschwartz10612 commented on GitHub (Nov 3, 2025): Hi I am sorry you are experiencing this! @v1rusnl do you see anything in the pangolin logs when you load the resource? It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up?
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

Hi I am sorry you are experiencing this! @v1rusnl do you see anything in the pangolin logs when you load the resource?

It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up?

Owen, that did the trick. Everything back to normal...
Are there any disadvantages in disabling it?

Image
@v1rusnl commented on GitHub (Nov 3, 2025): > Hi I am sorry you are experiencing this! [@v1rusnl](https://github.com/v1rusnl) do you see anything in the pangolin logs when you load the resource? > > It could be related to the new logging we added maybe. A log message is inserted into the database on each request. On 1.12.1 if you go into the organization settings (Settings tab at the bottom of the sidebar) and disable the request logs does it speed up at all? If not and you keep it disabled and do a restart of Pangolin does it speed up? Owen, that did the trick. Everything back to normal... Are there any disadvantages in disabling it? <img width="1910" height="597" alt="Image" src="https://github.com/user-attachments/assets/4242e4e1-f220-47ac-9cb0-b44c7497e64c" />
Author
Owner

@oschwartz10612 commented on GitHub (Nov 3, 2025):

Hum that's concerning but good we narrowed it down.

Are you using a sqlite or postgres database?

Do you see anything in the logs when it is enabled?

Do you serve a lot of traffic on this resource?

@oschwartz10612 commented on GitHub (Nov 3, 2025): Hum that's concerning but good we narrowed it down. Are you using a sqlite or postgres database? Do you see anything in the logs when it is enabled? Do you serve a lot of traffic on this resource?
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

sqlite

I have to say, in my Home LAN I run Pangolin on a microserver with a system-HDD, no SSD. Could the HDD be the culprit because it cannot keep up with the log writing cycles parallel to the ressource calls? I disabled also access logs on my home LAN from day one on, since I don't need them. On the VPS, I don't know what hardware is behind it. But there I use Traefik Log dashboard from HHF nevertheless.

When it' enabled, logs are shown

Image

Edit: I can live with having no logs in my LAN, enabling logs -> slow; diabling logs -> fast. Perhaps I try to move Pangolin to another machine with a SSD the next days. Also I try to upgrade again on the VPS and check that one out.

@v1rusnl commented on GitHub (Nov 3, 2025): sqlite I have to say, in my Home LAN I run Pangolin on a microserver with a system-HDD, no SSD. Could the HDD be the culprit because it cannot keep up with the log writing cycles parallel to the ressource calls? I disabled also access logs on my home LAN from day one on, since I don't need them. On the VPS, I don't know what hardware is behind it. But there I use Traefik Log dashboard from HHF nevertheless. When it' enabled, logs are shown <img width="549" height="576" alt="Image" src="https://github.com/user-attachments/assets/86fa3a6f-f539-4a99-b76b-0e3e4c28291c" /> Edit: I can live with having no logs in my LAN, enabling logs -> slow; diabling logs -> fast. Perhaps I try to move Pangolin to another machine with a SSD the next days. Also I try to upgrade again on the VPS and check that one out.
Author
Owner

@oschwartz10612 commented on GitHub (Nov 3, 2025):

Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster.

Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working!

@oschwartz10612 commented on GitHub (Nov 3, 2025): Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster. Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working!
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

Ok, went on the VPS again, rechecked traefik_config.yml, dynamic_config.yml and config.yml, aligned them all to look exactly like the actual repo ones, then updated from 1.11.1 to 1.12.0 to 1.12.1 and set badger to 1.2.1, and everythings working now with fast loading times. I guess Hetzner is using SSD storage on their VPS plans.

So it seems Pangolin logs should not be used on HDDs for now 😆

@v1rusnl commented on GitHub (Nov 3, 2025): Ok, went on the VPS again, rechecked traefik_config.yml, dynamic_config.yml and config.yml, aligned them all to look exactly like the actual repo ones, then updated from 1.11.1 to 1.12.0 to 1.12.1 and set badger to 1.2.1, and everythings working now with fast loading times. I guess Hetzner is using SSD storage on their VPS plans. So it seems Pangolin logs should not be used on HDDs for now :laughing:
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

With the new logging, it would be overkill to also use Traefik log viewer on the VPS, right?

@v1rusnl commented on GitHub (Nov 3, 2025): With the new logging, it would be overkill to also use Traefik log viewer on the VPS, right?
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster.

Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working!

Spun up a Pangolin on my SSD machine with a copy of the HDD pangolin folder and it's blazing fast even with logging.

So it's definitely a problem with the logging with parallel resource serving when using a HDD. It doesnt matter if the resource is traffic heavy like Emby with a lot of images or if it's just Portainer with small data chunks

@v1rusnl commented on GitHub (Nov 3, 2025): > Thanks for the info! It could in fact be that - I would be interested to know if using a SSD would make it faster. > > Also, the log writing for each request is supposed to be asynchronous and not hold up the actual request processing. I will look into why that is not working! Spun up a Pangolin on my SSD machine with a copy of the HDD pangolin folder and it's blazing fast even with logging. So it's definitely a problem with the logging with parallel resource serving when using a HDD. It doesnt matter if the resource is traffic heavy like Emby with a lot of images or if it's just Portainer with small data chunks
Author
Owner

@oschwartz10612 commented on GitHub (Nov 3, 2025):

Thanks for all of the info. Will look into making it async for the next release!

@oschwartz10612 commented on GitHub (Nov 3, 2025): Thanks for all of the info. Will look into making it async for the next release!
Author
Owner

@v1rusnl commented on GitHub (Nov 3, 2025):

Thank you for this awesome peace of Software!

@v1rusnl commented on GitHub (Nov 3, 2025): Thank you for this awesome peace of Software!
Author
Owner

@roadkingvrod commented on GitHub (Nov 5, 2025):

Just to add further data to this, I experienced the exact same issue. Glad it's a somewhat easy fix!

@roadkingvrod commented on GitHub (Nov 5, 2025): Just to add further data to this, I experienced the exact same issue. Glad it's a somewhat easy fix!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#756