[GH-ISSUE #2556] 1.16.0 breaks sites #19274

Closed
opened 2026-05-21 19:56:28 -05:00 by GiteaMirror · 9 comments
Owner

Originally created by @thaler-tech on GitHub (Feb 27, 2026).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/2556

Describe the Bug

I saw 1.16.0 was out, so I ran "docker compose pull" and "docker compose up -d --force-recreate" and after the stack was up and running again, all my sites had disappeared (screenshots attached of 1.15.4 vs 1.16.0). I just reverted back to 1.15.4 and everything is back to normal again.

Image Image

Environment

  • OS Type & Version: Ubuntu 24.04 (Docker)
  • Pangolin Version: 1.16.0
  • Gerbil Version: 1.3.0 (latest)
  • Traefik Version: 3.6.9 (latest)
  • Newt Version: 1.9.0

To Reproduce

Set Pangolin Docker image to "docker.io/fosrl/pangolin:ee-postgresql-latest", run "docker compose pull" and "docker compose up -d --force-recreate", after Pangolin is back up, Sites are gone with the latest version.

Expected Behavior

Sites should be populated.

Originally created by @thaler-tech on GitHub (Feb 27, 2026). Original GitHub issue: https://github.com/fosrl/pangolin/issues/2556 ### Describe the Bug I saw 1.16.0 was out, so I ran "docker compose pull" and "docker compose up -d --force-recreate" and after the stack was up and running again, all my sites had disappeared (screenshots attached of 1.15.4 vs 1.16.0). I just reverted back to 1.15.4 and everything is back to normal again. <img width="1280" height="593" alt="Image" src="https://github.com/user-attachments/assets/a7a5f793-cc25-4e91-8dd8-a0047c9d0236" /> <img width="1280" height="591" alt="Image" src="https://github.com/user-attachments/assets/4081f7dc-51d2-418a-ac31-7eb765741d7d" /> ### Environment - OS Type & Version: Ubuntu 24.04 (Docker) - Pangolin Version: 1.16.0 - Gerbil Version: 1.3.0 (latest) - Traefik Version: 3.6.9 (latest) - Newt Version: 1.9.0 ### To Reproduce Set Pangolin Docker image to "docker.io/fosrl/pangolin:ee-postgresql-latest", run "docker compose pull" and "docker compose up -d --force-recreate", after Pangolin is back up, Sites are gone with the latest version. ### Expected Behavior Sites should be populated.
Author
Owner

@oschwartz10612 commented on GitHub (Feb 27, 2026):

Hey do you see any logs that might indicate an issue on the latest version?

<!-- gh-comment-id:3970845169 --> @oschwartz10612 commented on GitHub (Feb 27, 2026): Hey do you see any logs that might indicate an issue on the latest version?
Author
Owner

@oschwartz10612 commented on GitHub (Feb 27, 2026):

We are doing some testing as wel.

<!-- gh-comment-id:3970845817 --> @oschwartz10612 commented on GitHub (Feb 27, 2026): We are doing some testing as wel.
Author
Owner

@thaler-tech commented on GitHub (Feb 27, 2026):

I will set it back to latest and pull logs for you. As far as I can tell, no functionality actually broke, the sites just disappeared, but public resources were still available. Private resources also disappeared which I forgot to mention.

I've just gone through and updated all my servers to the latest newt connector as well, so let me see if that made any difference, which I doubt since one of the servers was already on that version and it also wasn't visible.

Hold tight, I'll report back!

<!-- gh-comment-id:3970862422 --> @thaler-tech commented on GitHub (Feb 27, 2026): I will set it back to latest and pull logs for you. As far as I can tell, no functionality actually broke, the sites just disappeared, but public resources were still available. Private resources also disappeared which I forgot to mention. I've just gone through and updated all my servers to the latest newt connector as well, so let me see if that made any difference, which I doubt since one of the servers was already on that version and it also wasn't visible. Hold tight, I'll report back!
Author
Owner

@thaler-tech commented on GitHub (Feb 27, 2026):

Alright, I hope I didn't inadvertently leave anything sensitive in there, but here are the logs from "docker compose logs -f":

gerbil    | INFO: 2026/02/27 05:25:09 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:25:22 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:25:23 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:25:28 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:25:36 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:26:03 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:26:09 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:26:22 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:26:23 Cleared 0 sessions for WG IP: REDACTED
gerbil    | INFO: 2026/02/27 05:26:28 Cleared 0 sessions for WG IP: REDACTED
postgres  | 2026-02-27 05:26:36.118 UTC [126] ERROR:  subquery in FROM must have an alias at character 31
postgres  | 2026-02-27 05:26:36.118 UTC [126] HINT:  For example, FROM (SELECT ...) [AS] foo.
postgres  | 2026-02-27 05:26:36.118 UTC [126] STATEMENT:  select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8));
pangolin  | 2026-02-27T05:26:36+00:00 [error]: Failed query: select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8));
pangolin  | params: 5,6,7,8,9,10,11,REDACTED
pangolin  | Stack: Error: Failed query: select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8));
pangolin  | params: 5,6,7,8,9,10,11,REDACTED
pangolin  |     at NodePgPreparedQuery.queryWithCache (file:///app/node_modules/drizzle-orm/pg-core/session.js:41:15)
pangolin  |     at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
pangolin  |     at async NodePgSession.count (file:///app/node_modules/drizzle-orm/node-postgres/session.js:197:17)
pangolin  |     at async Promise.all (index 0)
pangolin  |     at async listSites (file:///app/dist/server.mjs:29183:32) {"query":"select count(*) as count from (select \"sites\".\"siteId\", \"sites\".\"niceId\", \"sites\".\"name\", \"sites\".\"pubKey\", \"sites\".\"subnet\", \"sites\".\"bytesIn\", \"sites\".\"bytesOut\", \"orgs\".\"name\", \"sites\".\"type\", \"sites\".\"online\", \"sites\".\"address\", \"newt\".\"version\", \"sites\".\"exitNode\", \"exitNodes\".\"name\", \"exitNodes\".\"endpoint\", \"remoteExitNode\".\"id\" from \"sites\" left join \"orgs\" on \"sites\".\"orgId\" = \"orgs\".\"orgId\" left join \"newt\" on \"newt\".\"siteId\" = \"sites\".\"siteId\" left join \"exitNodes\" on \"exitNodes\".\"exitNodeId\" = \"sites\".\"exitNode\" left join \"remoteExitNode\" on \"remoteExitNode\".\"exitNodeId\" = \"sites\".\"exitNode\" where (\"sites\".\"siteId\" in ($1, $2, $3, $4, $5, $6, $7) and \"sites\".\"orgId\" = $8));","params":[5,6,7,8,9,10,11,"REDACTED"],"cause":{"length":138,"name":"error","severity":"ERROR","code":"42601","hint":"For example, FROM (SELECT ...) [AS] foo.","position":"31","file":"gram.y","line":"13263","routine":"base_yyparse"}}```
<!-- gh-comment-id:3970890269 --> @thaler-tech commented on GitHub (Feb 27, 2026): Alright, I hope I didn't inadvertently leave anything sensitive in there, but here are the logs from "docker compose logs -f": ```gerbil | INFO: 2026/02/27 05:25:03 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:25:09 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:25:22 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:25:23 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:25:28 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:25:36 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:26:03 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:26:09 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:26:22 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:26:23 Cleared 0 sessions for WG IP: REDACTED gerbil | INFO: 2026/02/27 05:26:28 Cleared 0 sessions for WG IP: REDACTED postgres | 2026-02-27 05:26:36.118 UTC [126] ERROR: subquery in FROM must have an alias at character 31 postgres | 2026-02-27 05:26:36.118 UTC [126] HINT: For example, FROM (SELECT ...) [AS] foo. postgres | 2026-02-27 05:26:36.118 UTC [126] STATEMENT: select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8)); pangolin | 2026-02-27T05:26:36+00:00 [error]: Failed query: select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8)); pangolin | params: 5,6,7,8,9,10,11,REDACTED pangolin | Stack: Error: Failed query: select count(*) as count from (select "sites"."siteId", "sites"."niceId", "sites"."name", "sites"."pubKey", "sites"."subnet", "sites"."bytesIn", "sites"."bytesOut", "orgs"."name", "sites"."type", "sites"."online", "sites"."address", "newt"."version", "sites"."exitNode", "exitNodes"."name", "exitNodes"."endpoint", "remoteExitNode"."id" from "sites" left join "orgs" on "sites"."orgId" = "orgs"."orgId" left join "newt" on "newt"."siteId" = "sites"."siteId" left join "exitNodes" on "exitNodes"."exitNodeId" = "sites"."exitNode" left join "remoteExitNode" on "remoteExitNode"."exitNodeId" = "sites"."exitNode" where ("sites"."siteId" in ($1, $2, $3, $4, $5, $6, $7) and "sites"."orgId" = $8)); pangolin | params: 5,6,7,8,9,10,11,REDACTED pangolin | at NodePgPreparedQuery.queryWithCache (file:///app/node_modules/drizzle-orm/pg-core/session.js:41:15) pangolin | at process.processTicksAndRejections (node:internal/process/task_queues:104:5) pangolin | at async NodePgSession.count (file:///app/node_modules/drizzle-orm/node-postgres/session.js:197:17) pangolin | at async Promise.all (index 0) pangolin | at async listSites (file:///app/dist/server.mjs:29183:32) {"query":"select count(*) as count from (select \"sites\".\"siteId\", \"sites\".\"niceId\", \"sites\".\"name\", \"sites\".\"pubKey\", \"sites\".\"subnet\", \"sites\".\"bytesIn\", \"sites\".\"bytesOut\", \"orgs\".\"name\", \"sites\".\"type\", \"sites\".\"online\", \"sites\".\"address\", \"newt\".\"version\", \"sites\".\"exitNode\", \"exitNodes\".\"name\", \"exitNodes\".\"endpoint\", \"remoteExitNode\".\"id\" from \"sites\" left join \"orgs\" on \"sites\".\"orgId\" = \"orgs\".\"orgId\" left join \"newt\" on \"newt\".\"siteId\" = \"sites\".\"siteId\" left join \"exitNodes\" on \"exitNodes\".\"exitNodeId\" = \"sites\".\"exitNode\" left join \"remoteExitNode\" on \"remoteExitNode\".\"exitNodeId\" = \"sites\".\"exitNode\" where (\"sites\".\"siteId\" in ($1, $2, $3, $4, $5, $6, $7) and \"sites\".\"orgId\" = $8));","params":[5,6,7,8,9,10,11,"REDACTED"],"cause":{"length":138,"name":"error","severity":"ERROR","code":"42601","hint":"For example, FROM (SELECT ...) [AS] foo.","position":"31","file":"gram.y","line":"13263","routine":"base_yyparse"}}```
Author
Owner

@oschwartz10612 commented on GitHub (Feb 27, 2026):

This is helpful thank you looking into it

<!-- gh-comment-id:3970902024 --> @oschwartz10612 commented on GitHub (Feb 27, 2026): This is helpful thank you looking into it
Author
Owner

@oschwartz10612 commented on GitHub (Feb 27, 2026):

Rebuilding with the missing aliases fixed. Thanks for letting us know

<!-- gh-comment-id:3970920358 --> @oschwartz10612 commented on GitHub (Feb 27, 2026): Rebuilding with the missing aliases fixed. Thanks for letting us know
Author
Owner

@thaler-tech commented on GitHub (Feb 27, 2026):

Of course! Thank you guys for the amazing work! I've been loving Pangolin!

<!-- gh-comment-id:3970926694 --> @thaler-tech commented on GitHub (Feb 27, 2026): Of course! Thank you guys for the amazing work! I've been loving Pangolin!
Author
Owner

@oschwartz10612 commented on GitHub (Feb 27, 2026):

Container should be updated if you want to give it another shot! :}

<!-- gh-comment-id:3970957637 --> @oschwartz10612 commented on GitHub (Feb 27, 2026): Container should be updated if you want to give it another shot! :}
Author
Owner

@thaler-tech commented on GitHub (Feb 27, 2026):

Yep, I just did and it's good to go now. Only difference I am seeing is my public resource list order has changed and it doesn't let me sort it by name, but honestly I don't care all that much.

I will have to play with the new SSH auth-daemon support though, sounds like that could be super useful.

Thanks again!

<!-- gh-comment-id:3971006592 --> @thaler-tech commented on GitHub (Feb 27, 2026): Yep, I just did and it's good to go now. Only difference I am seeing is my public resource list order has changed and it doesn't let me sort it by name, but honestly I don't care all that much. I will have to play with the new SSH auth-daemon support though, sounds like that could be super useful. Thanks again!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/pangolin#19274