Making this an issue for increased visability, based on the discussion #970
This issue is for tracking the progress of adding High Availability to pangolin, so if one server goes down, traffic is routed to the available one, which is done with health checks, and my description so far is a bit of a over-simplification of High Availability but currently, as I was told in the discussion, that pangolin will blindly round-robin between hosts which may be down, and therfore is not HA. If this feature is added it will increase the uses for pangolin significantly.
I also hope this is added to the project board.
Originally created by @SpiderUnderUrBed on GitHub (Sep 13, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/1453
Making this an issue for increased visability, based on the discussion #970
This issue is for tracking the progress of adding High Availability to pangolin, so if one server goes down, traffic is routed to the available one, which is done with health checks, and my description so far is a bit of a over-simplification of High Availability but currently, as I was told in the discussion, that pangolin will blindly round-robin between hosts which may be down, and therfore is not HA. If this feature is added it will increase the uses for pangolin significantly.
I also hope this is added to the project board.
HA Proxy with a backend pointing to 2 indentical Pangolin instances (active/backup)
those 2 instances use a common PostgreSQL database on a third instance
(same configuration as https://github.com/orgs/fosrl/discussions/1342, with 2 Pangolin and external Postgresql DB)
I start first instance, do initial configuration (admin account, an org and a site)
I then create and start second instance . It has trouble starting gerbil
gerbil | FATAL: 2025/09/17 06:50:16 Failed to assign IP address: failed to parse IP address: invalid CIDR address:
gerbil | INFO: 2025/09/17 06:50:17 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
when recreating instances from scratch, not dropping the DB, same issue :
gerbil | INFO: 2025/09/17 07:11:53 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
gerbil | INFO: 2025/09/17 07:11:53 Created WireGuard interface wg0
gerbil | FATAL: 2025/09/17 07:11:53 Failed to assign IP address: failed to parse IP address: invalid CIDR address:
dropping database and recreating first instance is OK, second instance fails the same way as mentioned before.
<!-- gh-comment-id:3301757037 -->
@Esa-mimbias commented on GitHub (Sep 17, 2025):
Hi
I've been playing with this setup:
- HA Proxy with a backend pointing to 2 indentical Pangolin instances (active/backup)
- those 2 instances use a common PostgreSQL database on a third instance
(same configuration as https://github.com/orgs/fosrl/discussions/1342, with 2 Pangolin and external Postgresql DB)
I start first instance, do initial configuration (admin account, an org and a site)
I then create and start second instance . It has trouble starting gerbil
```
gerbil | FATAL: 2025/09/17 06:50:16 Failed to assign IP address: failed to parse IP address: invalid CIDR address:
gerbil | INFO: 2025/09/17 06:50:17 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
```
when recreating instances from scratch, not dropping the DB, same issue :
```
gerbil | INFO: 2025/09/17 07:11:53 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
gerbil | INFO: 2025/09/17 07:11:53 Created WireGuard interface wg0
gerbil | FATAL: 2025/09/17 07:11:53 Failed to assign IP address: failed to parse IP address: invalid CIDR address:
```
dropping database and recreating first instance is OK, second instance fails the same way as mentioned before.
table exitNodes contains one entry:
```
exitNodeId | name | address | endpoint | publicKey | listenPort | reachableAt | maxConnections | online | lastPing | type
------------+--------------------+-----------------+-----------------------------------------+----------------------------------------------+------------+--------------------+----------------+--------+----------+--------
1 | Exit Node K4l7Y6O8 | 100.89.128.1/24 | pangolin.<domain>.<tld> | K4l7Y6O8***********************************= | 51820 | http://gerbil:3003 | | f | | gerbil
```
I removed gerbil from both instances. With only Pangolin (and Traefik) service, both instances start, it works very well:
in an active/backup setup
in an active/active round robin setup. It's impressive to see both instances serve sessions simultaneously - even websocket Apps such as Gitlab WebIDE and Minio bucket browser (that use to fail on "concurrent" products such as F5 APM and Gravitional Teleport)
it's sufficient for my use cases now. May try Olm soon
Been thinking about how to use gerbil in an HA setup, I guess it might be tricky to maintain connexions (UDP...).
Great job !
<!-- gh-comment-id:3306309974 -->
@Esa-mimbias commented on GitHub (Sep 18, 2025):
Hi again,
I removed gerbil from both instances. With only Pangolin (and Traefik) service, both instances start, it works very well:
- in an active/backup setup
- in an active/active round robin setup. It's impressive to see both instances serve sessions simultaneously - even websocket Apps such as Gitlab WebIDE and Minio bucket browser (that use to fail on "concurrent" products such as F5 APM and Gravitional Teleport)
it's sufficient for my use cases now. May try Olm soon
Been thinking about how to use gerbil in an HA setup, I guess it might be tricky to maintain connexions (UDP...).
Great job !
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @SpiderUnderUrBed on GitHub (Sep 13, 2025).
Original GitHub issue: https://github.com/fosrl/pangolin/issues/1453
Making this an issue for increased visability, based on the discussion #970
This issue is for tracking the progress of adding High Availability to pangolin, so if one server goes down, traffic is routed to the available one, which is done with health checks, and my description so far is a bit of a over-simplification of High Availability but currently, as I was told in the discussion, that pangolin will blindly round-robin between hosts which may be down, and therfore is not HA. If this feature is added it will increase the uses for pangolin significantly.
I also hope this is added to the project board.
@Esa-mimbias commented on GitHub (Sep 17, 2025):
Hi
I've been playing with this setup:
(same configuration as https://github.com/orgs/fosrl/discussions/1342, with 2 Pangolin and external Postgresql DB)
I start first instance, do initial configuration (admin account, an org and a site)
I then create and start second instance . It has trouble starting gerbil
when recreating instances from scratch, not dropping the DB, same issue :
dropping database and recreating first instance is OK, second instance fails the same way as mentioned before.
table exitNodes contains one entry:
@Esa-mimbias commented on GitHub (Sep 18, 2025):
Hi again,
I removed gerbil from both instances. With only Pangolin (and Traefik) service, both instances start, it works very well:
it's sufficient for my use cases now. May try Olm soon
Been thinking about how to use gerbil in an HA setup, I guess it might be tricky to maintain connexions (UDP...).
Great job !
@miloschwartz commented on GitHub (Sep 27, 2025):
Thanks, converting this to a discussion thread