Hello hello,
I am having some trouble linking up newt on my edge server to the stack running on the VPS. Pangolin, Gerbil, and Traefik (with Badger) are all running (startup in that order), and the dashboard is available at https://<dashboardUrl>. Login and site creation work just fine, but running the command suggested by site creation never receives a response, see below: root@edge_server# newt --id ddmmqvgjehmnxwg --secret <secret> --endpoint https://<dashboardUrl>
INFO: 2025/05/07 20:53:10 Sent registration message
root@vps# journalctl -eu fossorial:
Starting Fossorial Service...
node[93818]: Running migrations...
node[93818]: Migrations completed successfully.
systemd[1]: Started Fossorial Service.
node[93830]: 2025-05-07T18:51:55.874Z [warn]: Email SMTP configuration is missing. Emails will not be sent.
node[93830]: 2025-05-07T18:51:56.032Z [info]: Server admin created
node[93830]: 2025-05-07T18:51:57.049Z [info]: API server is running on http://localhost:3000
node[93830]: 2025-05-07T18:51:57.050Z [info]: Internal server is running on http://localhost:3001
node[93830]: 2025-05-07T18:51:58.647Z [info]: Next.js server is running on http://localhost:3002
node[93830]: 2025-05-07T18:52:01.212Z [info]: Created new exit node Exit Node zAmAytTq with address 100.89.128.1/24 and port 51820
node[93830]: 2025-05-07T18:53:10.790Z [info]: Establishing websocket connection
node[93830]: 2025-05-07T18:53:10.790Z [info]: Client added to tracking - Newt ID: ddmmqvgjehmnxwg, Total connections: 1
node[93830]: 2025-05-07T18:53:10.790Z [info]: WebSocket connection established - Newt ID: ddmmqvgjehmnxwg
node[93830]: 2025-05-07T18:53:10.802Z [info]: Handling register message!
node[93830]: 2025-05-07T18:53:10.829Z [error]: Message handling error: HTTP error! status: undefined
node[93830]: Stack: Error: HTTP error! status: undefined
node[93830]: at addPeer (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2370:13)
node[93830]: at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
node[93830]: at async handleRegisterMessage (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2722:3)
node[93830]: at async WebSocket.<anonymous> (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2884:25)
What am i doing wrong here?
Originally created by @jackrosenberg on GitHub (May 7, 2025).
Original GitHub issue: https://github.com/fosrl/newt/issues/37
Hello hello,
I am having some trouble linking up newt on my edge server to the stack running on the VPS. Pangolin, Gerbil, and Traefik (with Badger) are all running (startup in that order), and the dashboard is available at `https://<dashboardUrl>`. Login and site creation work just fine, but running the command suggested by site creation never receives a response, see below:
`root@edge_server# newt --id ddmmqvgjehmnxwg --secret <secret> --endpoint https://<dashboardUrl>`
```
INFO: 2025/05/07 20:53:10 Sent registration message
```
`root@vps# journalctl -eu fossorial:`
```
Starting Fossorial Service...
node[93818]: Running migrations...
node[93818]: Migrations completed successfully.
systemd[1]: Started Fossorial Service.
node[93830]: 2025-05-07T18:51:55.874Z [warn]: Email SMTP configuration is missing. Emails will not be sent.
node[93830]: 2025-05-07T18:51:56.032Z [info]: Server admin created
node[93830]: 2025-05-07T18:51:57.049Z [info]: API server is running on http://localhost:3000
node[93830]: 2025-05-07T18:51:57.050Z [info]: Internal server is running on http://localhost:3001
node[93830]: 2025-05-07T18:51:58.647Z [info]: Next.js server is running on http://localhost:3002
node[93830]: 2025-05-07T18:52:01.212Z [info]: Created new exit node Exit Node zAmAytTq with address 100.89.128.1/24 and port 51820
node[93830]: 2025-05-07T18:53:10.790Z [info]: Establishing websocket connection
node[93830]: 2025-05-07T18:53:10.790Z [info]: Client added to tracking - Newt ID: ddmmqvgjehmnxwg, Total connections: 1
node[93830]: 2025-05-07T18:53:10.790Z [info]: WebSocket connection established - Newt ID: ddmmqvgjehmnxwg
node[93830]: 2025-05-07T18:53:10.802Z [info]: Handling register message!
node[93830]: 2025-05-07T18:53:10.829Z [error]: Message handling error: HTTP error! status: undefined
node[93830]: Stack: Error: HTTP error! status: undefined
node[93830]: at addPeer (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2370:13)
node[93830]: at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
node[93830]: at async handleRegisterMessage (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2722:3)
node[93830]: at async WebSocket.<anonymous> (file:///nix/store/4sa8kqr8vzjv73b08r7f4nxpidh8pff0-pangolin-frontend-1.2.0/dist/server.mjs:2884:25)
```
What am i doing wrong here?
This usually occurs when Pangolin can not address the Gerbil container. Do you see any logs from Gerbil when this happens indicating for some reason that it did not start properly? How are you restarting the stack on the cloud?
<!-- gh-comment-id:2880211222 -->
@justincmoy commented on GitHub (May 14, 2025):
This seems to be a similar issue to https://github.com/fosrl/newt/issues/32. Comment from that thread:
> This usually occurs when Pangolin can not address the Gerbil container. Do you see any logs from Gerbil when this happens indicating for some reason that it did not start properly? How are you restarting the stack on the cloud?
@jackrosenberg commented on GitHub (May 14, 2025):
Saw that but I think connectivity is alright:
gerbil[139495]: INFO: 2025/05/10 20:52:56 Fetching remote config from http://localhost:3001/api/v1/gerbil/get-config
gerbil[139495]: ERROR: 2025/05/10 20:52:56 Error fetching remote config http://localhost:3001/api/v1/gerbil/get-config: Post "http://loca>
gerbil[139495]: ERROR: 2025/05/10 20:52:56 Failed to load configuration: Post "http://localhost:3001/api/v1/gerbil/get-config": dial tcp >
gerbil[139495]: INFO: 2025/05/10 20:53:01 Fetching remote config from http://localhost:3001/api/v1/gerbil/get-config
It fails the first couple of times when the services are still starting up, but then stops throwing errors and appears to be connected.
<!-- gh-comment-id:2880260813 -->
@jackrosenberg commented on GitHub (May 14, 2025):
Saw that but I think connectivity is alright:
```
gerbil[139495]: INFO: 2025/05/10 20:52:56 Fetching remote config from http://localhost:3001/api/v1/gerbil/get-config
gerbil[139495]: ERROR: 2025/05/10 20:52:56 Error fetching remote config http://localhost:3001/api/v1/gerbil/get-config: Post "http://loca>
gerbil[139495]: ERROR: 2025/05/10 20:52:56 Failed to load configuration: Post "http://localhost:3001/api/v1/gerbil/get-config": dial tcp >
gerbil[139495]: INFO: 2025/05/10 20:53:01 Fetching remote config from http://localhost:3001/api/v1/gerbil/get-config
```
It fails the first couple of times when the services are still starting up, but then stops throwing errors and appears to be connected.
@jackrosenberg commented on GitHub (May 14, 2025):
Although I am getting some potentially weird behavior here:
[root@vps:~]# curl localhost:3001/api/v1
{"message":"Healthy"}
[root@vps:~]# curl localhost:3001/api/v1/gerbil/get-config
{"data":null,"success":false,"error":true,"message":"The requests url is not found - /api/v1/gerbil/get-config","status":404,"stack":null}
<!-- gh-comment-id:2880276692 -->
@jackrosenberg commented on GitHub (May 14, 2025):
Although I am getting some potentially weird behavior here:
```
[root@vps:~]# curl localhost:3001/api/v1
{"message":"Healthy"}
[root@vps:~]# curl localhost:3001/api/v1/gerbil/get-config
{"data":null,"success":false,"error":true,"message":"The requests url is not found - /api/v1/gerbil/get-config","status":404,"stack":null}
```
I can confirm that this issue is caused by Pangolin not being able to communicate with Gerbil.
For me, the issue was caused by an incorrect host name in the --reachableAt argument passed to Gerbil. I also had to manually update the exitNodes table in the database with the correct host name.
<!-- gh-comment-id:2923246311 -->
@TyDooo commented on GitHub (May 30, 2025):
I can confirm that this issue is caused by Pangolin not being able to communicate with Gerbil.
For me, the issue was caused by an incorrect host name in the `--reachableAt` argument passed to Gerbil. I also had to manually update the `exitNodes` table in the database with the correct host name.
@jackrosenberg commented on GitHub (May 30, 2025):
Thanks a bunch! This got me an acknowledged response! What do you mean by the exitNodes table? Where is that stored?
<!-- gh-comment-id:2923424095 -->
@jackrosenberg commented on GitHub (May 30, 2025):
Thanks a bunch! This got me an acknowledged response! What do you mean by the exitNodes table? Where is that stored?
The exitNodes table is located within an SQLite database file. It is located in the db/ directory next to your pangolin configuration file.
Something like this should work to read from the database:
sqlite3 /var/lib/fossorial/config/db/db.sqlite "SELECT * FROM exitNodes"
In the output, the last column should match the value in the --reachableAt argument. If it matches, then you should be all good. Otherwise you need to change the value in the column to match the argument.
This may have been an issue specific to my setup though. So if it all works on your end then you don't need to worry about this :p
<!-- gh-comment-id:2923680252 -->
@TyDooo commented on GitHub (May 30, 2025):
The `exitNodes` table is located within an SQLite database file. It is located in the `db/` directory next to your pangolin configuration file.
Something like this should work to read from the database:
```bash
sqlite3 /var/lib/fossorial/config/db/db.sqlite "SELECT * FROM exitNodes"
```
In the output, the last column should match the value in the `--reachableAt` argument. If it matches, then you should be all good. Otherwise you need to change the value in the column to match the argument.
This may have been an issue specific to my setup though. So if it all works on your end then you don't need to worry about this :p
@oschwartz10612 commented on GitHub (May 31, 2025):
^^^^ this is likely the issue. When the gerbil registers this value gets set. If you are not using docker make sure that the Pangolin api can reach the Gerbil API over this URL.
<!-- gh-comment-id:2925280829 -->
@oschwartz10612 commented on GitHub (May 31, 2025):
^^^^ this is likely the issue. When the gerbil registers this value gets set. If you are not using docker make sure that the Pangolin api can reach the Gerbil API over this URL.
@jackrosenberg commented on GitHub (May 31, 2025):
This helps a lot, i'm onto the next issue!! It looks like newt's wg connection isn't being created properly. running root@edge_server# ip a | grep wg on the yields nothing root@edge_server# ~/./newt --id ... --secret <...> --endpoint https:/<dashboardurl>
I don't think Newt creates a new network interface; only Gerbil should do that. Could you try restarting your VPS? I initially had the same issue where Newt was unable to ping the server. Rebooting the VPS and the Newt instance fixed it.
<!-- gh-comment-id:2925534751 -->
@TyDooo commented on GitHub (May 31, 2025):
I don't think Newt creates a new network interface; only Gerbil should do that. Could you try restarting your VPS? I initially had the same issue where Newt was unable to ping the server. Rebooting the VPS and the Newt instance fixed it.
there are no errors in the logs of pangolin, gerbil or traefik
Whats going on here?
<!-- gh-comment-id:2941025626 -->
@jackrosenberg commented on GitHub (Jun 4, 2025):
Okay rebooting fixes almost everything! I still can't get the services to connect
`root@edge_server# ./newt --id {if} --secret {secret} --endpoint https://{dashboardUrl}`
```
INFO: 2025/06/04 19:52:56 Sent registration message
INFO: 2025/06/04 19:52:56 Received registration message
INFO: 2025/06/04 19:52:56 Received: {Type:newt/wg/connect Data:map[endpoint:{dashboardUrl}:51820 publicKey:{...} serverIP:100.89.128.1 targets:map[tcp:[62968:localhost:8096 57669:localhost:3002] udp:[]] tunnelIP:100.89.128.4]}
INFO: 2025/06/04 19:52:56 WireGuard device created. Lets ping the server now...
...
INFO: 2025/06/04 19:52:56 Started tcp proxy from 100.89.128.4:62968 to localhost:8096
INFO: 2025/06/04 19:52:56 Started tcp proxy from 100.89.128.4:57669 to localhost:3002
...
INFO: 2025/06/04 19:58:27 Received: {Type:newt/tcp/add Data:map[targets:[55601:localhost:2283]]}
INFO: 2025/06/04 19:58:27 Started tcp proxy from 100.89.128.4:55601 to localhost:2283
```
then i create a resource that points from p.{baseDomain} to localhost:2283
and when accessing p.{baseDomain}:
Internal Server Error
curl-ing from the vps proves the tunnel is working though!
`root@vps # curl 100.89.128.4:55601`
```
<!doctype html>
<html>
<head>
<!-- (used for SSR) -->
<!-- metadata:tags -->
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
...
```
there are no errors in the logs of pangolin, gerbil or traefik
Whats going on here?
<!-- gh-comment-id:2941302112 -->
@TyDooo commented on GitHub (Jun 4, 2025):
It looks as if Traefik isn’t receiving the expected routing rules.
Could you please verify what Pangolin is sending?
```bash
curl -s http://localhost:3001/api/v1/traefik-config | jq
```
Indeed, make sure the value of internal_hostname in the Pangolin config is set to localhost.
<!-- gh-comment-id:2941357434 -->
@TyDooo commented on GitHub (Jun 4, 2025):
Indeed, make sure the value of `internal_hostname` in the Pangolin config is set to `localhost`.
I don't think Newt creates a new network interface; only Gerbil should do that. Could you try restarting your VPS? I initially had the same issue where Newt was unable to ping the server. Rebooting the VPS and the Newt instance fixed it.
Any idea on what this reboot reloads btw? It would be nice to not need to reboot after install. @TyDooo @oschwartz10612
<!-- gh-comment-id:3041330621 -->
@jackrosenberg commented on GitHub (Jul 6, 2025):
> I don't think Newt creates a new network interface; only Gerbil should do that. Could you try restarting your VPS? I initially had the same issue where Newt was unable to ping the server. Rebooting the VPS and the Newt instance fixed it.
Any idea on what this reboot reloads btw? It would be nice to not need to reboot after install. @TyDooo @oschwartz10612
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Hard to say but I think restarting newt just re initiates the connection workflow that sometimes does not recover well when it looses connection to the vps. There are some changes coming to newt that should improve its reconnect and uptime performance!
<!-- gh-comment-id:3042205981 -->
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Hard to say but I think restarting newt just re initiates the connection workflow that sometimes does not recover well when it looses connection to the vps. There are some changes coming to newt that should improve its reconnect and uptime performance!
Oh I mean the vps. Is there some wireguard thing that can be manually refreshed?
<!-- gh-comment-id:3042212719 -->
@jackrosenberg commented on GitHub (Jul 6, 2025):
Oh I mean the vps. Is there some wireguard thing that can be manually refreshed?
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Yeah rebooting is kind of nuclear haha. You can just use docker and restart the gerbil container or do docker compose down and then docker compose up -d to restart the stack. This will restart wg.
<!-- gh-comment-id:3042215131 -->
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Yeah rebooting is kind of nuclear haha. You can just use docker and restart the gerbil container or do `docker compose down` and then `docker compose up -d` to restart the stack. This will restart wg.
Uhhh haha soo I run Pangolin and friends without docker, thru NixOs. I've tried restarting all systemd services with with Network in the name already.
<!-- gh-comment-id:3042334481 -->
@jackrosenberg commented on GitHub (Jul 6, 2025):
Uhhh haha soo I run Pangolin and friends without docker, thru NixOs. I've tried restarting all systemd services with with Network in the name already.
@jackrosenberg commented on GitHub (Aug 10, 2025):
@oschwartz10612, any idea what service/other thing this could be ?
<!-- gh-comment-id:3172572192 -->
@jackrosenberg commented on GitHub (Aug 10, 2025):
@oschwartz10612, any idea what service/other thing this could be ?
@oschwartz10612 commented on GitHub (Aug 16, 2025):
Ahh okay @jackrosenberg yeah its because Gerbil creates wg0 interface
that persists across restarts of the binary. The idea was it kind of
puppets and syncs the config of the wg interface if it persists accross
restarts of the binary but that might have some flaws.
To remove it you could do ip link del wg0 and then restart gerbil and
it will recreate it. I bet that would work!
<!-- gh-comment-id:3193385911 -->
@oschwartz10612 commented on GitHub (Aug 16, 2025):
Ahh okay @jackrosenberg yeah its because Gerbil creates wg0 interface
that persists across restarts of the binary. The idea was it kind of
puppets and syncs the config of the wg interface if it persists accross
restarts of the binary but that might have some flaws.
To remove it you could do `ip link del wg0` and then restart gerbil and
it will recreate it. I bet that would work!
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @jackrosenberg on GitHub (May 7, 2025).
Original GitHub issue: https://github.com/fosrl/newt/issues/37
Hello hello,
I am having some trouble linking up newt on my edge server to the stack running on the VPS. Pangolin, Gerbil, and Traefik (with Badger) are all running (startup in that order), and the dashboard is available at
https://<dashboardUrl>. Login and site creation work just fine, but running the command suggested by site creation never receives a response, see below:root@edge_server# newt --id ddmmqvgjehmnxwg --secret <secret> --endpoint https://<dashboardUrl>root@vps# journalctl -eu fossorial:What am i doing wrong here?
@justincmoy commented on GitHub (May 14, 2025):
This seems to be a similar issue to https://github.com/fosrl/newt/issues/32. Comment from that thread:
@jackrosenberg commented on GitHub (May 14, 2025):
Saw that but I think connectivity is alright:
It fails the first couple of times when the services are still starting up, but then stops throwing errors and appears to be connected.
@jackrosenberg commented on GitHub (May 14, 2025):
Although I am getting some potentially weird behavior here:
@TyDooo commented on GitHub (May 30, 2025):
I can confirm that this issue is caused by Pangolin not being able to communicate with Gerbil.
For me, the issue was caused by an incorrect host name in the
--reachableAtargument passed to Gerbil. I also had to manually update theexitNodestable in the database with the correct host name.@jackrosenberg commented on GitHub (May 30, 2025):
Thanks a bunch! This got me an acknowledged response! What do you mean by the exitNodes table? Where is that stored?
@TyDooo commented on GitHub (May 30, 2025):
The
exitNodestable is located within an SQLite database file. It is located in thedb/directory next to your pangolin configuration file.Something like this should work to read from the database:
In the output, the last column should match the value in the
--reachableAtargument. If it matches, then you should be all good. Otherwise you need to change the value in the column to match the argument.This may have been an issue specific to my setup though. So if it all works on your end then you don't need to worry about this :p
@oschwartz10612 commented on GitHub (May 31, 2025):
^^^^ this is likely the issue. When the gerbil registers this value gets set. If you are not using docker make sure that the Pangolin api can reach the Gerbil API over this URL.
@jackrosenberg commented on GitHub (May 31, 2025):
This helps a lot, i'm onto the next issue!! It looks like newt's wg connection isn't being created properly. running
root@edge_server# ip a | grep wgon the yields nothingroot@edge_server# ~/./newt --id ... --secret <...> --endpoint https:/<dashboardurl>gerbil conf:@TyDooo commented on GitHub (May 31, 2025):
I don't think Newt creates a new network interface; only Gerbil should do that. Could you try restarting your VPS? I initially had the same issue where Newt was unable to ping the server. Rebooting the VPS and the Newt instance fixed it.
@jackrosenberg commented on GitHub (Jun 4, 2025):
Okay rebooting fixes almost everything! I still can't get the services to connect
root@edge_server# ./newt --id {if} --secret {secret} --endpoint https://{dashboardUrl}then i create a resource that points from p.{baseDomain} to localhost:2283
and when accessing p.{baseDomain}:
Internal Server Error
curl-ing from the vps proves the tunnel is working though!
root@vps # curl 100.89.128.4:55601there are no errors in the logs of pangolin, gerbil or traefik
Whats going on here?
@TyDooo commented on GitHub (Jun 4, 2025):
It looks as if Traefik isn’t receiving the expected routing rules.
Could you please verify what Pangolin is sending?
@jackrosenberg commented on GitHub (Jun 4, 2025):
root@vps# curl -s http://localhost:3001/api/v1/traefik-config | jq"apiBaseUrl": "http://pangolin:3001/api/v1"should probably be localhost:3001/api/v1, since the pangolin interface doesn't exist
@TyDooo commented on GitHub (Jun 4, 2025):
Indeed, make sure the value of
internal_hostnamein the Pangolin config is set tolocalhost.@jackrosenberg commented on GitHub (Jun 4, 2025):
Works like a charm! Thanks so much for your help!!!
@jackrosenberg commented on GitHub (Jul 6, 2025):
Any idea on what this reboot reloads btw? It would be nice to not need to reboot after install. @TyDooo @oschwartz10612
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Hard to say but I think restarting newt just re initiates the connection workflow that sometimes does not recover well when it looses connection to the vps. There are some changes coming to newt that should improve its reconnect and uptime performance!
@jackrosenberg commented on GitHub (Jul 6, 2025):
Oh I mean the vps. Is there some wireguard thing that can be manually refreshed?
@oschwartz10612 commented on GitHub (Jul 6, 2025):
Yeah rebooting is kind of nuclear haha. You can just use docker and restart the gerbil container or do
docker compose downand thendocker compose up -dto restart the stack. This will restart wg.@jackrosenberg commented on GitHub (Jul 6, 2025):
Uhhh haha soo I run Pangolin and friends without docker, thru NixOs. I've tried restarting all systemd services with with Network in the name already.
@jackrosenberg commented on GitHub (Aug 10, 2025):
@oschwartz10612, any idea what service/other thing this could be ?
@oschwartz10612 commented on GitHub (Aug 16, 2025):
Ahh okay @jackrosenberg yeah its because Gerbil creates wg0 interface
that persists across restarts of the binary. The idea was it kind of
puppets and syncs the config of the wg interface if it persists accross
restarts of the binary but that might have some flaws.
To remove it you could do
ip link del wg0and then restart gerbil andit will recreate it. I bet that would work!