Compare commits

..

86 Commits
1.3.1 ... 1.4.2

Author SHA1 Message Date
Owen
e9752f868e Merge branch 'main' into dev 2025-08-23 12:17:58 -07:00
Owen Schwartz
866afaf749 Merge pull request #108 from firecat53/main
Bugfix for #107. Only update main.go
2025-08-22 21:42:36 -07:00
Owen
a12ae17a66 Add note about config 2025-08-22 21:34:47 -07:00
Owen
e0cba2e5c6 Merge branch 'site-targets' into dev 2025-08-19 10:57:25 -07:00
Scott Hansen
79f3db6fb6 Bugfix for #107. Only update main.go 2025-08-16 15:25:23 -07:00
Owen Schwartz
009b4cf425 Merge pull request #107 from firecat53/main
Update version to 1.4.1 and update version_replaceme when using nix build
2025-08-15 09:40:40 -07:00
Scott Hansen
9c28d75155 Update version to 1.4.1 and update version_replaceme when using nix build 2025-08-14 11:47:40 -07:00
Owen
bad244d0ea Merge branch 'main' into dev 2025-08-13 14:56:02 -07:00
Owen
d013dc0543 Adjust logging 2025-08-13 14:18:47 -07:00
Owen
0047b54e94 Dont override ENV
Fixes #101
2025-08-12 20:44:34 -07:00
Owen
f0c8d2c7c7 Change permissions to 0600
Fixes #104
2025-08-11 08:15:36 -07:00
Owen
28b6865f73 Healthcheck working 2025-08-11 08:14:29 -07:00
Owen
289cce3a22 Add health checks 2025-08-03 18:43:43 -07:00
Owen
e8612c7e6b Handle adding and removing healthchecks 2025-08-03 17:02:15 -07:00
Owen
6820f8d23e Add basic heathchecks 2025-08-03 16:12:00 -07:00
Owen
151d0e38e6 Stop sending requests when you get a terminate 2025-08-03 14:47:36 -07:00
Owen Schwartz
3ccd755d55 Merge pull request #95 from fosrl/dependabot/go_modules/prod-patch-updates-e08645070f
Bump github.com/docker/docker from 28.3.2+incompatible to 28.3.3+incompatible in the prod-patch-updates group
2025-07-29 23:24:19 -07:00
Owen Schwartz
a0f0b674e8 Merge pull request #96 from firecat53/main
Update flake.nix to 1.4.0
2025-07-29 23:24:03 -07:00
Owen
9e675121d3 Dont reset dns 2025-07-29 22:42:54 -07:00
Owen
45d17da570 Fix the bind problem by just recreating the dev
TODO: WHY CANT WE REBIND TO A PORT - WE NEED TO FIX THIS BETTER
2025-07-29 20:58:48 -07:00
Owen
dfba35f8bb Use the tunnel ip 2025-07-29 16:31:42 -07:00
Scott Hansen
9e73aab21d Update flake.nix to 1.4.0 2025-07-29 14:14:42 -07:00
dependabot[bot]
e1ddad006a Bump github.com/docker/docker in the prod-patch-updates group
Bumps the prod-patch-updates group with 1 update: [github.com/docker/docker](https://github.com/docker/docker).


Updates `github.com/docker/docker` from 28.3.2+incompatible to 28.3.3+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](https://github.com/docker/docker/compare/v28.3.2...v28.3.3)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-version: 28.3.3+incompatible
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-patch-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-29 16:39:02 +00:00
Owen
29567d6e0b Dont print private key 2025-07-28 20:07:13 -07:00
Owen
47321ea9ad Update readme: env 2025-07-28 12:34:38 -07:00
Owen
abfc9d8efc Update readme: cli 2025-07-28 12:12:40 -07:00
Owen
c6929621e7 Merge branch 'main' into dev 2025-07-28 12:02:22 -07:00
Owen
46993203a3 Update readme 2025-07-28 12:02:10 -07:00
Owen
8306084354 SSH not ready 2025-07-28 12:02:09 -07:00
Owen
02c1e2b7d0 Compute kind of works now!? 2025-07-28 12:02:09 -07:00
Owen
ae7e2a1055 Clean up operation 2025-07-28 12:02:09 -07:00
Owen Schwartz
88f1335cff Merge pull request #93 from Lokowitz/sync-go-version
Sync go version
2025-07-28 11:59:10 -07:00
Owen
8bf9c9795b Netstack working 2025-07-27 10:25:34 -07:00
Marvin
5d343cd420 modified: go.mod
modified:   go.sum
2025-07-26 13:25:52 +00:00
Marvin
d1473b7e22 go.mod aktualisieren 2025-07-26 10:32:20 +02:00
Marvin
2efbd7dd6a Dockerfile aktualisieren 2025-07-26 10:31:53 +02:00
Marvin
82a3a39a1f .go-version aktualisieren 2025-07-26 10:31:35 +02:00
Marvin
df09193834 cicd.yml aktualisieren 2025-07-26 10:31:20 +02:00
Marvin
b2fe4e3b03 test.yml aktualisieren 2025-07-26 10:31:05 +02:00
Owen
e14d53087f Starting to work on option 2025-07-25 16:16:33 -07:00
Owen
3583270f73 Adding option for netstack 2025-07-25 16:16:00 -07:00
Owen
f5be05c55a Add flag 2025-07-25 16:14:25 -07:00
Owen
d09e3fbd60 Proxies working 2025-07-25 16:10:53 -07:00
Owen
493831b5f0 Pm working 2025-07-25 13:09:11 -07:00
Owen
9fc692c090 Proxy working? 2025-07-25 12:00:09 -07:00
Owen
ccb7008579 Just hp like olm 2025-07-25 11:42:36 -07:00
Owen
f17dbe1fef Use normal udp 2025-07-25 11:05:24 -07:00
Owen
27561f52ca Dont restart netstack 2025-07-25 11:01:54 -07:00
Owen
499ebcd928 Maybe its working? 2025-07-25 10:59:34 -07:00
Owen
40dfab31a5 Maybe basic func 2025-07-25 10:50:02 -07:00
Owen
56377ec87e Exit well 2025-07-24 20:46:33 -07:00
Owen
008be54c55 Add get config 2025-07-24 12:40:14 -07:00
Owen
64c22a94a4 Log to file optionally and update config locations 2025-07-24 12:01:53 -07:00
Owen Schwartz
468c93c581 Merge pull request #91 from fosrl/dependabot/go_modules/prod-minor-updates-17f8beca3b
Bump software.sslmate.com/src/go-pkcs12 from 0.5.0 to 0.6.0 in the prod-minor-updates group
2025-07-23 11:26:32 -07:00
Owen Schwartz
c53b859cda Merge pull request #92 from nepthar/patch-1
Nit: Typo fix in help string
2025-07-23 11:26:15 -07:00
Jordan Parker
6cd824baf2 Nit: Typo fix in help string 2025-07-23 10:25:11 -04:00
dependabot[bot]
d8c5182acd Bump software.sslmate.com/src/go-pkcs12 in the prod-minor-updates group
Bumps the prod-minor-updates group with 1 update: software.sslmate.com/src/go-pkcs12.


Updates `software.sslmate.com/src/go-pkcs12` from 0.5.0 to 0.6.0

---
updated-dependencies:
- dependency-name: software.sslmate.com/src/go-pkcs12
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-minor-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-23 10:01:13 +00:00
Owen
c8c4666d63 Change rm to keep 2025-07-22 11:36:31 -07:00
Owen
f1fcc13e66 Holepunch to the right endpoint 2025-07-21 17:04:05 -07:00
Owen Schwartz
52bbc2fe31 Merge pull request #90 from firecat53/main
Update flake.nix to 1.3.4
2025-07-21 14:54:22 -07:00
Scott Hansen
b5ee12f84a Update flake.nix for 1.3.4 2025-07-21 11:23:50 -07:00
Owen
510e78437c Add client type 2025-07-18 16:55:38 -07:00
Owen
e14cffce1c Merge branch 'main' into clients-fr 2025-07-18 16:53:27 -07:00
Owen
629a92ee81 Make client work for olm 2025-07-18 16:53:13 -07:00
Owen
56df75544d Adjust logging 2025-07-18 16:52:59 -07:00
Owen
5b2e743470 Remove defers causing bad file descriptor issues 2025-07-18 15:49:57 -07:00
Owen
b5025c142f Working on it 2025-07-18 15:25:39 -07:00
Owen
cd86e6b6de Dont ping if there is just 1 2025-07-17 15:01:02 -07:00
Owen
230c34e4e0 Make sure to only exclude if there is anouther 2025-07-16 21:15:03 -07:00
Owen
a038ce1458 Move docker messages to debug for #86 2025-07-14 10:16:06 -07:00
Owen Schwartz
cd83efd365 Merge pull request #85 from firecat53/main
Update flake to 1.3.2
2025-07-13 19:47:06 -07:00
Scott Hansen
702f39e870 Update flake to 1.3.2 2025-07-13 18:47:20 -07:00
Owen Schwartz
02b7ea51af Merge pull request #84 from fosrl/dev
Ping improvements, dependabot, hostname in docker
2025-07-13 16:26:11 -07:00
Owen
e8421364fc Merge branch 'woutervanelten-patch-2' into dev 2025-07-13 16:12:42 -07:00
Owen
7264bb7001 Merge branch 'patch-2' of github.com:woutervanelten/newt into woutervanelten-patch-2 2025-07-13 16:10:23 -07:00
Owen
86e262ac1e Merge branch 'main' into dev 2025-07-13 16:10:02 -07:00
Owen Schwartz
dcacc03e96 Merge pull request #81 from fosrl/dependabot/go_modules/prod-patch-updates-f7fa3bf88c
Bump github.com/vishvananda/netlink from 1.3.0 to 1.3.1 in the prod-patch-updates group
2025-07-13 16:09:48 -07:00
Owen
6f4469a5a4 Merge branch 'woutervanelten-patch-5' into dev 2025-07-13 16:09:33 -07:00
Owen
663e28329b Fix typo with _ 2025-07-13 16:08:32 -07:00
Owen
f513f97fc3 Working on better ping 2025-07-13 16:07:46 -07:00
dependabot[bot]
ce4f3e4cdf Bump github.com/vishvananda/netlink in the prod-patch-updates group
Bumps the prod-patch-updates group with 1 update: [github.com/vishvananda/netlink](https://github.com/vishvananda/netlink).


Updates `github.com/vishvananda/netlink` from 1.3.0 to 1.3.1
- [Release notes](https://github.com/vishvananda/netlink/releases)
- [Commits](https://github.com/vishvananda/netlink/compare/v1.3.0...v1.3.1)

---
updated-dependencies:
- dependency-name: github.com/vishvananda/netlink
  dependency-version: 1.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-patch-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-13 23:07:14 +00:00
Owen
58a74fce6f Merge branch 'patch-5' of github.com:woutervanelten/newt into woutervanelten-patch-5 2025-07-13 16:06:54 -07:00
Owen Schwartz
fc965abbc4 Merge pull request #82 from fosrl/dependabot/go_modules/prod-minor-updates-d992c0ea53
Bump the prod-minor-updates group with 2 updates
2025-07-13 16:05:52 -07:00
dependabot[bot]
6160e4c8a6 Bump the prod-minor-updates group with 2 updates
Bumps the prod-minor-updates group with 2 updates: [golang.org/x/crypto](https://github.com/golang/crypto) and [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/crypto` from 0.39.0 to 0.40.0
- [Commits](https://github.com/golang/crypto/compare/v0.39.0...v0.40.0)

Updates `golang.org/x/net` from 0.41.0 to 0.42.0
- [Commits](https://github.com/golang/net/compare/v0.41.0...v0.42.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.40.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-minor-updates
- dependency-name: golang.org/x/net
  dependency-version: 0.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-minor-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-11 09:32:02 +00:00
Wouter van Elten
4d343e3541 Update README.md for health check
Added explanation of health_file
2025-07-11 08:14:32 +02:00
Wouter van Elten
9348842e2c added use of hostname if available 2025-07-01 17:26:49 +02:00
25 changed files with 3151 additions and 399 deletions

View File

@@ -33,7 +33,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v4
with:
go-version: 1.23.1
go-version: 1.24
- name: Update version in main.go
run: |

View File

@@ -16,7 +16,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.23'
go-version: '1.24'
- name: Build go
run: go build

View File

@@ -1 +1 @@
1.23.2
1.24

View File

@@ -1,4 +1,4 @@
FROM golang:1.24.5-alpine AS builder
FROM golang:1.24-alpine AS builder
# Set the working directory inside the container
WORKDIR /app

274
README.md
View File

@@ -12,7 +12,7 @@ Newt is used with Pangolin and Gerbil as part of the larger system. See document
<img src="public/screenshots/preview.png" alt="Preview"/>
_Sample output of a Newt container connected to Pangolin and hosting various resource target proxies._
_Sample output of a Newt connected to Pangolin and hosting various resource target proxies._
## Key Functions
@@ -22,7 +22,7 @@ Using the Newt ID and a secret, the client will make HTTP requests to Pangolin t
### Receives WireGuard Control Messages
When Newt receives WireGuard control messages, it will use the information encoded (endpoint, public key) to bring up a WireGuard tunnel using [netstack](https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go) fully in user space. It will ping over the tunnel to ensure the peer on the Gerbil side is brought up.
When Newt receives WireGuard control messages, it will use the information encoded (endpoint, public key) to bring up a WireGuard tunnel using [netstack](https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go) fully in user space. It will ping over the tunnel to ensure the peer on the Gerbil side is brought up.
### Receives Proxy Control Messages
@@ -30,21 +30,80 @@ When Newt receives WireGuard control messages, it will use the information encod
## CLI Args
- `endpoint`: The endpoint where both Gerbil and Pangolin reside in order to connect to the websocket.
- `id`: Newt ID generated by Pangolin to identify the client.
- `secret`: A unique secret (not shared and kept private) used to authenticate the client ID with the websocket in order to receive commands.
- `mtu`: MTU for the internal WG interface. Default: 1280
- `dns`: DNS server to use to resolve the endpoint
- `log-level` (optional): The log level to use. Default: INFO
- `updown` (optional): A script to be called when targets are added or removed.
- `tls-client-cert` (optional): Client certificate (p12 or pfx) for mTLS. See [mTLS](#mtls)
- `docker-socket` (optional): Set the Docker socket to use the container discovery integration
- `docker-enforce-network-validation` (optional): Validate the container target is on the same network as the newt process
- `id`: Newt ID generated by Pangolin to identify the client.
- `secret`: A unique secret (not shared and kept private) used to authenticate the client ID with the websocket in order to receive commands.
- `endpoint`: The endpoint where both Gerbil and Pangolin reside in order to connect to the websocket.
- Example:
- `mtu` (optional): MTU for the internal WG interface. Default: 1280
- `dns` (optional): DNS server to use to resolve the endpoint. Default: 8.8.8.8
- `log-level` (optional): The log level to use (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO
- `docker-socket` (optional): Set the Docker socket to use the container discovery integration
- `ping-interval` (optional): Interval for pinging the server. Default: 3s
- `ping-timeout` (optional): Timeout for each ping. Default: 5s
- `updown` (optional): A script to be called when targets are added or removed.
- `tls-client-cert` (optional): Client certificate (p12 or pfx) for mTLS. See [mTLS](#mtls)
- `docker-enforce-network-validation` (optional): Validate the container target is on the same network as the newt process. Default: false
- `health-file` (optional): Check if connection to WG server (pangolin) is ok. creates a file if ok, removes it if not ok. Can be used with docker healtcheck to restart newt
- `accept-clients` (optional): Enable WireGuard server mode to accept incoming newt client connections. Default: false
- `generateAndSaveKeyTo` (optional): Path to save generated private key
- `native` (optional): Use native WireGuard interface when accepting clients (requires WireGuard kernel module and Linux, must run as root). Default: false (uses userspace netstack)
- `interface` (optional): Name of the WireGuard interface. Default: newt
- `keep-interface` (optional): Keep the WireGuard interface. Default: false
## Environment Variables
All CLI arguments can be set using environment variables as an alternative to command line flags. Environment variables are particularly useful when running Newt in containerized environments.
- `PANGOLIN_ENDPOINT`: Endpoint of your pangolin server (equivalent to `--endpoint`)
- `NEWT_ID`: Newt ID generated by Pangolin (equivalent to `--id`)
- `NEWT_SECRET`: Newt secret for authentication (equivalent to `--secret`)
- `MTU`: MTU for the internal WG interface. Default: 1280 (equivalent to `--mtu`)
- `DNS`: DNS server to use to resolve the endpoint. Default: 8.8.8.8 (equivalent to `--dns`)
- `LOG_LEVEL`: Log level (DEBUG, INFO, WARN, ERROR, FATAL). Default: INFO (equivalent to `--log-level`)
- `DOCKER_SOCKET`: Path to Docker socket for container discovery (equivalent to `--docker-socket`)
- `PING_INTERVAL`: Interval for pinging the server. Default: 3s (equivalent to `--ping-interval`)
- `PING_TIMEOUT`: Timeout for each ping. Default: 5s (equivalent to `--ping-timeout`)
- `UPDOWN_SCRIPT`: Path to updown script for target add/remove events (equivalent to `--updown`)
- `TLS_CLIENT_CERT`: Path to client certificate for mTLS (equivalent to `--tls-client-cert`)
- `DOCKER_ENFORCE_NETWORK_VALIDATION`: Validate container targets are on same network. Default: false (equivalent to `--docker-enforce-network-validation`)
- `HEALTH_FILE`: Path to health file for connection monitoring (equivalent to `--health-file`)
- `ACCEPT_CLIENTS`: Enable WireGuard server mode. Default: false (equivalent to `--accept-clients`)
- `GENERATE_AND_SAVE_KEY_TO`: Path to save generated private key (equivalent to `--generateAndSaveKeyTo`)
- `USE_NATIVE_INTERFACE`: Use native WireGuard interface (Linux only). Default: false (equivalent to `--native`)
- `INTERFACE`: Name of the WireGuard interface. Default: newt (equivalent to `--interface`)
- `KEEP_INTERFACE`: Keep the WireGuard interface after shutdown. Default: false (equivalent to `--keep-interface`)
- `CONFIG_FILE`: Load the config json from this file instead of in the home folder.
## Loading secrets from files
You can use `CONFIG_FILE` to define a location of a config file to store the credentials between runs.
```
$ cat ~/.config/newt-client/config.json
{
"id": "spmzu8rbpzj1qq6",
"secret": "f6v61mjutwme2kkydbw3fjo227zl60a2tsf5psw9r25hgae3",
"endpoint": "https://pangolin.fossorial.io",
"tlsClientCert": ""
}
```
This file is also written to when newt first starts up. So you do not need to run every time with --id and secret if you have run it once!
Default locations:
- **macOS**: `~/Library/Application Support/newt-client/config.json`
- **Windows**: `%PROGRAMDATA%\newt\newt-client\config.json`
- **Linux/Others**: `~/.config/newt-client/config.json`
## Examples
**Note**: When both environment variables and CLI arguments are provided, CLI arguments take precedence.
- Example:
```bash
./newt \
newt \
--id 31frd0uzbjvp721 \
--secret h51mmlknrvrwv8s4r1i210azhumt6isgbpyavxodibx1k2d6 \
--endpoint https://example.com
@@ -54,30 +113,116 @@ You can also run it with Docker compose. For example, a service in your `docker-
```yaml
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- HEALTH_FILE=/tmp/healthy
```
You can also pass the CLI args to the container:
```yaml
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
command:
- --id 31frd0uzbjvp721
- --secret h51mmlknrvrwv8s4r1i210azhumt6isgbpyavxodibx1k2d6
- --endpoint https://example.com
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
command:
- --id 31frd0uzbjvp721
- --secret h51mmlknrvrwv8s4r1i210azhumt6isgbpyavxodibx1k2d6
- --endpoint https://example.com
- --health-file /tmp/healthy
```
## Accept Client Connections
When the `--accept-clients` flag is enabled (or `ACCEPT_CLIENTS=true` environment variable is set), Newt operates as a WireGuard server that can accept incoming client connections from other devices. This enables peer-to-peer connectivity through the Newt instance.
### How It Works
In client acceptance mode, Newt:
- **Creates a WireGuard service** that can accept incoming connections from other WireGuard clients
- **Starts a connection testing server** (WGTester) that responds to connectivity checks from remote clients
- **Manages peer configurations** dynamically based on Pangolin's instructions
- **Enables bidirectional communication** between the Newt instance and connected clients
### Use Cases
- **Site-to-site connectivity**: Connect multiple locations through a central Newt instance
- **Client access to private networks**: Allow remote clients to access resources behind the Newt instance
- **Development environments**: Provide developers secure access to internal services
### Client Tunneling Modes
Newt supports two WireGuard tunneling modes:
#### Userspace Mode (Default)
By default, Newt uses a fully userspace WireGuard implementation using [netstack](https://github.com/WireGuard/wireguard-go/blob/master/tun/netstack/examples/http_server.go). This mode:
- **Does not require root privileges**
- **Works on all supported platforms** (Linux, Windows, macOS)
- **Does not require WireGuard kernel module** to be installed
- **Runs entirely in userspace** - no system network interface is created
- **Is containerization-friendly** - works seamlessly in Docker containers
This is the recommended mode for most deployments, especially containerized environments.
In this mode, TCP and UDP is proxied out of newt from the remote client using TCP/UDP resources in Pangolin.
#### Native Mode (Linux only)
When using the `--native` flag or setting `USE_NATIVE_INTERFACE=true`, Newt uses the native WireGuard kernel module. This mode:
- **Requires root privileges** to create and manage network interfaces
- **Only works on Linux** with the WireGuard kernel module installed
- **Creates a real network interface** (e.g., `newt0`) on the system
- **May offer better performance** for high-throughput scenarios
- **Requires proper network permissions** and may conflict with existing network configurations
In this mode it functions like a traditional VPN interface - all data arrives on the interface and you must get it to the destination (or access things locally).
#### Native Mode Requirements
To use native mode:
1. Run on a Linux system
2. Install the WireGuard kernel module
3. Run Newt as root (`sudo`)
4. Ensure the system allows creation of network interfaces
Docker Compose example:
```yaml
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- ACCEPT_CLIENTS=true
```
### Technical Details
When client acceptance is enabled:
- **WGTester Server**: Runs on `port + 1` (e.g., if WireGuard uses port 51820, WGTester uses 51821)
- **Connection Testing**: Responds to UDP packets with magic header `0xDEADBEEF` for connectivity verification
- **Dynamic Configuration**: Peer configurations are managed remotely through Pangolin
- **Proxy Integration**: Can work with both userspace (netstack) and native WireGuard modes
**Note**: Client acceptance mode requires coordination with Pangolin for peer management and configuration distribution.
### Docker Socket Integration
Newt can integrate with the Docker socket to provide remote inspection of Docker containers. This allows Pangolin to query and retrieve detailed information about containers running on the Newt client, including metadata, network configuration, port mappings, and more.
@@ -88,26 +233,27 @@ You can specify the Docker socket path using the `--docker-socket` CLI argument
```yaml
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- DOCKER_SOCKET=/var/run/docker.sock
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- DOCKER_SOCKET=/var/run/docker.sock
```
#### Hostnames vs IPs
When the Docker Socket Integration is used, depending on the network which Newt is run with, either the hostname (generally considered the container name) or the IP address of the container will be sent to Pangolin. Here are some of the scenarios where IPs or hostname of the container will be utilised:
- **Running in Network Mode 'host'**: IP addresses will be used
- **Running in Network Mode 'bridge'**: IP addresses will be used
- **Running in docker-compose without a network specification**: Docker compose creates a network for the compose by default, hostnames will be used
- **Running on docker-compose with defined network**: Hostnames will be used
- **Running in Network Mode 'host'**: IP addresses will be used
- **Running in Network Mode 'bridge'**: IP addresses will be used
- **Running in docker-compose without a network specification**: Docker compose creates a network for the compose by default, hostnames will be used
- **Running on docker-compose with defined network**: Hostnames will be used
### Docker Enforce Network Validation
@@ -127,7 +273,7 @@ You can pass in a updown script for Newt to call when it is adding or removing a
`--updown "python3 test.py"`
It will get called with args when a target is added:
It will get called with args when a target is added:
`python3 test.py add tcp localhost:8556`
`python3 test.py remove tcp localhost:8556`
@@ -136,18 +282,20 @@ Returning a string from the script in the format of a target (`ip:dst` so `10.0.
You can look at updown.py as a reference script to get started!
### mTLS
Newt supports mutual TLS (mTLS) authentication, if the server has been configured to request a client certificate.
* Only PKCS12 (.p12 or .pfx) file format is accepted
* The PKCS12 file must contain:
* Private key
* Public certificate
* CA certificate
* Encrypted PKCS12 files are currently not supported
- Only PKCS12 (.p12 or .pfx) file format is accepted
- The PKCS12 file must contain:
- Private key
- Public certificate
- CA certificate
- Encrypted PKCS12 files are currently not supported
Examples:
```bash
./newt \
newt \
--id 31frd0uzbjvp721 \
--secret h51mmlknrvrwv8s4r1i210azhumt6isgbpyavxodibx1k2d6 \
--endpoint https://example.com \
@@ -156,20 +304,20 @@ Examples:
```yaml
services:
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- TLS_CLIENT_CERT=./client.p12
newt:
image: fosrl/newt
container_name: newt
restart: unless-stopped
environment:
- PANGOLIN_ENDPOINT=https://example.com
- NEWT_ID=2ix2t8xk22ubpfy
- NEWT_SECRET=nnisrfsdfc7prqsp9ewo1dvtvci50j5uiqotez00dgap0ii2
- TLS_CLIENT_CERT=./client.p12
```
## Build
### Container
### Container
Ensure Docker is installed.

132
clients.go Normal file
View File

@@ -0,0 +1,132 @@
package main
import (
"fmt"
"strings"
"github.com/fosrl/newt/logger"
"github.com/fosrl/newt/proxy"
"github.com/fosrl/newt/websocket"
"golang.zx2c4.com/wireguard/tun/netstack"
"github.com/fosrl/newt/wgnetstack"
"github.com/fosrl/newt/wgtester"
)
var wgService *wgnetstack.WireGuardService
var wgTesterServer *wgtester.Server
var ready bool
func setupClients(client *websocket.Client) {
var host = endpoint
if strings.HasPrefix(host, "http://") {
host = strings.TrimPrefix(host, "http://")
} else if strings.HasPrefix(host, "https://") {
host = strings.TrimPrefix(host, "https://")
}
host = strings.TrimSuffix(host, "/")
if useNativeInterface {
setupClientsNative(client, host)
} else {
setupClientsNetstack(client, host)
}
ready = true
}
func setupClientsNetstack(client *websocket.Client, host string) {
logger.Info("Setting up clients with netstack...")
// Create WireGuard service
wgService, err = wgnetstack.NewWireGuardService(interfaceName, mtuInt, generateAndSaveKeyTo, host, id, client, "8.8.8.8")
if err != nil {
logger.Fatal("Failed to create WireGuard service: %v", err)
}
// // Set up callback to restart wgtester with netstack when WireGuard is ready
wgService.SetOnNetstackReady(func(tnet *netstack.Net) {
wgTesterServer = wgtester.NewServerWithNetstack("0.0.0.0", wgService.Port, id, tnet) // TODO: maybe make this the same ip of the wg server?
err := wgTesterServer.Start()
if err != nil {
logger.Error("Failed to start WireGuard tester server: %v", err)
}
})
wgService.SetOnNetstackClose(func() {
if wgTesterServer != nil {
wgTesterServer.Stop()
wgTesterServer = nil
}
})
client.OnTokenUpdate(func(token string) {
wgService.SetToken(token)
})
}
func setDownstreamTNetstack(tnet *netstack.Net) {
if wgService != nil {
wgService.SetOthertnet(tnet)
}
}
func closeClients() {
logger.Info("Closing clients...")
if wgService != nil {
wgService.Close(!keepInterface)
wgService = nil
}
closeWgServiceNative()
if wgTesterServer != nil {
wgTesterServer.Stop()
wgTesterServer = nil
}
}
func clientsHandleNewtConnection(publicKey string, endpoint string) {
if !ready {
return
}
// split off the port from the endpoint
parts := strings.Split(endpoint, ":")
if len(parts) < 2 {
logger.Error("Invalid endpoint format: %s", endpoint)
return
}
endpoint = strings.Join(parts[:len(parts)-1], ":")
if wgService != nil {
wgService.StartHolepunch(publicKey, endpoint)
}
clientsHandleNewtConnectionNative(publicKey, endpoint)
}
func clientsOnConnect() {
if !ready {
return
}
if wgService != nil {
wgService.LoadRemoteConfig()
}
clientsOnConnectNative()
}
func clientsAddProxyTarget(pm *proxy.ProxyManager, tunnelIp string) {
if !ready {
return
}
// add a udp proxy for localost and the wgService port
// TODO: make sure this port is not used in a target
if wgService != nil {
pm.AddTarget("udp", tunnelIp, int(wgService.Port), fmt.Sprintf("127.0.0.1:%d", wgService.Port))
}
clientsAddProxyTargetNative(pm, tunnelIp)
}

View File

@@ -26,6 +26,8 @@ type Container struct {
Labels map[string]string `json:"labels"`
Created int64 `json:"created"`
Networks map[string]Network `json:"networks"`
Hostname string `json:"hostname"` // added to use hostname if available instead of network address
}
// Port represents a port mapping for a Docker container
@@ -173,6 +175,14 @@ func ListContainers(socketPath string, enforceNetworkValidation bool) ([]Contain
// Short ID like docker ps
shortId := c.ID[:12]
// Inspect container to get hostname
hostname := ""
containerInfo, err := cli.ContainerInspect(ctx, c.ID)
if err == nil && containerInfo.Config != nil {
hostname = containerInfo.Config.Hostname
}
// Skip host container if set
if hostContainerId != "" && c.ID == hostContainerId {
continue
@@ -238,6 +248,7 @@ func ListContainers(socketPath string, enforceNetworkValidation bool) ([]Contain
Labels: c.Labels,
Created: c.Created,
Networks: networks,
Hostname: hostname, // added
}
dockerContainers = append(dockerContainers, dockerContainer)

6
flake.lock generated
View File

@@ -2,11 +2,11 @@
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1749086602,
"narHash": "sha256-DJcgJMekoxVesl9kKjfLPix2Nbr42i7cpEHJiTnBUwU=",
"lastModified": 1753489912,
"narHash": "sha256-uDCFHeXdRIgJpYmtcUxGEsZ+hYlLPBhR83fdU+vbC1s=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "4792576cb003c994bd7cc1edada3129def20b27d",
"rev": "13e8d35b7d6028b7198f8186bc0347c6abaa2701",
"type": "github"
},
"original": {

View File

@@ -22,17 +22,25 @@
system:
let
pkgs = pkgsFor system;
# Update version when releasing
version = "1.4.1";
# Update the version in a new source tree
srcWithReplacedVersion = pkgs.runCommand "newt-src-with-version" { } ''
cp -r ${./.} $out
chmod -R +w $out
rm -rf $out/.git $out/result $out/.envrc $out/.direnv
sed -i "s/version_replaceme/${version}/g" $out/main.go
'';
in
{
default = self.packages.${system}.pangolin-newt;
pangolin-newt = pkgs.buildGoModule {
pname = "pangolin-newt";
version = "1.2.1";
src = ./.;
vendorHash = "sha256-Yc5IXnShciek/bKkVezkAcaq47zGiZP8vUHFb9p09LI=";
version = version;
src = srcWithReplacedVersion;
vendorHash = "sha256-PENsCO2yFxLVZNPgx2OP+gWVNfjJAfXkwWS7tzlm490=";
meta = with pkgs.lib; {
description = "A tunneling client for Pangolin";
homepage = "https://github.com/fosrl/newt";

42
go.mod
View File

@@ -1,35 +1,33 @@
module github.com/fosrl/newt
go 1.23.1
toolchain go1.23.2
go 1.24
require (
github.com/docker/docker v28.3.2+incompatible
github.com/docker/docker v28.3.3+incompatible
github.com/google/gopacket v1.1.19
github.com/gorilla/websocket v1.5.3
github.com/vishvananda/netlink v1.3.0
golang.org/x/crypto v0.39.0
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa
golang.org/x/net v0.41.0
github.com/vishvananda/netlink v1.3.1
golang.org/x/crypto v0.40.0
golang.org/x/exp v0.0.0-20250718183923-645b1fa84792
golang.org/x/net v0.42.0
golang.zx2c4.com/wireguard v0.0.0-20250521234502-f333402bd9cb
golang.zx2c4.com/wireguard/wgctrl v0.0.0-20241231184526-a9ab2273dd10
gvisor.dev/gvisor v0.0.0-20250503011706-39ed1f5ac29c
software.sslmate.com/src/go-pkcs12 v0.5.0
software.sslmate.com/src/go-pkcs12 v0.6.0
)
require (
github.com/Microsoft/go-winio v0.6.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/mdlayher/genetlink v1.3.2 // indirect
@@ -42,17 +40,15 @@ require (
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/vishvananda/netns v0.0.4 // indirect
github.com/vishvananda/netns v0.0.5 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel v1.36.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect
go.opentelemetry.io/otel v1.37.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/otel/trace v1.36.0 // indirect
golang.org/x/mod v0.23.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/time v0.7.0 // indirect
golang.org/x/tools v0.30.0 // indirect
go.opentelemetry.io/otel/metric v1.37.0 // indirect
go.opentelemetry.io/otel/trace v1.37.0 // indirect
golang.org/x/sync v0.16.0 // indirect
golang.org/x/sys v0.34.0 // indirect
golang.org/x/time v0.12.0 // indirect
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
)

88
go.sum
View File

@@ -1,7 +1,7 @@
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Microsoft/go-winio v0.6.0 h1:slsWYD/zyx7lCXoZVlvQrj0hPTM1HI4+v1sIda2yDvg=
github.com/Microsoft/go-winio v0.6.0/go.mod h1:cTAf44im0RAYeL23bpB+fzCyDH2MJiz2BO69KH/soAE=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
github.com/cenkalti/backoff/v5 v5.0.2 h1:rIfFVxEf1QsI7E1ZHfp/B4DF/6QBAUhmgkxc0H7Zss8=
github.com/cenkalti/backoff/v5 v5.0.2/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
@@ -15,23 +15,23 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/docker v28.3.2+incompatible h1:wn66NJ6pWB1vBZIilP8G3qQPqHy5XymfYn5vsqeA5oA=
github.com/docker/docker v28.3.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v28.3.3+incompatible h1:Dypm25kh4rmk49v1eiVbsAtpAsYURjYkaKubwuBdxEI=
github.com/docker/docker v28.3.3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw=
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU=
github.com/google/btree v1.1.2/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
@@ -76,76 +76,72 @@ github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/vishvananda/netlink v1.3.0 h1:X7l42GfcV4S6E4vHTsw48qbrV+9PVojNfIhZcwQdrZk=
github.com/vishvananda/netlink v1.3.0/go.mod h1:i6NetklAujEcC6fK0JPjT8qSwWyO0HLn4UKG+hGqeJs=
github.com/vishvananda/netns v0.0.4 h1:Oeaw1EM2JMxD51g9uhtC0D7erkIjgmj8+JZc26m1YX8=
github.com/vishvananda/netns v0.0.4/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 h1:Hf9xI/XLML9ElpiHVDNwvqI0hIFlzV8dgIr35kV1kRU=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0/go.mod h1:NfchwuyNoMcZ5MLHwPrODwUF1HWCXWrL31s8gSAdIKY=
go.opentelemetry.io/otel v1.37.0 h1:9zhNfelUvx0KBfu/gb+ZgeAfAgtWrfHJZcAqFC228wQ=
go.opentelemetry.io/otel v1.37.0/go.mod h1:ehE/umFRLnuLa/vSccNq9oS1ErUlkkK71gMcN34UG8I=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 h1:dNzwXjZKpMpE2JhmO+9HsPl42NIXFIFSUSSs0fiqra0=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0/go.mod h1:90PoxvaEB5n6AOdZvi+yWJQoE95U8Dhhw2bSyRqnTD0=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 h1:nRVXXvf78e00EwY6Wp0YII8ww2JVWshZ20HfTlE11AM=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0/go.mod h1:r49hO7CgrxY9Voaj3Xe8pANWtr0Oq916d0XAmOoCZAQ=
go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
go.opentelemetry.io/otel/sdk/metric v1.36.0 h1:r0ntwwGosWGaa0CrSt8cuNuTcccMXERFwHX4dThiPis=
go.opentelemetry.io/otel/sdk/metric v1.36.0/go.mod h1:qTNOhFDfKRwX0yXOqJYegL5WRaW376QbB7P4Pb0qva4=
go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
go.opentelemetry.io/otel/metric v1.37.0 h1:mvwbQS5m0tbmqML4NqK+e3aDiO02vsf/WgbsdpcPoZE=
go.opentelemetry.io/otel/metric v1.37.0/go.mod h1:04wGrZurHYKOc+RKeye86GwKiTb9FKm1WHtO+4EVr2E=
go.opentelemetry.io/otel/sdk v1.37.0 h1:ItB0QUqnjesGRvNcmAcU0LyvkVyGJ2xftD29bWdDvKI=
go.opentelemetry.io/otel/sdk v1.37.0/go.mod h1:VredYzxUvuo2q3WRcDnKDjbdvmO0sCzOvVAiY+yUkAg=
go.opentelemetry.io/otel/sdk/metric v1.37.0 h1:90lI228XrB9jCMuSdA0673aubgRobVZFhbjxHHspCPc=
go.opentelemetry.io/otel/sdk/metric v1.37.0/go.mod h1:cNen4ZWfiD37l5NhS+Keb5RXVWZWpRE+9WyVCpbo5ps=
go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mxVK7z4=
go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0=
go.opentelemetry.io/proto/otlp v1.6.0 h1:jQjP+AQyTf+Fe7OKj/MfkDrmK4MNVtw2NpXsf9fefDI=
go.opentelemetry.io/proto/otlp v1.6.0/go.mod h1:cicgGehlFuNdgZkcALOCh3VE6K/u2tAjzlRhDwmVpZc=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM=
golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U=
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa h1:t2QcU6V556bFjYgu4L6C+6VrCPyJZ+eyRsABUPs1mz4=
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa/go.mod h1:BHOTPb3L19zxehTsLoJXVaTktb06DFgmdW6Wb9s8jqk=
golang.org/x/crypto v0.40.0 h1:r4x+VvoG5Fm+eJcxMaY8CQM7Lb0l1lsmjGBQ6s8BfKM=
golang.org/x/crypto v0.40.0/go.mod h1:Qr1vMER5WyS2dfPHAlsOj01wgLbsyWtFn/aY+5+ZdxY=
golang.org/x/exp v0.0.0-20250718183923-645b1fa84792 h1:R9PFI6EUdfVKgwKjZef7QIwGcBKu86OEFpJ9nUEP2l4=
golang.org/x/exp v0.0.0-20250718183923-645b1fa84792/go.mod h1:A+z0yzpGtvnG90cToK5n2tu8UJVP2XUATh+r+sfOOOc=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.23.0 h1:Zb7khfcRGKk+kqfxFaP5tZqCnDZMjC5VtUBs87Hr6QM=
golang.org/x/mod v0.23.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=
golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs=
golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=
golang.org/x/time v0.7.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/text v0.27.0 h1:4fGWRpyh641NLlecmyl4LOe6yDdfaYNrGb2zdfo4JV4=
golang.org/x/text v0.27.0/go.mod h1:1D28KMCvyooCX9hBiosv5Tz/+YLxj0j7XhWjpSUF7CU=
golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.30.0 h1:BgcpHewrV5AUp2G9MebG4XPFI1E2W41zU1SaqVA9vJY=
golang.org/x/tools v0.30.0/go.mod h1:c347cR/OJfw5TI+GfX7RUPNMdDRRbjvYTS0jPyvsVtY=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -171,5 +167,5 @@ gotest.tools/v3 v3.4.0 h1:ZazjZUfuVeZGLAmlKKuyv3IKP5orXcwtOwDQH6YVr6o=
gotest.tools/v3 v3.4.0/go.mod h1:CtbdzLSsqVhDgMtKsx03ird5YTGB3ar27v0u/yKBW5g=
gvisor.dev/gvisor v0.0.0-20250503011706-39ed1f5ac29c h1:m/r7OM+Y2Ty1sgBQ7Qb27VgIMBW8ZZhT4gLnUyDIhzI=
gvisor.dev/gvisor v0.0.0-20250503011706-39ed1f5ac29c/go.mod h1:3r5CMtNQMKIvBlrmM9xWUNamjKBYPOWyXOjmg5Kts3g=
software.sslmate.com/src/go-pkcs12 v0.5.0 h1:EC6R394xgENTpZ4RltKydeDUjtlM5drOYIG9c6TVj2M=
software.sslmate.com/src/go-pkcs12 v0.5.0/go.mod h1:Qiz0EyvDRJjjxGyUQa2cCNZn/wMyzrRJ/qcDXOQazLI=
software.sslmate.com/src/go-pkcs12 v0.6.0 h1:f3sQittAeF+pao32Vb+mkli+ZyT+VwKaD014qFGq6oU=
software.sslmate.com/src/go-pkcs12 v0.6.0/go.mod h1:Qiz0EyvDRJjjxGyUQa2cCNZn/wMyzrRJ/qcDXOQazLI=

500
healthcheck/healthcheck.go Normal file
View File

@@ -0,0 +1,500 @@
package healthcheck
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"sync"
"time"
"github.com/fosrl/newt/logger"
)
// Health represents the health status of a target
type Health int
const (
StatusUnknown Health = iota
StatusHealthy
StatusUnhealthy
)
func (s Health) String() string {
switch s {
case StatusHealthy:
return "healthy"
case StatusUnhealthy:
return "unhealthy"
default:
return "unknown"
}
}
// Config holds the health check configuration for a target
type Config struct {
ID int `json:"id"`
Enabled bool `json:"hcEnabled"`
Path string `json:"hcPath"`
Scheme string `json:"hcScheme"`
Mode string `json:"hcMode"`
Hostname string `json:"hcHostname"`
Port int `json:"hcPort"`
Interval int `json:"hcInterval"` // in seconds
UnhealthyInterval int `json:"hcUnhealthyInterval"` // in seconds
Timeout int `json:"hcTimeout"` // in seconds
Headers map[string]string `json:"hcHeaders"`
Method string `json:"hcMethod"`
Status int `json:"hcStatus"` // HTTP status code
}
// Target represents a health check target with its current status
type Target struct {
Config Config `json:"config"`
Status Health `json:"status"`
LastCheck time.Time `json:"lastCheck"`
LastError string `json:"lastError,omitempty"`
CheckCount int `json:"checkCount"`
ticker *time.Ticker
ctx context.Context
cancel context.CancelFunc
}
// StatusChangeCallback is called when any target's status changes
type StatusChangeCallback func(targets map[int]*Target)
// Monitor manages health check targets and their monitoring
type Monitor struct {
targets map[int]*Target
mutex sync.RWMutex
callback StatusChangeCallback
client *http.Client
}
// NewMonitor creates a new health check monitor
func NewMonitor(callback StatusChangeCallback) *Monitor {
logger.Info("Creating new health check monitor")
return &Monitor{
targets: make(map[int]*Target),
callback: callback,
client: &http.Client{
Timeout: 30 * time.Second,
},
}
}
// parseHeaders parses the headers string into a map
func parseHeaders(headersStr string) map[string]string {
headers := make(map[string]string)
if headersStr == "" {
return headers
}
// Try to parse as JSON first
if err := json.Unmarshal([]byte(headersStr), &headers); err == nil {
return headers
}
// Fallback to simple key:value parsing
pairs := strings.Split(headersStr, ",")
for _, pair := range pairs {
kv := strings.SplitN(strings.TrimSpace(pair), ":", 2)
if len(kv) == 2 {
headers[strings.TrimSpace(kv[0])] = strings.TrimSpace(kv[1])
}
}
return headers
}
// AddTarget adds a new health check target
func (m *Monitor) AddTarget(config Config) error {
m.mutex.Lock()
defer m.mutex.Unlock()
logger.Info("Adding health check target: ID=%d, hostname=%s, port=%d, enabled=%t",
config.ID, config.Hostname, config.Port, config.Enabled)
return m.addTargetUnsafe(config)
}
// AddTargets adds multiple health check targets in bulk
func (m *Monitor) AddTargets(configs []Config) error {
m.mutex.Lock()
defer m.mutex.Unlock()
logger.Debug("Adding %d health check targets in bulk", len(configs))
for _, config := range configs {
if err := m.addTargetUnsafe(config); err != nil {
logger.Error("Failed to add target %d: %v", config.ID, err)
return fmt.Errorf("failed to add target %d: %v", config.ID, err)
}
logger.Debug("Successfully added target: ID=%d, hostname=%s", config.ID, config.Hostname)
}
// Don't notify callback immediately - let the initial health checks complete first
// The callback will be triggered when the first health check results are available
logger.Debug("Successfully added all %d health check targets", len(configs))
return nil
}
// addTargetUnsafe adds a target without acquiring the mutex (internal method)
func (m *Monitor) addTargetUnsafe(config Config) error {
// Set defaults
if config.Scheme == "" {
config.Scheme = "http"
}
if config.Mode == "" {
config.Mode = "http"
}
if config.Method == "" {
config.Method = "GET"
}
if config.Interval == 0 {
config.Interval = 30
}
if config.UnhealthyInterval == 0 {
config.UnhealthyInterval = 30
}
if config.Timeout == 0 {
config.Timeout = 5
}
logger.Debug("Target %d configuration: scheme=%s, method=%s, interval=%ds, timeout=%ds",
config.ID, config.Scheme, config.Method, config.Interval, config.Timeout)
// Parse headers if provided as string
if len(config.Headers) == 0 && config.Path != "" {
// This is a simplified header parsing - in real use you might want more robust parsing
config.Headers = make(map[string]string)
}
// Remove existing target if it exists
if existing, exists := m.targets[config.ID]; exists {
logger.Info("Replacing existing target with ID %d", config.ID)
existing.cancel()
}
// Create new target
ctx, cancel := context.WithCancel(context.Background())
target := &Target{
Config: config,
Status: StatusUnknown,
ctx: ctx,
cancel: cancel,
}
m.targets[config.ID] = target
// Start monitoring if enabled
if config.Enabled {
logger.Info("Starting monitoring for target %d (%s:%d)", config.ID, config.Hostname, config.Port)
go m.monitorTarget(target)
} else {
logger.Debug("Target %d added but monitoring is disabled", config.ID)
}
return nil
}
// RemoveTarget removes a health check target
func (m *Monitor) RemoveTarget(id int) error {
m.mutex.Lock()
defer m.mutex.Unlock()
target, exists := m.targets[id]
if !exists {
logger.Warn("Attempted to remove non-existent target with ID %d", id)
return fmt.Errorf("target with id %d not found", id)
}
logger.Info("Removing health check target: ID=%d", id)
target.cancel()
delete(m.targets, id)
// Notify callback of status change
if m.callback != nil {
go m.callback(m.GetTargets())
}
logger.Info("Successfully removed target %d", id)
return nil
}
// RemoveTargets removes multiple health check targets
func (m *Monitor) RemoveTargets(ids []int) error {
m.mutex.Lock()
defer m.mutex.Unlock()
logger.Info("Removing %d health check targets", len(ids))
var notFound []int
for _, id := range ids {
target, exists := m.targets[id]
if !exists {
notFound = append(notFound, id)
logger.Warn("Target with ID %d not found during bulk removal", id)
continue
}
logger.Debug("Removing target %d", id)
target.cancel()
delete(m.targets, id)
}
removedCount := len(ids) - len(notFound)
logger.Info("Successfully removed %d targets", removedCount)
// Notify callback of status change if any targets were removed
if len(notFound) != len(ids) && m.callback != nil {
go m.callback(m.GetTargets())
}
if len(notFound) > 0 {
logger.Error("Some targets not found during removal: %v", notFound)
return fmt.Errorf("targets not found: %v", notFound)
}
return nil
}
// RemoveTargetsByID is a convenience method that accepts either a single ID or multiple IDs
func (m *Monitor) RemoveTargetsByID(ids ...int) error {
return m.RemoveTargets(ids)
}
// GetTargets returns a copy of all targets
func (m *Monitor) GetTargets() map[int]*Target {
m.mutex.RLock()
defer m.mutex.RUnlock()
return m.getAllTargetsUnsafe()
}
// getAllTargetsUnsafe returns a copy of all targets without acquiring the mutex (internal method)
func (m *Monitor) getAllTargetsUnsafe() map[int]*Target {
targets := make(map[int]*Target)
for id, target := range m.targets {
// Create a copy to avoid race conditions
targetCopy := *target
targets[id] = &targetCopy
}
return targets
}
// getAllTargets returns a copy of all targets (deprecated, use GetTargets)
func (m *Monitor) getAllTargets() map[int]*Target {
return m.GetTargets()
}
// monitorTarget monitors a single target
func (m *Monitor) monitorTarget(target *Target) {
logger.Info("Starting health check monitoring for target %d (%s:%d)",
target.Config.ID, target.Config.Hostname, target.Config.Port)
// Initial check
oldStatus := target.Status
m.performHealthCheck(target)
// Notify callback after initial check if status changed or if it's the first check
if (oldStatus != target.Status || oldStatus == StatusUnknown) && m.callback != nil {
logger.Info("Target %d initial status: %s", target.Config.ID, target.Status.String())
go m.callback(m.GetTargets())
}
// Set up ticker based on current status
interval := time.Duration(target.Config.Interval) * time.Second
if target.Status == StatusUnhealthy {
interval = time.Duration(target.Config.UnhealthyInterval) * time.Second
}
logger.Debug("Target %d: initial check interval set to %v", target.Config.ID, interval)
target.ticker = time.NewTicker(interval)
defer target.ticker.Stop()
for {
select {
case <-target.ctx.Done():
logger.Info("Stopping health check monitoring for target %d", target.Config.ID)
return
case <-target.ticker.C:
oldStatus := target.Status
m.performHealthCheck(target)
// Update ticker interval if status changed
newInterval := time.Duration(target.Config.Interval) * time.Second
if target.Status == StatusUnhealthy {
newInterval = time.Duration(target.Config.UnhealthyInterval) * time.Second
}
if newInterval != interval {
logger.Debug("Target %d: updating check interval from %v to %v due to status change",
target.Config.ID, interval, newInterval)
target.ticker.Stop()
target.ticker = time.NewTicker(newInterval)
interval = newInterval
}
// Notify callback if status changed
if oldStatus != target.Status && m.callback != nil {
logger.Info("Target %d status changed: %s -> %s",
target.Config.ID, oldStatus.String(), target.Status.String())
go m.callback(m.GetTargets())
}
}
}
}
// performHealthCheck performs a health check on a target
func (m *Monitor) performHealthCheck(target *Target) {
target.CheckCount++
target.LastCheck = time.Now()
target.LastError = ""
// Build URL
url := fmt.Sprintf("%s://%s", target.Config.Scheme, target.Config.Hostname)
if target.Config.Port > 0 {
url = fmt.Sprintf("%s:%d", url, target.Config.Port)
}
if target.Config.Path != "" {
if !strings.HasPrefix(target.Config.Path, "/") {
url += "/"
}
url += target.Config.Path
}
logger.Debug("Target %d: performing health check %d to %s",
target.Config.ID, target.CheckCount, url)
// Create request
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(target.Config.Timeout)*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, target.Config.Method, url, nil)
if err != nil {
target.Status = StatusUnhealthy
target.LastError = fmt.Sprintf("failed to create request: %v", err)
logger.Warn("Target %d: failed to create request: %v", target.Config.ID, err)
return
}
// Add headers
for key, value := range target.Config.Headers {
req.Header.Set(key, value)
}
// Perform request
resp, err := m.client.Do(req)
if err != nil {
target.Status = StatusUnhealthy
target.LastError = fmt.Sprintf("request failed: %v", err)
logger.Warn("Target %d: health check failed: %v", target.Config.ID, err)
return
}
defer resp.Body.Close()
// Check response status
var expectedStatus int
if target.Config.Status > 0 {
expectedStatus = target.Config.Status
} else {
expectedStatus = 0 // Use range check for 200-299
}
if expectedStatus > 0 {
logger.Debug("Target %d: checking health status against expected code %d", target.Config.ID, expectedStatus)
// Check for specific status code
if resp.StatusCode == expectedStatus {
target.Status = StatusHealthy
logger.Debug("Target %d: health check passed (status: %d, expected: %d)", target.Config.ID, resp.StatusCode, expectedStatus)
} else {
target.Status = StatusUnhealthy
target.LastError = fmt.Sprintf("unexpected status code: %d (expected: %d)", resp.StatusCode, expectedStatus)
logger.Warn("Target %d: health check failed with status code %d (expected: %d)", target.Config.ID, resp.StatusCode, expectedStatus)
}
} else {
// Check for 2xx range
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
target.Status = StatusHealthy
logger.Debug("Target %d: health check passed (status: %d)", target.Config.ID, resp.StatusCode)
} else {
target.Status = StatusUnhealthy
target.LastError = fmt.Sprintf("unhealthy status code: %d", resp.StatusCode)
logger.Warn("Target %d: health check failed with status code %d", target.Config.ID, resp.StatusCode)
}
}
}
// Stop stops monitoring all targets
func (m *Monitor) Stop() {
m.mutex.Lock()
defer m.mutex.Unlock()
targetCount := len(m.targets)
logger.Info("Stopping health check monitor with %d targets", targetCount)
for id, target := range m.targets {
logger.Debug("Stopping monitoring for target %d", id)
target.cancel()
}
m.targets = make(map[int]*Target)
logger.Info("Health check monitor stopped")
}
// EnableTarget enables monitoring for a specific target
func (m *Monitor) EnableTarget(id int) error {
m.mutex.Lock()
defer m.mutex.Unlock()
target, exists := m.targets[id]
if !exists {
logger.Warn("Attempted to enable non-existent target with ID %d", id)
return fmt.Errorf("target with id %d not found", id)
}
if !target.Config.Enabled {
logger.Info("Enabling health check monitoring for target %d", id)
target.Config.Enabled = true
target.cancel() // Stop existing monitoring
ctx, cancel := context.WithCancel(context.Background())
target.ctx = ctx
target.cancel = cancel
go m.monitorTarget(target)
} else {
logger.Debug("Target %d is already enabled", id)
}
return nil
}
// DisableTarget disables monitoring for a specific target
func (m *Monitor) DisableTarget(id int) error {
m.mutex.Lock()
defer m.mutex.Unlock()
target, exists := m.targets[id]
if !exists {
logger.Warn("Attempted to disable non-existent target with ID %d", id)
return fmt.Errorf("target with id %d not found", id)
}
if target.Config.Enabled {
logger.Info("Disabling health check monitoring for target %d", id)
target.Config.Enabled = false
target.cancel()
target.Status = StatusUnknown
// Notify callback of status change
if m.callback != nil {
go m.callback(m.GetTargets())
}
} else {
logger.Debug("Target %d is already disabled", id)
}
return nil
}

View File

@@ -4,7 +4,8 @@ package main
import (
"fmt"
"strings"
"os"
"runtime"
"github.com/fosrl/newt/logger"
"github.com/fosrl/newt/proxy"
@@ -13,71 +14,61 @@ import (
"github.com/fosrl/newt/wgtester"
)
var wgService *wg.WireGuardService
var wgTesterServer *wgtester.Server
var wgServiceNative *wg.WireGuardService
func setupClients(client *websocket.Client) {
var host = endpoint
if strings.HasPrefix(host, "http://") {
host = strings.TrimPrefix(host, "http://")
} else if strings.HasPrefix(host, "https://") {
host = strings.TrimPrefix(host, "https://")
func setupClientsNative(client *websocket.Client, host string) {
if runtime.GOOS != "linux" {
logger.Fatal("Tunnel management is only supported on Linux right now!")
os.Exit(1)
}
host = strings.TrimSuffix(host, "/")
// make sure we are sudo
if os.Geteuid() != 0 {
logger.Fatal("You must run this program as root to manage tunnels on Linux.")
os.Exit(1)
}
// Create WireGuard service
wgService, err = wg.NewWireGuardService(interfaceName, mtuInt, generateAndSaveKeyTo, host, id, client)
wgServiceNative, err = wg.NewWireGuardService(interfaceName, mtuInt, generateAndSaveKeyTo, host, id, client)
if err != nil {
logger.Fatal("Failed to create WireGuard service: %v", err)
}
defer wgService.Close(rm)
wgTesterServer = wgtester.NewServer("0.0.0.0", wgService.Port, id) // TODO: maybe make this the same ip of the wg server?
wgTesterServer = wgtester.NewServer("0.0.0.0", wgServiceNative.Port, id) // TODO: maybe make this the same ip of the wg server?
err := wgTesterServer.Start()
if err != nil {
logger.Error("Failed to start WireGuard tester server: %v", err)
} else {
// Make sure to stop the server on exit
defer wgTesterServer.Stop()
}
client.OnTokenUpdate(func(token string) {
wgService.SetToken(token)
wgServiceNative.SetToken(token)
})
}
func closeClients() {
if wgService != nil {
wgService.Close(rm)
wgService = nil
}
if wgTesterServer != nil {
wgTesterServer.Stop()
wgTesterServer = nil
func closeWgServiceNative() {
if wgServiceNative != nil {
wgServiceNative.Close(!keepInterface)
wgServiceNative = nil
}
}
func clientsHandleNewtConnection(publicKey string) {
if wgService == nil {
return
func clientsOnConnectNative() {
if wgServiceNative != nil {
wgServiceNative.LoadRemoteConfig()
}
wgService.SetServerPubKey(publicKey)
}
func clientsOnConnect() {
if wgService == nil {
return
func clientsHandleNewtConnectionNative(publicKey, endpoint string) {
if wgServiceNative != nil {
wgServiceNative.StartHolepunch(publicKey, endpoint)
}
wgService.LoadRemoteConfig()
}
func clientsAddProxyTarget(pm *proxy.ProxyManager, tunnelIp string) {
if wgService == nil {
return
}
func clientsAddProxyTargetNative(pm *proxy.ProxyManager, tunnelIp string) {
// add a udp proxy for localost and the wgService port
// TODO: make sure this port is not used in a target
pm.AddTarget("udp", tunnelIp, int(wgService.Port), fmt.Sprintf("127.0.0.1:%d", wgService.Port))
if wgServiceNative != nil {
pm.AddTarget("udp", tunnelIp, int(wgServiceNative.Port), fmt.Sprintf("127.0.0.1:%d", wgServiceNative.Port))
}
}

View File

@@ -2,6 +2,7 @@ package logger
import (
"fmt"
"io"
"log"
"os"
"sync"
@@ -48,6 +49,11 @@ func (l *Logger) SetLevel(level LogLevel) {
l.level = level
}
// SetOutput sets the output destination for the logger
func (l *Logger) SetOutput(w io.Writer) {
l.logger.SetOutput(w)
}
// log handles the actual logging
func (l *Logger) log(level LogLevel, format string, args ...interface{}) {
if level < l.level {
@@ -120,3 +126,8 @@ func Error(format string, args ...interface{}) {
func Fatal(format string, args ...interface{}) {
GetLogger().Fatal(format, args...)
}
// SetOutput sets the output destination for the default logger
func SetOutput(w io.Writer) {
GetLogger().SetOutput(w)
}

483
main.go
View File

@@ -9,13 +9,14 @@ import (
"net/netip"
"os"
"os/signal"
"runtime"
"path/filepath"
"strconv"
"strings"
"syscall"
"time"
"github.com/fosrl/newt/docker"
"github.com/fosrl/newt/healthcheck"
"github.com/fosrl/newt/logger"
"github.com/fosrl/newt/proxy"
"github.com/fosrl/newt/updates"
@@ -29,11 +30,12 @@ import (
)
type WgData struct {
Endpoint string `json:"endpoint"`
PublicKey string `json:"publicKey"`
ServerIP string `json:"serverIP"`
TunnelIP string `json:"tunnelIP"`
Targets TargetsByType `json:"targets"`
Endpoint string `json:"endpoint"`
PublicKey string `json:"publicKey"`
ServerIP string `json:"serverIP"`
TunnelIP string `json:"tunnelIP"`
Targets TargetsByType `json:"targets"`
HealthCheckTargets []healthcheck.Config `json:"healthCheckTargets"`
}
type TargetsByType struct {
@@ -49,6 +51,10 @@ type ExitNodeData struct {
ExitNodes []ExitNode `json:"exitNodes"`
}
type SSHPublicKeyData struct {
PublicKey string `json:"publicKey"`
}
// ExitNode represents an exit node with an ID, endpoint, and weight.
type ExitNode struct {
ID int `json:"exitNodeId"`
@@ -80,7 +86,7 @@ var (
logLevel string
interfaceName string
generateAndSaveKeyTo string
rm bool
keepInterface bool
acceptClients bool
updownScript string
tlsPrivateKey string
@@ -93,6 +99,10 @@ var (
pingStopChan chan struct{}
stopFunc func()
healthFile string
useNativeInterface bool
authorizedKeysFile string
preferEndpoint string
healthMonitor *healthcheck.Monitor
)
func main() {
@@ -104,16 +114,24 @@ func main() {
dns = os.Getenv("DNS")
logLevel = os.Getenv("LOG_LEVEL")
updownScript = os.Getenv("UPDOWN_SCRIPT")
// interfaceName = os.Getenv("INTERFACE")
// generateAndSaveKeyTo = os.Getenv("GENERATE_AND_SAVE_KEY_TO")
// rm = os.Getenv("RM") == "true"
// acceptClients = os.Getenv("ACCEPT_CLIENTS") == "true"
interfaceName = os.Getenv("INTERFACE")
generateAndSaveKeyTo = os.Getenv("GENERATE_AND_SAVE_KEY_TO")
keepInterfaceEnv := os.Getenv("KEEP_INTERFACE")
acceptClientsEnv := os.Getenv("ACCEPT_CLIENTS")
useNativeInterfaceEnv := os.Getenv("USE_NATIVE_INTERFACE")
keepInterface = keepInterfaceEnv == "true"
acceptClients = acceptClientsEnv == "true"
useNativeInterface = useNativeInterfaceEnv == "true"
tlsPrivateKey = os.Getenv("TLS_CLIENT_CERT")
dockerSocket = os.Getenv("DOCKER_SOCKET")
pingIntervalStr := os.Getenv("PING_INTERVAL")
pingTimeoutStr := os.Getenv("PING_TIMEOUT")
dockerEnforceNetworkValidation = os.Getenv("DOCKER_ENFORCE_NETWORK_VALIDATION")
healthFile = os.Getenv("HEALTH_FILE")
// authorizedKeysFile = os.Getenv("AUTHORIZED_KEYS_FILE")
authorizedKeysFile = ""
if endpoint == "" {
flag.StringVar(&endpoint, "endpoint", "", "Endpoint of your pangolin server")
@@ -136,14 +154,21 @@ func main() {
if updownScript == "" {
flag.StringVar(&updownScript, "updown", "", "Path to updown script to be called when targets are added or removed")
}
// if interfaceName == "" {
// flag.StringVar(&interfaceName, "interface", "wg1", "Name of the WireGuard interface")
// }
// if generateAndSaveKeyTo == "" {
// flag.StringVar(&generateAndSaveKeyTo, "generateAndSaveKeyTo", "/tmp/newtkey", "Path to save generated private key")
// }
// flag.BoolVar(&rm, "rm", false, "Remove the WireGuard interface")
// flag.BoolVar(&acceptClients, "accept-clients", false, "Accept clients on the WireGuard interface")
if interfaceName == "" {
flag.StringVar(&interfaceName, "interface", "newt", "Name of the WireGuard interface")
}
if generateAndSaveKeyTo == "" {
flag.StringVar(&generateAndSaveKeyTo, "generateAndSaveKeyTo", "", "Path to save generated private key")
}
if keepInterfaceEnv == "" {
flag.BoolVar(&keepInterface, "keep-interface", false, "Keep the WireGuard interface")
}
if useNativeInterfaceEnv == "" {
flag.BoolVar(&useNativeInterface, "native", false, "Use native WireGuard interface (requires WireGuard kernel module) and linux")
}
if acceptClientsEnv == "" {
flag.BoolVar(&acceptClients, "accept-clients", false, "Accept clients on the WireGuard interface")
}
if tlsPrivateKey == "" {
flag.StringVar(&tlsPrivateKey, "tls-client-cert", "", "Path to client certificate used for mTLS")
}
@@ -154,23 +179,33 @@ func main() {
flag.StringVar(&pingIntervalStr, "ping-interval", "3s", "Interval for pinging the server (default 3s)")
}
if pingTimeoutStr == "" {
flag.StringVar(&pingTimeoutStr, "ping-timeout", "3s", " Timeout for each ping (default 3s)")
flag.StringVar(&pingTimeoutStr, "ping-timeout", "5s", " Timeout for each ping (default 5s)")
}
// load the prefer endpoint just as a flag
flag.StringVar(&preferEndpoint, "prefer-endpoint", "", "Prefer this endpoint for the connection (if set, will override the endpoint from the server)")
// if authorizedKeysFile == "" {
// flag.StringVar(&authorizedKeysFile, "authorized-keys-file", "~/.ssh/authorized_keys", "Path to authorized keys file (if unset, no keys will be authorized)")
// }
if pingIntervalStr != "" {
pingInterval, err = time.ParseDuration(pingIntervalStr)
if err != nil {
fmt.Printf("Invalid PING_INTERVAL value: %s, using default 1 second\n", pingIntervalStr)
fmt.Printf("Invalid PING_INTERVAL value: %s, using default 3 seconds\n", pingIntervalStr)
pingInterval = 3 * time.Second
}
} else {
pingInterval = 3 * time.Second
}
if pingTimeoutStr != "" {
pingTimeout, err = time.ParseDuration(pingTimeoutStr)
if err != nil {
fmt.Printf("Invalid PING_TIMEOUT value: %s, using default 2 seconds\n", pingTimeoutStr)
pingTimeout = 3 * time.Second
fmt.Printf("Invalid PING_TIMEOUT value: %s, using default 5 seconds\n", pingTimeoutStr)
pingTimeout = 5 * time.Second
}
} else {
pingTimeout = 5 * time.Second
}
if dockerEnforceNetworkValidation == "" {
@@ -224,6 +259,7 @@ func main() {
}
// Create a new client
client, err := websocket.NewClient(
"newt",
id, // CLI arg takes precedence
secret, // CLI arg takes precedence
endpoint,
@@ -234,6 +270,8 @@ func main() {
if err != nil {
logger.Fatal("Failed to create client: %v", err)
}
endpoint = client.GetConfig().Endpoint // Update endpoint from config
id = client.GetConfig().ID // Update ID from config
// output env var values if set
logger.Debug("Endpoint: %v", endpoint)
@@ -262,15 +300,36 @@ func main() {
var wgData WgData
if acceptClients {
// make sure we are running on linux
if runtime.GOOS != "linux" {
logger.Fatal("Tunnel management is only supported on Linux right now!")
os.Exit(1)
}
setupClients(client)
}
// Initialize health check monitor with status change callback
healthMonitor = healthcheck.NewMonitor(func(targets map[int]*healthcheck.Target) {
logger.Debug("Health check status update for %d targets", len(targets))
// Send health status update to the server
healthStatuses := make(map[int]interface{})
for id, target := range targets {
healthStatuses[id] = map[string]interface{}{
"status": target.Status.String(),
"lastCheck": target.LastCheck.Format(time.RFC3339),
"checkCount": target.CheckCount,
"lastError": target.LastError,
"config": target.Config,
}
}
// print the status of the targets
logger.Debug("Health check status: %+v", healthStatuses)
err := client.SendMessage("newt/healthcheck/status", map[string]interface{}{
"targets": healthStatuses,
})
if err != nil {
logger.Error("Failed to send health check status update: %v", err)
}
})
var pingWithRetryStopChan chan struct{}
closeWgTunnel := func() {
@@ -318,6 +377,9 @@ func main() {
connected = false
}
// print out the data
logger.Debug("Received registration message data: %+v", msg.Data)
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Info("Error marshaling data: %v", err)
@@ -329,8 +391,6 @@ func main() {
return
}
clientsHandleNewtConnection(wgData.PublicKey)
logger.Debug("Received: %+v", msg)
tun, tnet, err = netstack.CreateNetTUN(
[]netip.Addr{netip.MustParseAddr(wgData.TunnelIP)},
@@ -340,6 +400,8 @@ func main() {
logger.Error("Failed to create TUN device: %v", err)
}
setDownstreamTNetstack(tnet)
// Create WireGuard device
dev = device.NewDevice(tun, conn.NewDefaultBind(), device.NewLogger(
mapToWireGuardLogLevel(loggerLevel),
@@ -360,6 +422,8 @@ func main() {
return
}
clientsHandleNewtConnection(wgData.PublicKey, endpoint)
// Configure WireGuard
config := fmt.Sprintf(`private_key=%s
public_key=%s
@@ -386,6 +450,15 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
close(pingWithRetryStopChan)
pingWithRetryStopChan = nil
}
// Use reliable ping for initial connection test
logger.Debug("Testing initial connection with reliable ping...")
_, err = reliablePing(tnet, wgData.ServerIP, pingTimeout, 5)
if err != nil {
logger.Warn("Initial reliable ping failed, but continuing: %v", err)
} else {
logger.Info("Initial connection test successful")
}
pingWithRetryStopChan, _ = pingWithRetry(tnet, wgData.ServerIP, pingTimeout)
// Always mark as connected and start the proxy manager regardless of initial ping result
@@ -403,14 +476,28 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
// add the targets if there are any
if len(wgData.Targets.TCP) > 0 {
updateTargets(pm, "add", wgData.TunnelIP, "tcp", TargetData{Targets: wgData.Targets.TCP})
// Also update wgnetstack proxy manager
// if wgService != nil {
// updateTargets(wgService.GetProxyManager(), "add", wgData.TunnelIP, "tcp", TargetData{Targets: wgData.Targets.TCP})
// }
}
if len(wgData.Targets.UDP) > 0 {
updateTargets(pm, "add", wgData.TunnelIP, "udp", TargetData{Targets: wgData.Targets.UDP})
// Also update wgnetstack proxy manager
// if wgService != nil {
// updateTargets(wgService.GetProxyManager(), "add", wgData.TunnelIP, "udp", TargetData{Targets: wgData.Targets.UDP})
// }
}
clientsAddProxyTarget(pm, wgData.TunnelIP)
if err := healthMonitor.AddTargets(wgData.HealthCheckTargets); err != nil {
logger.Error("Failed to bulk add health check targets: %v", err)
} else {
logger.Info("Successfully added %d health check targets", len(wgData.HealthCheckTargets))
}
err = pm.Start()
if err != nil {
logger.Error("Failed to start proxy manager: %v", err)
@@ -443,6 +530,11 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
// Close the WireGuard device and TUN
closeWgTunnel()
if stopFunc != nil {
stopFunc() // stop the ws from sending more requests
stopFunc = nil // reset stopFunc to nil to avoid double stopping
}
// Mark as disconnected
connected = false
@@ -475,6 +567,42 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
return
}
// If there is just one exit node, we can skip pinging it and use it directly
if len(exitNodes) == 1 || preferEndpoint != "" {
logger.Debug("Only one exit node available, using it directly: %s", exitNodes[0].Endpoint)
// if the preferEndpoint is set, we will use it instead of the exit node endpoint. first you need to find the exit node with that endpoint in the list and send that one
if preferEndpoint != "" {
for _, node := range exitNodes {
if node.Endpoint == preferEndpoint {
exitNodes[0] = node
break
}
}
}
// Prepare data to send to the cloud for selection
pingResults := []ExitNodePingResult{
{
ExitNodeID: exitNodes[0].ID,
LatencyMs: 0, // No ping latency since we are using it directly
Weight: exitNodes[0].Weight,
Error: "",
Name: exitNodes[0].Name,
Endpoint: exitNodes[0].Endpoint,
WasPreviouslyConnected: exitNodes[0].WasPreviouslyConnected,
},
}
stopFunc = client.SendMessageInterval("newt/wg/register", map[string]interface{}{
"publicKey": publicKey.String(),
"pingResults": pingResults,
"newtVersion": newtVersion,
}, 1*time.Second)
return
}
type nodeResult struct {
Node ExitNode
Latency time.Duration
@@ -484,11 +612,6 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
results := make([]nodeResult, len(exitNodes))
const pingAttempts = 3
for i, node := range exitNodes {
if connected && node.WasPreviouslyConnected {
logger.Info("Skipping ping for previously connected exit node so we pick another %d (%s)", node.ID, node.Endpoint)
continue
}
var totalLatency time.Duration
var lastErr error
successes := 0
@@ -544,6 +667,35 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
})
}
// If we were previously connected and there is at least one other good node,
// exclude the previously connected node from pingResults sent to the cloud so we don't try to reconnect to it
// This is to avoid issues where the previously connected node might be down or unreachable
if connected {
var filteredPingResults []ExitNodePingResult
previouslyConnectedNodeIdx := -1
for i, res := range pingResults {
if res.WasPreviouslyConnected {
previouslyConnectedNodeIdx = i
}
}
// Count good nodes (latency > 0, no error, not previously connected)
goodNodeCount := 0
for i, res := range pingResults {
if i != previouslyConnectedNodeIdx && res.LatencyMs > 0 && res.Error == "" {
goodNodeCount++
}
}
if previouslyConnectedNodeIdx != -1 && goodNodeCount > 0 {
for i, res := range pingResults {
if i != previouslyConnectedNodeIdx {
filteredPingResults = append(filteredPingResults, res)
}
}
pingResults = filteredPingResults
logger.Info("Excluding previously connected exit node from ping results due to other available nodes")
}
}
// Send the ping results to the cloud for selection
stopFunc = client.SendMessageInterval("newt/wg/register", map[string]interface{}{
"publicKey": publicKey.String(),
@@ -571,6 +723,11 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
if len(targetData.Targets) > 0 {
updateTargets(pm, "add", wgData.TunnelIP, "tcp", targetData)
// Also update wgnetstack proxy manager
// if wgService != nil && wgService.GetNetstackNet() != nil && wgService.GetProxyManager() != nil {
// updateTargets(wgService.GetProxyManager(), "add", wgData.TunnelIP, "tcp", targetData)
// }
}
})
@@ -591,6 +748,11 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
if len(targetData.Targets) > 0 {
updateTargets(pm, "add", wgData.TunnelIP, "udp", targetData)
// Also update wgnetstack proxy manager
// if wgService != nil && wgService.GetNetstackNet() != nil && wgService.GetProxyManager() != nil {
// updateTargets(wgService.GetProxyManager(), "add", wgData.TunnelIP, "udp", targetData)
// }
}
})
@@ -611,6 +773,11 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
if len(targetData.Targets) > 0 {
updateTargets(pm, "remove", wgData.TunnelIP, "udp", targetData)
// Also update wgnetstack proxy manager
// if wgService != nil && wgService.GetNetstackNet() != nil && wgService.GetProxyManager() != nil {
// updateTargets(wgService.GetProxyManager(), "remove", wgData.TunnelIP, "udp", targetData)
// }
}
})
@@ -631,15 +798,20 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
if len(targetData.Targets) > 0 {
updateTargets(pm, "remove", wgData.TunnelIP, "tcp", targetData)
// Also update wgnetstack proxy manager
// if wgService != nil && wgService.GetNetstackNet() != nil && wgService.GetProxyManager() != nil {
// updateTargets(wgService.GetProxyManager(), "remove", wgData.TunnelIP, "tcp", targetData)
// }
}
})
// Register handler for Docker socket check
client.RegisterHandler("newt/socket/check", func(msg websocket.WSMessage) {
logger.Info("Received Docker socket check request")
logger.Debug("Received Docker socket check request")
if dockerSocket == "" {
logger.Info("Docker socket path is not set")
logger.Debug("Docker socket path is not set")
err := client.SendMessage("newt/socket/status", map[string]interface{}{
"available": false,
"socketPath": dockerSocket,
@@ -667,10 +839,10 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
// Register handler for Docker container listing
client.RegisterHandler("newt/socket/fetch", func(msg websocket.WSMessage) {
logger.Info("Received Docker container fetch request")
logger.Debug("Received Docker container fetch request")
if dockerSocket == "" {
logger.Info("Docker socket path is not set")
logger.Debug("Docker socket path is not set")
return
}
@@ -696,6 +868,226 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
}
})
// EXPERIMENTAL: WHAT SHOULD WE DO ABOUT SECURITY?
client.RegisterHandler("newt/send/ssh/publicKey", func(msg websocket.WSMessage) {
logger.Debug("Received SSH public key request")
var sshPublicKeyData SSHPublicKeyData
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Info("Error marshaling data: %v", err)
return
}
if err := json.Unmarshal(jsonData, &sshPublicKeyData); err != nil {
logger.Info("Error unmarshaling SSH public key data: %v", err)
return
}
sshPublicKey := sshPublicKeyData.PublicKey
if authorizedKeysFile == "" {
logger.Debug("No authorized keys file set, skipping public key response")
return
}
// Expand tilde to home directory if present
expandedPath := authorizedKeysFile
if strings.HasPrefix(authorizedKeysFile, "~/") {
homeDir, err := os.UserHomeDir()
if err != nil {
logger.Error("Failed to get user home directory: %v", err)
return
}
expandedPath = filepath.Join(homeDir, authorizedKeysFile[2:])
}
// if it is set but the file does not exist, create it
if _, err := os.Stat(expandedPath); os.IsNotExist(err) {
logger.Debug("Authorized keys file does not exist, creating it: %s", expandedPath)
if err := os.MkdirAll(filepath.Dir(expandedPath), 0755); err != nil {
logger.Error("Failed to create directory for authorized keys file: %v", err)
return
}
if _, err := os.Create(expandedPath); err != nil {
logger.Error("Failed to create authorized keys file: %v", err)
return
}
}
// Check if the public key already exists in the file
fileContent, err := os.ReadFile(expandedPath)
if err != nil {
logger.Error("Failed to read authorized keys file: %v", err)
return
}
// Check if the key already exists (trim whitespace for comparison)
existingKeys := strings.Split(string(fileContent), "\n")
keyAlreadyExists := false
trimmedNewKey := strings.TrimSpace(sshPublicKey)
for _, existingKey := range existingKeys {
if strings.TrimSpace(existingKey) == trimmedNewKey && trimmedNewKey != "" {
keyAlreadyExists = true
break
}
}
if keyAlreadyExists {
logger.Info("SSH public key already exists in authorized keys file, skipping")
return
}
// append the public key to the authorized keys file
logger.Debug("Appending public key to authorized keys file: %s", sshPublicKey)
file, err := os.OpenFile(expandedPath, os.O_APPEND|os.O_WRONLY, 0644)
if err != nil {
logger.Error("Failed to open authorized keys file: %v", err)
return
}
defer file.Close()
if _, err := file.WriteString(sshPublicKey + "\n"); err != nil {
logger.Error("Failed to write public key to authorized keys file: %v", err)
return
}
logger.Info("SSH public key appended to authorized keys file")
})
// Register handler for adding health check targets
client.RegisterHandler("newt/healthcheck/add", func(msg websocket.WSMessage) {
logger.Debug("Received health check add request: %+v", msg)
type HealthCheckConfig struct {
Targets []healthcheck.Config `json:"targets"`
}
var config HealthCheckConfig
// add a bunch of targets at once
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Error("Error marshaling health check data: %v", err)
return
}
if err := json.Unmarshal(jsonData, &config); err != nil {
logger.Error("Error unmarshaling health check config: %v", err)
return
}
if err := healthMonitor.AddTargets(config.Targets); err != nil {
logger.Error("Failed to add health check targets: %v", err)
} else {
logger.Info("Added %d health check targets", len(config.Targets))
}
logger.Debug("Health check targets added: %+v", config.Targets)
})
// Register handler for removing health check targets
client.RegisterHandler("newt/healthcheck/remove", func(msg websocket.WSMessage) {
logger.Debug("Received health check remove request: %+v", msg)
type HealthCheckConfig struct {
IDs []int `json:"ids"`
}
var requestData HealthCheckConfig
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Error("Error marshaling health check remove data: %v", err)
return
}
if err := json.Unmarshal(jsonData, &requestData); err != nil {
logger.Error("Error unmarshaling health check remove request: %v", err)
return
}
// Multiple target removal
if err := healthMonitor.RemoveTargets(requestData.IDs); err != nil {
logger.Error("Failed to remove health check targets %v: %v", requestData.IDs, err)
} else {
logger.Info("Removed %d health check targets: %v", len(requestData.IDs), requestData.IDs)
}
})
// Register handler for enabling health check targets
client.RegisterHandler("newt/healthcheck/enable", func(msg websocket.WSMessage) {
logger.Debug("Received health check enable request: %+v", msg)
var requestData struct {
ID int `json:"id"`
}
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Error("Error marshaling health check enable data: %v", err)
return
}
if err := json.Unmarshal(jsonData, &requestData); err != nil {
logger.Error("Error unmarshaling health check enable request: %v", err)
return
}
if err := healthMonitor.EnableTarget(requestData.ID); err != nil {
logger.Error("Failed to enable health check target %s: %v", requestData.ID, err)
} else {
logger.Info("Enabled health check target: %s", requestData.ID)
}
})
// Register handler for disabling health check targets
client.RegisterHandler("newt/healthcheck/disable", func(msg websocket.WSMessage) {
logger.Debug("Received health check disable request: %+v", msg)
var requestData struct {
ID int `json:"id"`
}
jsonData, err := json.Marshal(msg.Data)
if err != nil {
logger.Error("Error marshaling health check disable data: %v", err)
return
}
if err := json.Unmarshal(jsonData, &requestData); err != nil {
logger.Error("Error unmarshaling health check disable request: %v", err)
return
}
if err := healthMonitor.DisableTarget(requestData.ID); err != nil {
logger.Error("Failed to disable health check target %s: %v", requestData.ID, err)
} else {
logger.Info("Disabled health check target: %s", requestData.ID)
}
})
// Register handler for getting health check status
client.RegisterHandler("newt/healthcheck/status/request", func(msg websocket.WSMessage) {
logger.Debug("Received health check status request")
targets := healthMonitor.GetTargets()
healthStatuses := make(map[int]interface{})
for id, target := range targets {
healthStatuses[id] = map[string]interface{}{
"status": target.Status.String(),
"lastCheck": target.LastCheck.Format(time.RFC3339),
"checkCount": target.CheckCount,
"lastError": target.LastError,
"config": target.Config,
}
}
err := client.SendMessage("newt/healthcheck/status", map[string]interface{}{
"targets": healthStatuses,
})
if err != nil {
logger.Error("Failed to send health check status response: %v", err)
}
})
client.OnConnect(func() error {
publicKey = privateKey.PublicKey()
logger.Debug("Public key: %s", publicKey)
@@ -734,10 +1126,17 @@ persistent_keepalive_interval=5`, fixKey(privateKey.String()), fixKey(wgData.Pub
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
<-sigCh
dev.Close()
// Close clients first (including WGTester)
closeClients()
if healthMonitor != nil {
healthMonitor.Stop()
}
if dev != nil {
dev.Close()
}
if pm != nil {
pm.Stop()
}

View File

@@ -41,6 +41,23 @@ func NewProxyManager(tnet *netstack.Net) *ProxyManager {
}
}
// init function without tnet
func NewProxyManagerWithoutTNet() *ProxyManager {
return &ProxyManager{
tcpTargets: make(map[string]map[int]string),
udpTargets: make(map[string]map[int]string),
listeners: make([]*gonet.TCPListener, 0),
udpConns: make([]*gonet.UDPConn, 0),
}
}
// Function to add tnet to existing ProxyManager
func (pm *ProxyManager) SetTNet(tnet *netstack.Net) {
pm.mutex.Lock()
defer pm.mutex.Unlock()
pm.tnet = tnet
}
// AddTarget adds as new target for proxying
func (pm *ProxyManager) AddTarget(proto, listenIP string, port int, targetAddr string) error {
pm.mutex.Lock()
@@ -174,13 +191,13 @@ func (pm *ProxyManager) Stop() error {
pm.udpConns = append(pm.udpConns[:i], pm.udpConns[i+1:]...)
}
// Clear the target maps
for k := range pm.tcpTargets {
delete(pm.tcpTargets, k)
}
for k := range pm.udpTargets {
delete(pm.udpTargets, k)
}
// // Clear the target maps
// for k := range pm.tcpTargets {
// delete(pm.tcpTargets, k)
// }
// for k := range pm.udpTargets {
// delete(pm.udpTargets, k)
// }
// Give active connections a chance to close gracefully
time.Sleep(100 * time.Millisecond)
@@ -279,6 +296,13 @@ func (pm *ProxyManager) handleUDPProxy(conn *gonet.UDPConn, targetAddr string) {
n, remoteAddr, err := conn.ReadFrom(buffer)
if err != nil {
if !pm.running {
// Clean up all connections when stopping
clientsMutex.Lock()
for _, targetConn := range clientConns {
targetConn.Close()
}
clientConns = nil
clientsMutex.Unlock()
return
}
@@ -323,22 +347,32 @@ func (pm *ProxyManager) handleUDPProxy(conn *gonet.UDPConn, targetAddr string) {
clientConns[clientKey] = targetConn
clientsMutex.Unlock()
go func() {
go func(clientKey string, targetConn *net.UDPConn, remoteAddr net.Addr) {
defer func() {
// Always clean up when this goroutine exits
clientsMutex.Lock()
if storedConn, exists := clientConns[clientKey]; exists && storedConn == targetConn {
delete(clientConns, clientKey)
targetConn.Close()
}
clientsMutex.Unlock()
}()
buffer := make([]byte, 65507)
for {
n, _, err := targetConn.ReadFromUDP(buffer)
if err != nil {
logger.Error("Error reading from target: %v", err)
return
return // defer will handle cleanup
}
_, err = conn.WriteTo(buffer[:n], remoteAddr)
if err != nil {
logger.Error("Error writing to client: %v", err)
return
return // defer will handle cleanup
}
}
}()
}(clientKey, targetConn, remoteAddr)
}
_, err = targetConn.Write(buffer[:n])
@@ -351,3 +385,23 @@ func (pm *ProxyManager) handleUDPProxy(conn *gonet.UDPConn, targetAddr string) {
}
}
}
// write a function to print out the current targets in the ProxyManager
func (pm *ProxyManager) PrintTargets() {
pm.mutex.RLock()
defer pm.mutex.RUnlock()
logger.Info("Current TCP Targets:")
for listenIP, targets := range pm.tcpTargets {
for port, targetAddr := range targets {
logger.Info("TCP %s:%d -> %s", listenIP, port, targetAddr)
}
}
logger.Info("Current UDP Targets:")
for listenIP, targets := range pm.udpTargets {
for port, targetAddr := range targets {
logger.Info("UDP %s:%d -> %s", listenIP, port, targetAddr)
}
}
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 774 KiB

After

Width:  |  Height:  |  Size: 93 KiB

18
stub.go
View File

@@ -7,26 +7,26 @@ import (
"github.com/fosrl/newt/websocket"
)
func setupClients(client *websocket.Client) {
func setupClientsNative(client *websocket.Client, host string) {
return // This function is not implemented for non-Linux systems.
}
func closeClients() {
// This function is not implemented for non-Linux systems.
func closeWgServiceNative() {
// No-op for non-Linux systems
return
}
func clientsHandleNewtConnection(publicKey string) {
// This function is not implemented for non-Linux systems.
func clientsOnConnectNative() {
// No-op for non-Linux systems
return
}
func clientsOnConnect() {
// This function is not implemented for non-Linux systems.
func clientsHandleNewtConnectionNative(publicKey, endpoint string) {
// No-op for non-Linux systems
return
}
func clientsAddProxyTarget(pm *proxy.ProxyManager, tunnelIp string) {
// This function is not implemented for non-Linux systems.
func clientsAddProxyTargetNative(pm *proxy.ProxyManager, tunnelIp string) {
// No-op for non-Linux systems
return
}

120
util.go
View File

@@ -45,9 +45,17 @@ func ping(tnet *netstack.Net, dst string, timeout time.Duration) (time.Duration,
}
defer socket.Close()
// Set socket buffer sizes to handle high bandwidth scenarios
if tcpConn, ok := socket.(interface{ SetReadBuffer(int) error }); ok {
tcpConn.SetReadBuffer(64 * 1024)
}
if tcpConn, ok := socket.(interface{ SetWriteBuffer(int) error }); ok {
tcpConn.SetWriteBuffer(64 * 1024)
}
requestPing := icmp.Echo{
Seq: rand.Intn(1 << 16),
Data: []byte("f"),
Data: []byte("newtping"),
}
icmpBytes, err := (&icmp.Message{Type: ipv4.ICMPTypeEcho, Code: 0, Body: &requestPing}).Marshal(nil)
@@ -65,12 +73,14 @@ func ping(tnet *netstack.Net, dst string, timeout time.Duration) (time.Duration,
return 0, fmt.Errorf("failed to write ICMP packet: %w", err)
}
n, err := socket.Read(icmpBytes[:])
// Use larger buffer for reading to handle potential network congestion
readBuffer := make([]byte, 1500)
n, err := socket.Read(readBuffer)
if err != nil {
return 0, fmt.Errorf("failed to read ICMP packet: %w", err)
}
replyPacket, err := icmp.ParseMessage(1, icmpBytes[:n])
replyPacket, err := icmp.ParseMessage(1, readBuffer[:n])
if err != nil {
return 0, fmt.Errorf("failed to parse ICMP packet: %w", err)
}
@@ -92,6 +102,51 @@ func ping(tnet *netstack.Net, dst string, timeout time.Duration) (time.Duration,
return latency, nil
}
// reliablePing performs multiple ping attempts with adaptive timeout
func reliablePing(tnet *netstack.Net, dst string, baseTimeout time.Duration, maxAttempts int) (time.Duration, error) {
var lastErr error
var totalLatency time.Duration
successCount := 0
for attempt := 1; attempt <= maxAttempts; attempt++ {
// Adaptive timeout: increase timeout for later attempts
timeout := baseTimeout + time.Duration(attempt-1)*500*time.Millisecond
// Add jitter to prevent thundering herd
jitter := time.Duration(rand.Intn(100)) * time.Millisecond
timeout += jitter
latency, err := ping(tnet, dst, timeout)
if err != nil {
lastErr = err
logger.Debug("Ping attempt %d/%d failed: %v", attempt, maxAttempts, err)
// Brief pause between attempts with exponential backoff
if attempt < maxAttempts {
backoff := time.Duration(attempt) * 50 * time.Millisecond
time.Sleep(backoff)
}
continue
}
totalLatency += latency
successCount++
// If we get at least one success, we can return early for health checks
if successCount > 0 {
avgLatency := totalLatency / time.Duration(successCount)
logger.Debug("Reliable ping succeeded after %d attempts, avg latency: %v", attempt, avgLatency)
return avgLatency, nil
}
}
if successCount == 0 {
return 0, fmt.Errorf("all %d ping attempts failed, last error: %v", maxAttempts, lastErr)
}
return totalLatency / time.Duration(successCount), nil
}
func pingWithRetry(tnet *netstack.Net, dst string, timeout time.Duration) (stopChan chan struct{}, err error) {
if healthFile != "" {
@@ -180,6 +235,9 @@ func startPingCheck(tnet *netstack.Net, serverIP string, client *websocket.Clien
consecutiveFailures := 0
connectionLost := false
// Track recent latencies for adaptive timeout calculation
recentLatencies := make([]time.Duration, 0, 10)
pingStopChan := make(chan struct{})
go func() {
@@ -188,18 +246,52 @@ func startPingCheck(tnet *netstack.Net, serverIP string, client *websocket.Clien
for {
select {
case <-ticker.C:
_, err := ping(tnet, serverIP, pingTimeout)
// Calculate adaptive timeout based on recent latencies
adaptiveTimeout := pingTimeout
if len(recentLatencies) > 0 {
var sum time.Duration
for _, lat := range recentLatencies {
sum += lat
}
avgLatency := sum / time.Duration(len(recentLatencies))
// Use 3x average latency as timeout, with minimum of pingTimeout
adaptiveTimeout = avgLatency * 3
if adaptiveTimeout < pingTimeout {
adaptiveTimeout = pingTimeout
}
if adaptiveTimeout > 15*time.Second {
adaptiveTimeout = 15 * time.Second
}
}
// Use reliable ping with multiple attempts
maxAttempts := 2
if consecutiveFailures > 4 {
maxAttempts = 4 // More attempts when connection is unstable
}
latency, err := reliablePing(tnet, serverIP, adaptiveTimeout, maxAttempts)
if err != nil {
consecutiveFailures++
if consecutiveFailures < 4 {
// Track recent latencies (add a high value for failures)
recentLatencies = append(recentLatencies, adaptiveTimeout)
if len(recentLatencies) > 10 {
recentLatencies = recentLatencies[1:]
}
if consecutiveFailures < 2 {
logger.Debug("Periodic ping failed (%d consecutive failures): %v", consecutiveFailures, err)
} else {
logger.Warn("Periodic ping failed (%d consecutive failures): %v", consecutiveFailures, err)
}
if consecutiveFailures >= 8 && currentInterval < maxInterval {
// More lenient threshold for declaring connection lost under load
failureThreshold := 4
if consecutiveFailures >= failureThreshold && currentInterval < maxInterval {
if !connectionLost {
connectionLost = true
logger.Warn("Connection to server lost. Continuous reconnection attempts will be made.")
logger.Warn("Connection to server lost after %d failures. Continuous reconnection attempts will be made.", consecutiveFailures)
stopFunc = client.SendMessageInterval("newt/ping/request", map[string]interface{}{}, 3*time.Second)
// Send registration message to the server for backward compatibility
err := client.SendMessage("newt/wg/register", map[string]interface{}{
@@ -216,7 +308,7 @@ func startPingCheck(tnet *netstack.Net, serverIP string, client *websocket.Clien
}
}
}
currentInterval = time.Duration(float64(currentInterval) * 1.5)
currentInterval = time.Duration(float64(currentInterval) * 1.3) // Slower increase
if currentInterval > maxInterval {
currentInterval = maxInterval
}
@@ -224,9 +316,15 @@ func startPingCheck(tnet *netstack.Net, serverIP string, client *websocket.Clien
logger.Debug("Increased ping check interval to %v due to consecutive failures", currentInterval)
}
} else {
// Track recent latencies
recentLatencies = append(recentLatencies, latency)
if len(recentLatencies) > 10 {
recentLatencies = recentLatencies[1:]
}
if connectionLost {
connectionLost = false
logger.Info("Connection to server restored!")
logger.Info("Connection to server restored after %d failures!", consecutiveFailures)
if healthFile != "" {
err := os.WriteFile(healthFile, []byte("ok"), 0644)
if err != nil {
@@ -235,12 +333,12 @@ func startPingCheck(tnet *netstack.Net, serverIP string, client *websocket.Clien
}
}
if currentInterval > pingInterval {
currentInterval = time.Duration(float64(currentInterval) * 0.8)
currentInterval = time.Duration(float64(currentInterval) * 0.9) // Slower decrease
if currentInterval < pingInterval {
currentInterval = pingInterval
}
ticker.Reset(currentInterval)
logger.Info("Decreased ping check interval to %v after successful ping", currentInterval)
logger.Debug("Decreased ping check interval to %v after successful ping", currentInterval)
}
consecutiveFailures = 0
}

View File

@@ -34,6 +34,7 @@ type Client struct {
onConnect func() error
onTokenUpdate func(token string)
writeMux sync.Mutex
clientType string // Type of client (e.g., "newt", "olm")
}
type ClientOption func(*Client)
@@ -61,10 +62,10 @@ func (c *Client) OnTokenUpdate(callback func(token string)) {
c.onTokenUpdate = callback
}
// NewClient creates a new Newt client
func NewClient(newtID, secret string, endpoint string, pingInterval time.Duration, pingTimeout time.Duration, opts ...ClientOption) (*Client, error) {
// NewClient creates a new websocket client
func NewClient(clientType string, ID, secret string, endpoint string, pingInterval time.Duration, pingTimeout time.Duration, opts ...ClientOption) (*Client, error) {
config := &Config{
NewtID: newtID,
ID: ID,
Secret: secret,
Endpoint: endpoint,
}
@@ -78,6 +79,7 @@ func NewClient(newtID, secret string, endpoint string, pingInterval time.Duratio
isConnected: false,
pingInterval: pingInterval,
pingTimeout: pingTimeout,
clientType: clientType,
}
// Apply options before loading config
@@ -96,6 +98,10 @@ func NewClient(newtID, secret string, endpoint string, pingInterval time.Duratio
return client, nil
}
func (c *Client) GetConfig() *Config {
return c.config
}
// Connect establishes the WebSocket connection
func (c *Client) Connect() error {
go c.connectWithRetry()
@@ -199,12 +205,31 @@ func (c *Client) getToken() (string, error) {
}
}
// Check for environment variable to skip TLS verification
if os.Getenv("SKIP_TLS_VERIFY") == "true" {
if tlsConfig == nil {
tlsConfig = &tls.Config{}
}
tlsConfig.InsecureSkipVerify = true
logger.Debug("TLS certificate verification disabled via SKIP_TLS_VERIFY environment variable")
}
var tokenData map[string]interface{}
// Get a new token
tokenData := map[string]interface{}{
"newtId": c.config.NewtID,
"secret": c.config.Secret,
if c.clientType == "newt" {
tokenData = map[string]interface{}{
"newtId": c.config.ID,
"secret": c.config.Secret,
}
} else if c.clientType == "olm" {
tokenData = map[string]interface{}{
"olmId": c.config.ID,
"secret": c.config.Secret,
}
}
jsonData, err := json.Marshal(tokenData)
if err != nil {
return "", fmt.Errorf("failed to marshal token request data: %w", err)
}
@@ -212,7 +237,7 @@ func (c *Client) getToken() (string, error) {
// Create a new request
req, err := http.NewRequest(
"POST",
baseEndpoint+"/api/v1/auth/newt/get-token",
baseEndpoint+"/api/v1/auth/"+c.clientType+"/get-token",
bytes.NewBuffer(jsonData),
)
if err != nil {
@@ -310,7 +335,7 @@ func (c *Client) establishConnection() error {
// Add token to query parameters
q := u.Query()
q.Set("token", token)
q.Set("clientType", "newt")
q.Set("clientType", c.clientType)
u.RawQuery = q.Encode()
// Connect to WebSocket
@@ -323,6 +348,15 @@ func (c *Client) establishConnection() error {
}
dialer.TLSClientConfig = tlsConfig
}
// Check for environment variable to skip TLS verification for WebSocket connection
if os.Getenv("SKIP_TLS_VERIFY") == "true" {
if dialer.TLSClientConfig == nil {
dialer.TLSClientConfig = &tls.Config{}
}
dialer.TLSClientConfig.InsecureSkipVerify = true
logger.Debug("WebSocket TLS certificate verification disabled via SKIP_TLS_VERIFY environment variable")
}
conn, _, err := dialer.Dial(u.String(), nil)
if err != nil {
return fmt.Errorf("failed to connect to WebSocket: %w", err)

View File

@@ -8,30 +8,36 @@ import (
"runtime"
)
func getConfigPath() string {
var configDir string
switch runtime.GOOS {
case "darwin":
configDir = filepath.Join(os.Getenv("HOME"), "Library", "Application Support", "newt-client")
case "windows":
configDir = filepath.Join(os.Getenv("APPDATA"), "newt-client")
default: // linux and others
configDir = filepath.Join(os.Getenv("HOME"), ".config", "newt-client")
func getConfigPath(clientType string) string {
configFile := os.Getenv("CONFIG_FILE")
if configFile == "" {
var configDir string
switch runtime.GOOS {
case "darwin":
configDir = filepath.Join(os.Getenv("HOME"), "Library", "Application Support", clientType+"-client")
case "windows":
logDir := filepath.Join(os.Getenv("PROGRAMDATA"), "olm")
configDir = filepath.Join(logDir, clientType+"-client")
default: // linux and others
configDir = filepath.Join(os.Getenv("HOME"), ".config", clientType+"-client")
}
if err := os.MkdirAll(configDir, 0755); err != nil {
log.Printf("Failed to create config directory: %v", err)
}
return filepath.Join(configDir, "config.json")
}
if err := os.MkdirAll(configDir, 0755); err != nil {
log.Printf("Failed to create config directory: %v", err)
}
return filepath.Join(configDir, "config.json")
return configFile
}
func (c *Client) loadConfig() error {
if c.config.NewtID != "" && c.config.Secret != "" && c.config.Endpoint != "" {
if c.config.ID != "" && c.config.Secret != "" && c.config.Endpoint != "" {
return nil
}
configPath := getConfigPath()
configPath := getConfigPath(c.clientType)
data, err := os.ReadFile(configPath)
if err != nil {
if os.IsNotExist(err) {
@@ -45,8 +51,8 @@ func (c *Client) loadConfig() error {
return err
}
if c.config.NewtID == "" {
c.config.NewtID = config.NewtID
if c.config.ID == "" {
c.config.ID = config.ID
}
if c.config.Secret == "" {
c.config.Secret = config.Secret
@@ -63,7 +69,7 @@ func (c *Client) loadConfig() error {
}
func (c *Client) saveConfig() error {
configPath := getConfigPath()
configPath := getConfigPath(c.clientType)
data, err := json.MarshalIndent(c.config, "", " ")
if err != nil {
return err

View File

@@ -1,7 +1,7 @@
package websocket
type Config struct {
NewtID string `json:"newtId"`
ID string `json:"id"`
Secret string `json:"secret"`
Endpoint string `json:"endpoint"`
TlsClientCert string `json:"tlsClientCert"`

202
wg/wg.go
View File

@@ -4,6 +4,7 @@ package wg
import (
"encoding/json"
"errors"
"fmt"
"net"
"os"
@@ -48,21 +49,24 @@ type PeerReading struct {
}
type WireGuardService struct {
interfaceName string
mtu int
client *websocket.Client
wgClient *wgctrl.Client
config WgConfig
key wgtypes.Key
newtId string
lastReadings map[string]PeerReading
mu sync.Mutex
Port uint16
stopHolepunch chan struct{}
host string
serverPubKey string
token string
stopGetConfig chan struct{}
interfaceName string
mtu int
client *websocket.Client
wgClient *wgctrl.Client
config WgConfig
key wgtypes.Key
keyFilePath string
newtId string
lastReadings map[string]PeerReading
mu sync.Mutex
Port uint16
stopHolepunch chan struct{}
host string
serverPubKey string
holePunchEndpoint string
token string
stopGetConfig func()
interfaceCreated bool
}
// Add this type definition
@@ -149,25 +153,27 @@ func NewWireGuardService(interfaceName string, mtu int, generateAndSaveKeyTo str
var key wgtypes.Key
// if generateAndSaveKeyTo is provided, generate a private key and save it to the file. if the file already exists, load the key from the file
if _, err := os.Stat(generateAndSaveKeyTo); os.IsNotExist(err) {
// generate a new private key
key, err = wgtypes.GeneratePrivateKey()
if err != nil {
logger.Fatal("Failed to generate private key: %v", err)
}
// save the key to the file
err = os.WriteFile(generateAndSaveKeyTo, []byte(key.String()), 0644)
if err != nil {
logger.Fatal("Failed to save private key: %v", err)
}
} else {
keyData, err := os.ReadFile(generateAndSaveKeyTo)
if err != nil {
logger.Fatal("Failed to read private key: %v", err)
}
key, err = wgtypes.ParseKey(string(keyData))
if err != nil {
logger.Fatal("Failed to parse private key: %v", err)
key, err = wgtypes.GeneratePrivateKey()
if err != nil {
return nil, fmt.Errorf("failed to generate private key: %v", err)
}
// Load or generate private key
if generateAndSaveKeyTo != "" {
if _, err := os.Stat(generateAndSaveKeyTo); os.IsNotExist(err) {
keyData, err := os.ReadFile(generateAndSaveKeyTo)
if err != nil {
return nil, fmt.Errorf("failed to read private key: %v", err)
}
key, err = wgtypes.ParseKey(strings.TrimSpace(string(keyData)))
if err != nil {
return nil, fmt.Errorf("failed to parse private key: %v", err)
}
} else {
err = os.WriteFile(generateAndSaveKeyTo, []byte(key.String()), 0600)
if err != nil {
return nil, fmt.Errorf("failed to save private key: %v", err)
}
}
}
@@ -177,18 +183,26 @@ func NewWireGuardService(interfaceName string, mtu int, generateAndSaveKeyTo str
client: wsClient,
wgClient: wgClient,
key: key,
keyFilePath: generateAndSaveKeyTo,
newtId: newtId,
host: host,
lastReadings: make(map[string]PeerReading),
stopHolepunch: make(chan struct{}),
stopGetConfig: make(chan struct{}),
}
// Get the existing wireguard port (keep this part)
device, err := service.wgClient.Device(service.interfaceName)
if err == nil {
service.Port = uint16(device.ListenPort)
logger.Info("WireGuard interface %s already exists with port %d\n", service.interfaceName, service.Port)
if service.Port != 0 {
logger.Info("WireGuard interface %s already exists with port %d\n", service.interfaceName, service.Port)
} else {
service.Port, err = FindAvailableUDPPort(49152, 65535)
if err != nil {
fmt.Printf("Error finding available port: %v\n", err)
return nil, err
}
}
} else {
service.Port, err = FindAvailableUDPPort(49152, 65535)
if err != nil {
@@ -203,22 +217,13 @@ func NewWireGuardService(interfaceName string, mtu int, generateAndSaveKeyTo str
wsClient.RegisterHandler("newt/wg/peer/remove", service.handleRemovePeer)
wsClient.RegisterHandler("newt/wg/peer/update", service.handleUpdatePeer)
if err := service.sendUDPHolePunch(service.host + ":21820"); err != nil {
logger.Error("Failed to send UDP hole punch: %v", err)
}
// start the UDP holepunch
go service.keepSendingUDPHolePunch(service.host)
return service, nil
}
func (s *WireGuardService) Close(rm bool) {
select {
case <-s.stopGetConfig:
// Already closed, do nothing
default:
close(s.stopGetConfig)
if s.stopGetConfig != nil {
s.stopGetConfig()
s.stopGetConfig = nil
}
s.wgClient.Close()
@@ -229,14 +234,29 @@ func (s *WireGuardService) Close(rm bool) {
}
// Remove the private key file
if err := os.Remove(s.key.String()); err != nil {
logger.Error("Failed to remove private key file: %v", err)
}
// if s.keyFilePath != "" {
// if err := os.Remove(s.keyFilePath); err != nil {
// logger.Error("Failed to remove private key file: %v", err)
// }
// }
}
}
func (s *WireGuardService) SetServerPubKey(serverPubKey string) {
func (s *WireGuardService) StartHolepunch(serverPubKey string, endpoint string) {
// if the device is already created dont start a new holepunch
if s.interfaceCreated {
return
}
s.serverPubKey = serverPubKey
s.holePunchEndpoint = endpoint
logger.Debug("Starting UDP hole punch to %s", s.holePunchEndpoint)
s.stopHolepunch = make(chan struct{})
// start the UDP holepunch
go s.keepSendingUDPHolePunch(s.holePunchEndpoint)
}
func (s *WireGuardService) SetToken(token string) {
@@ -244,16 +264,12 @@ func (s *WireGuardService) SetToken(token string) {
}
func (s *WireGuardService) LoadRemoteConfig() error {
// Send the initial message
err := s.sendGetConfigMessage()
if err != nil {
logger.Error("Failed to send initial get-config message: %v", err)
return err
}
// Start goroutine to periodically send the message until config is received
go s.keepSendingGetConfig()
s.stopGetConfig = s.client.SendMessageInterval("newt/wg/get-config", map[string]interface{}{
"publicKey": s.key.PublicKey().String(),
"port": s.Port,
}, 2*time.Second)
logger.Info("Requesting WireGuard configuration from remote server")
go s.periodicBandwidthCheck()
return nil
@@ -262,7 +278,8 @@ func (s *WireGuardService) LoadRemoteConfig() error {
func (s *WireGuardService) handleConfig(msg websocket.WSMessage) {
var config WgConfig
logger.Info("Received message: %v", msg)
logger.Debug("Received message: %v", msg)
logger.Info("Received WireGuard clients configuration from remote server")
jsonData, err := json.Marshal(msg.Data)
if err != nil {
@@ -276,7 +293,10 @@ func (s *WireGuardService) handleConfig(msg websocket.WSMessage) {
}
s.config = config
close(s.stopGetConfig)
if s.stopGetConfig != nil {
s.stopGetConfig()
s.stopGetConfig = nil
}
// Ensure the WireGuard interface and peers are configured
if err := s.ensureWireguardInterface(config); err != nil {
@@ -298,6 +318,7 @@ func (s *WireGuardService) ensureWireguardInterface(wgconfig WgConfig) error {
if err != nil {
logger.Fatal("Failed to create WireGuard interface: %v", err)
}
s.interfaceCreated = true
logger.Info("Created WireGuard interface %s\n", s.interfaceName)
} else {
logger.Fatal("Error checking for WireGuard interface: %v", err)
@@ -315,9 +336,16 @@ func (s *WireGuardService) ensureWireguardInterface(wgconfig WgConfig) error {
s.Port = uint16(device.ListenPort)
logger.Info("WireGuard interface %s already exists with port %d\n", s.interfaceName, s.Port)
s.interfaceCreated = true
return nil
}
// stop the holepunch its a channel
if s.stopHolepunch != nil {
close(s.stopHolepunch)
s.stopHolepunch = nil
}
logger.Info("Assigning IP address %s to interface %s\n", wgconfig.IpAddress, s.interfaceName)
// Assign IP address to the interface
err = s.assignIPAddress(wgconfig.IpAddress)
@@ -328,7 +356,10 @@ func (s *WireGuardService) ensureWireguardInterface(wgconfig WgConfig) error {
// Check if the interface already exists
_, err = s.wgClient.Device(s.interfaceName)
if err != nil {
return fmt.Errorf("interface %s does not exist", s.interfaceName)
if errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("interface %s does not exist", s.interfaceName)
}
return fmt.Errorf("failed to get device: %v", err)
}
// Parse the private key
@@ -447,7 +478,7 @@ func (s *WireGuardService) ensureWireguardPeers(peers []Peer) error {
}
func (s *WireGuardService) handleAddPeer(msg websocket.WSMessage) {
logger.Info("Received message: %v", msg.Data)
logger.Debug("Received message: %v", msg.Data)
var peer Peer
jsonData, err := json.Marshal(msg.Data)
@@ -520,7 +551,7 @@ func (s *WireGuardService) addPeer(peer Peer) error {
}
func (s *WireGuardService) handleRemovePeer(msg websocket.WSMessage) {
logger.Info("Received message: %v", msg.Data)
logger.Debug("Received message: %v", msg.Data)
// parse the publicKey from the message which is json { "publicKey": "asdfasdfl;akjsdf" }
type RemoveRequest struct {
PublicKey string `json:"publicKey"`
@@ -568,7 +599,7 @@ func (s *WireGuardService) removePeer(publicKey string) error {
}
func (s *WireGuardService) handleUpdatePeer(msg websocket.WSMessage) {
logger.Info("Received message: %v", msg.Data)
logger.Debug("Received message: %v", msg.Data)
// Define a struct to match the incoming message structure with optional fields
type UpdatePeerRequest struct {
PublicKey string `json:"publicKey"`
@@ -629,7 +660,7 @@ func (s *WireGuardService) handleUpdatePeer(msg websocket.WSMessage) {
}
// Only update AllowedIPs if provided in the request
if request.AllowedIPs != nil && len(request.AllowedIPs) > 0 {
if len(request.AllowedIPs) > 0 {
var allowedIPs []net.IPNet
for _, ipStr := range request.AllowedIPs {
_, ipNet, err := net.ParseCIDR(ipStr)
@@ -917,6 +948,11 @@ func (s *WireGuardService) encryptPayload(payload []byte) (interface{}, error) {
}
func (s *WireGuardService) keepSendingUDPHolePunch(host string) {
// send initial hole punch
if err := s.sendUDPHolePunch(host + ":21820"); err != nil {
logger.Error("Failed to send initial UDP hole punch: %v", err)
}
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
@@ -949,33 +985,3 @@ func (s *WireGuardService) removeInterface() error {
return nil
}
func (s *WireGuardService) sendGetConfigMessage() error {
err := s.client.SendMessage("newt/wg/get-config", map[string]interface{}{
"publicKey": fmt.Sprintf("%s", s.key.PublicKey().String()),
"port": s.Port,
})
if err != nil {
logger.Error("Failed to send get-config message: %v", err)
return err
}
logger.Info("Requesting WireGuard configuration from remote server")
return nil
}
func (s *WireGuardService) keepSendingGetConfig() {
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
for {
select {
case <-s.stopGetConfig:
logger.Info("Stopping get-config messages")
return
case <-ticker.C:
if err := s.sendGetConfigMessage(); err != nil {
logger.Error("Failed to send periodic get-config: %v", err)
}
}
}
}

1284
wgnetstack/wgnetstack.go Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -8,6 +8,8 @@ import (
"time"
"github.com/fosrl/newt/logger"
"golang.zx2c4.com/wireguard/tun/netstack"
"gvisor.dev/gvisor/pkg/tcpip/adapters/gonet"
)
const (
@@ -26,7 +28,9 @@ const (
// Server handles listening for connection check requests using UDP
type Server struct {
conn *net.UDPConn
conn net.Conn // Generic net.Conn interface (could be *net.UDPConn or *gonet.UDPConn)
udpConn *net.UDPConn // Regular UDP connection (when not using netstack)
netstackConn interface{} // Netstack UDP connection (when using netstack)
serverAddr string
serverPort uint16
shutdownCh chan struct{}
@@ -34,6 +38,8 @@ type Server struct {
runningLock sync.Mutex
newtID string
outputPrefix string
useNetstack bool
tnet interface{} // Will be *netstack.Net when using netstack
}
// NewServer creates a new connection test server using UDP
@@ -44,6 +50,21 @@ func NewServer(serverAddr string, serverPort uint16, newtID string) *Server {
shutdownCh: make(chan struct{}),
newtID: newtID,
outputPrefix: "[WGTester] ",
useNetstack: false,
tnet: nil,
}
}
// NewServerWithNetstack creates a new connection test server using WireGuard netstack
func NewServerWithNetstack(serverAddr string, serverPort uint16, newtID string, tnet *netstack.Net) *Server {
return &Server{
serverAddr: serverAddr,
serverPort: serverPort + 1, // use the next port for the server
shutdownCh: make(chan struct{}),
newtID: newtID,
outputPrefix: "[WGTester] ",
useNetstack: true,
tnet: tnet,
}
}
@@ -59,18 +80,30 @@ func (s *Server) Start() error {
//create the address to listen on
addr := net.JoinHostPort(s.serverAddr, fmt.Sprintf("%d", s.serverPort))
// Create UDP address to listen on
udpAddr, err := net.ResolveUDPAddr("udp", addr)
if err != nil {
return err
}
if s.useNetstack && s.tnet != nil {
// Use WireGuard netstack
tnet := s.tnet.(*netstack.Net)
udpAddr := &net.UDPAddr{Port: int(s.serverPort)}
netstackConn, err := tnet.ListenUDP(udpAddr)
if err != nil {
return err
}
s.netstackConn = netstackConn
s.conn = netstackConn
} else {
// Use regular UDP socket
udpAddr, err := net.ResolveUDPAddr("udp", addr)
if err != nil {
return err
}
// Create UDP connection
conn, err := net.ListenUDP("udp", udpAddr)
if err != nil {
return err
udpConn, err := net.ListenUDP("udp", udpAddr)
if err != nil {
return err
}
s.udpConn = udpConn
s.conn = udpConn
}
s.conn = conn
s.isRunning = true
go s.handleConnections()
@@ -96,6 +129,26 @@ func (s *Server) Stop() {
logger.Info(s.outputPrefix + "Server stopped")
}
// RestartWithNetstack stops the current server and restarts it with netstack
func (s *Server) RestartWithNetstack(tnet *netstack.Net) error {
s.Stop()
// Update configuration to use netstack
s.useNetstack = true
s.tnet = tnet
// Clear previous connections
s.conn = nil
s.udpConn = nil
s.netstackConn = nil
// Create new shutdown channel
s.shutdownCh = make(chan struct{})
// Restart the server
return s.Start()
}
// handleConnections processes incoming packets
func (s *Server) handleConnections() {
buffer := make([]byte, 2000) // Buffer large enough for any UDP packet
@@ -112,14 +165,30 @@ func (s *Server) handleConnections() {
continue
}
// Read from UDP connection
n, addr, err := s.conn.ReadFromUDP(buffer)
// Read from UDP connection - handle both regular UDP and netstack UDP
var n int
var addr net.Addr
if s.useNetstack {
// Use netstack UDP connection
netstackConn := s.netstackConn.(*gonet.UDPConn)
n, addr, err = netstackConn.ReadFrom(buffer)
} else {
// Use regular UDP connection
n, addr, err = s.udpConn.ReadFromUDP(buffer)
}
if err != nil {
if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
// Just a timeout, keep going
continue
}
logger.Error(s.outputPrefix+"Error reading from UDP: %v", err)
// Check if we're shutting down and the connection was closed
select {
case <-s.shutdownCh:
return // Don't log error if we're shutting down
default:
logger.Error(s.outputPrefix+"Error reading from UDP: %v", err)
}
continue
}
@@ -152,8 +221,17 @@ func (s *Server) handleConnections() {
// Log response being sent for debugging
logger.Debug(s.outputPrefix+"Sending response to %s", addr.String())
// Send the response packet directly to the source address
_, err = s.conn.WriteToUDP(responsePacket, addr)
// Send the response packet - handle both regular UDP and netstack UDP
if s.useNetstack {
// Use netstack UDP connection
netstackConn := s.netstackConn.(*gonet.UDPConn)
_, err = netstackConn.WriteTo(responsePacket, addr)
} else {
// Use regular UDP connection
udpAddr := addr.(*net.UDPAddr)
_, err = s.udpConn.WriteToUDP(responsePacket, udpAddr)
}
if err != nil {
logger.Error(s.outputPrefix+"Error sending response: %v", err)
} else {