[GH-ISSUE #96] Hole-punch endpoint dedup skips re-registration on reconnect → clients stuck "connected but not registered" (regression in 1.4.x) #1055

New Issue

GiteaMirror · 2026-07-12T09:16:40-05:00

GiteaMirror commented

2026-07-12 09:16:40 -05:00

Originally created by @DanivosYoun on GitHub (Jun 13, 2026).
Original GitHub issue: https://github.com/fosrl/gerbil/issues/96

Originally assigned to: @oschwartz10612 on GitHub.

Affected versions

gerbil 1.4.0 / 1.4.1 (works on 1.3.1). Seen with olm 1.6.0 (latest), newt 1.12.2, Pangolin 1.19.x, gerbil running on a remote exit node.

Symptom

After upgrading the remote node's gerbil to 1.4.x, olm clients can no longer finish connecting. The control-plane websocket comes up but registration never completes. olm /status:

{ "connected": true, "registered": false, "networkSettings": {} }

The client is stuck on "Registering" indefinitely. Downgrading gerbil to 1.3.1 restores connectivity. The latest olm (1.6.0) does not help — so this is gerbil-side.

Root cause

relay/relay.go (1.4.x) added an endpoint-notification dedup (lastEndpointCache). On a hole-punch packet, gerbil skips notifyServer() (POST /gerbil/update-hole-punch) when the client's endpoint state is unchanged:

cacheKey := endpoint.OlmID + ":" + endpoint.NewtID
newState := cachedEndpointState{OlmID, NewtID, Token, IP, Port, PublicKey}
if cached, ok := s.lastEndpointCache.Load(cacheKey); ok && cached.(cachedEndpointState) == newState {
    // Endpoint unchanged - skip the HTTP call but still clear stale sessions.
    s.clearSessionsForIP(endpoint.IP)
    continue   // notifyServer() skipped
}

notifyServer() is what registers the client with the server and returns the ProxyMapping that establishes routing — not just an idempotent endpoint update. Because olm credentials/public key are persistent and reconnects often land on the same NAT IP:port, the newState matches the cached one, so notifyServer() is skipped. The server has already torn down the client's session on disconnect, so it never re-registers the client and never returns a fresh ProxyMapping → registered: false. The cache is in-process, so once primed, every reconnect from the same endpoint is deduplicated → permanent failure until gerbil restarts or the endpoint changes.

Reproduce

gerbil 1.4.x on a (remote) exit node.
Connect an olm client once (primes lastEndpointCache).
Disconnect and reconnect from the same NAT endpoint with the same key.
olm stays connected: true, registered: false; no ProxyMapping is created.

Restarting gerbil (clearing the cache) lets exactly one subsequent connect register again, then it breaks on the next reconnect — confirming the dedup is the cause.

Suggested fix

The dedup must not suppress re-registration when the server no longer has the client's session. Options:

Invalidate the lastEndpointCache entry whenever the client's sessions are cleared, so a reconnect always re-notifies (simplest), or
Only skip when an active ProxyMapping for the client still exists, or
Don't dedup the registration notification at all (keep dedup only for genuinely redundant high-frequency updates).

Relevant code: relay/relay.go — the lastEndpointCache / cachedEndpointState dedup block, and notifyServer() → POST /gerbil/update-hole-punch.

Hotfix PR

I've already opened a hotfix PR implementing the first option (invalidate the cache entry in the dedup path): #95. Happy to adjust to whichever approach the maintainers prefer.

Originally created by @DanivosYoun on GitHub (Jun 13, 2026). Original GitHub issue: https://github.com/fosrl/gerbil/issues/96 Originally assigned to: @oschwartz10612 on GitHub. ### Affected versions gerbil **1.4.0 / 1.4.1** (works on **1.3.1**). Seen with olm **1.6.0** (latest), newt **1.12.2**, Pangolin **1.19.x**, gerbil running on a **remote exit node**. ### Symptom After upgrading the remote node's gerbil to 1.4.x, olm clients can no longer finish connecting. The control-plane websocket comes up but registration never completes. `olm /status`: ```json { "connected": true, "registered": false, "networkSettings": {} } ``` The client is stuck on "Registering" indefinitely. Downgrading gerbil to 1.3.1 restores connectivity. The latest olm (1.6.0) does not help — so this is gerbil-side. ### Root cause `relay/relay.go` (1.4.x) added an endpoint-notification dedup (`lastEndpointCache`). On a hole-punch packet, gerbil skips `notifyServer()` (`POST /gerbil/update-hole-punch`) when the client's endpoint state is unchanged: ```go cacheKey := endpoint.OlmID + ":" + endpoint.NewtID newState := cachedEndpointState{OlmID, NewtID, Token, IP, Port, PublicKey} if cached, ok := s.lastEndpointCache.Load(cacheKey); ok && cached.(cachedEndpointState) == newState { // Endpoint unchanged - skip the HTTP call but still clear stale sessions. s.clearSessionsForIP(endpoint.IP) continue // notifyServer() skipped } ``` `notifyServer()` is what registers the client with the server and returns the `ProxyMapping` that establishes routing — not just an idempotent endpoint update. Because olm credentials/public key are persistent and reconnects often land on the same NAT IP:port, the `newState` matches the cached one, so `notifyServer()` is skipped. The server has already torn down the client's session on disconnect, so it never re-registers the client and never returns a fresh `ProxyMapping` → `registered: false`. The cache is in-process, so once primed, **every** reconnect from the same endpoint is deduplicated → permanent failure until gerbil restarts or the endpoint changes. ### Reproduce 1. gerbil 1.4.x on a (remote) exit node. 2. Connect an olm client once (primes `lastEndpointCache`). 3. Disconnect and reconnect from the same NAT endpoint with the same key. 4. olm stays `connected: true, registered: false`; no `ProxyMapping` is created. - Restarting gerbil (clearing the cache) lets exactly one subsequent connect register again, then it breaks on the next reconnect — confirming the dedup is the cause. ### Suggested fix The dedup must not suppress re-registration when the server no longer has the client's session. Options: - Invalidate the `lastEndpointCache` entry whenever the client's sessions are cleared, so a reconnect always re-notifies (simplest), or - Only skip when an active `ProxyMapping` for the client still exists, or - Don't dedup the registration notification at all (keep dedup only for genuinely redundant high-frequency updates). Relevant code: `relay/relay.go` — the `lastEndpointCache` / `cachedEndpointState` dedup block, and `notifyServer()` → `POST /gerbil/update-hole-punch`. ### Hotfix PR I've already opened a hotfix PR implementing the first option (invalidate the cache entry in the dedup path): #95. Happy to adjust to whichever approach the maintainers prefer.

GiteaMirror added the bug label 2026-07-12 09:16:40 -05:00

GiteaMirror closed this issue

2026-07-12 09:16:41 -05:00

GiteaMirror commented

2026-07-12 09:16:42 -05:00

@oschwartz10612 commented on GitHub (Jun 13, 2026):

Ahh this sounds like it could be an issue

@oschwartz10612 commented on GitHub (Jun 13, 2026): Ahh this sounds like it could be an issue

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gerbil#1055