Skip to content

daemon: blackholed peers stay on the relay; the next packet no longer overwrites a cached peer key - #505

Open
TeoSlayer wants to merge 1 commit into
mainfrom
fix/peer-key-alias-and-relay-unpin
Open

TeoSlayer wants to merge 1 commit into
mainfrom
fix/peer-key-alias-and-relay-unpin

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Two daemon bugs found while tracing why one node lost its path to list-agents for over half an hour on 2026-10-01. Both are on main and in every release since the code they touch was added.

1. A blackholed peer was put back on its dead direct path every 15 s.
tryDirectUpgrade runs each RelayProbeInterval for every relay peer. To let a working direct path win it unpins a peer the blackhole heuristic pinned, and it did so with SetRelayPeerPinned(id, false), which clears the relay flag as well as the pin. Traffic went direct again until three more silent sends tripped the heuristic. In the live log this is direct path silent, flipping to relay every ~90 s with silent_for growing past 29 minutes and no relay→direct auto-cleared in between: about 75 s of every 90 s spent sending to an address that never answered.

2. A peer's cached Ed25519 key was overwritten by the next packet.
HandleAuthFrame cached data[36:68], a slice of the frame it was handed. That frame is the socket's single reused receive buffer (udpio.Socket.Recv: "valid only until the next Recv"), so the following packet replaced the cached key with its own bytes. The next PILA from that peer mismatched the cache and forced a registry lookup on the UDP read loop, stalling all packet processing for that round trip. The log line auth key exchange: peer pubkey updated from registry appeared 97 times in a six-hour session, for peers whose registry key never changed.

Changes

  • routing.Manager.UnpinRelayPeer removes only the pin. tryDirectUpgrade uses it, so the peer stays on the relay until ClearRelayOnDirect has seen DirectClearsRequired direct packets from it.
  • HandleAuthFrame copies the key out of the frame when parsing; SetPeerPubKey copies what it is given, so no other caller can reintroduce the alias.

Related, not changed here

The key-bound trust check on the data plane (Daemon.handshakeTrusts → IsTrustedWithKey, #424) reads this same cache. It never runs in a shipped daemon: runtime.NewHandshakeServiceAdapter does not forward IsTrustedWithKey, so the daemon falls back to IsTrusted(nodeID). Had the adapter forwarded it while the cache held overwritten bytes, every trust record would have been dropped on the peer's first inbound SYN. The adapter gap is a separate decision; this PR removes the hazard so it can be closed safely.

Test Plan

  • TestDirectUpgradeUnpinsWithoutLeavingTheRelay, TestBlackholedPeerStaysOnRelayAcrossProbeTicks and TestCachedPeerKeySurvivesReceiveBufferReuse fail on main and pass here
  • GOWORK=off go build ./..., go vet ./pkg/daemon/..., go test ./pkg/... ./cmd/... ./internal/... -short
  • Full go test -parallel 4 -count=1 ./tests/: pass (274 s)
  • Two Docker nodes with UDP between them dropped by iptables (harness from tests: make the Docker integration suite build and boot again #506), four send-message calls 8 s apart:
Build "flipping to relay" lines on the sender in ~90 s Messages delivered
main 3 (36 s, 71 s, 116 s after the partition: the flip does not hold) 0 of 4
this branch 1 2 of 4 (the two sent after both sides had moved to the relay)

The two early sends on this branch fail at once with sendto: operation not permitted: a local send error on the direct path fails the dial instead of trying the relay. That is a separate gap, not changed here; it is also why test_force_relay_* still fail (they send 10 s after the partition).

Also visible in that run on main: the receiving node logs auth key exchange: peer pubkey updated from registry twice within 5 ms of the first tunnel coming up, with two nodes and a registry that has held one key per node since boot.

Checklist

  • New files carry the SPDX license header
  • go.mod / go.sum unchanged
  • CHANGELOG updated

🤖 Generated with Claude Code

…writing a cached peer key

Two bugs found while tracing why a node kept losing its path to one
peer (list-agents) on 2026-10-01.

Relay flag cleared on unpin. tryDirectUpgrade runs every
RelayProbeInterval (15 s) for each relay peer. To let a working direct
path win it must unpin a peer the blackhole heuristic pinned, and it
called SetRelayPeerPinned(id, false), which clears the relay flag as
well as the pin. Every blackholed peer was therefore back on its dead
direct path within 15 s and stayed there until three more silent sends
flipped it again: about 75 s of every 90 s spent sending to an address
that never answers. A new routing.UnpinRelayPeer removes only the pin;
ClearRelayOnDirect still moves the peer back after DirectClearsRequired
direct packets.

Cached peer key aliased the receive buffer. HandleAuthFrame sliced the
peer's Ed25519 key out of the frame it was handed and cached that slice.
The frame is the socket's single reused read buffer, so the next packet
overwrote the cached key. The next PILA from the peer mismatched the
cache and triggered a registry lookup on the UDP read loop ("auth key
exchange: peer pubkey updated from registry"). The key is now copied
when parsed, and SetPeerPubKey copies what it is given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer marked this pull request as ready for review October 1, 2026 21:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant