Fix datagram port leak, 40ms stream stalls and several pilotctl failure paths - #490
Merged
Merged
Conversation
…re paths Found by running a node in a 1 GB container under load and profiling it. Daemon - A datagram's source port was never released: only closing a connection cleared an ephemeral port, and a datagram has none. After ~16k datagrams every dial failed until restart. The port is released once the datagram is sent. - DelayedACKTimeout 40ms -> 5ms. A write held by Nagle waits for the data before it to be ACKed; a lone or odd segment's ACK waited 40ms. That was 40ms on the first exchange of every connection and one stall per 48KB file chunk (~1.5 MB/s on any link). The hold itself stays: removing it bursts a full window into the peer's socket buffer and collapses under loss. - A dial is woken when its handshake completes (conn.DialCh) instead of on the next 10ms poll; the poll remains as a backstop. - The tunnel socket requests 4 MB kernel buffers (best effort). - The info reply builds typed rows instead of a map per peer/connection; it was most of the daemon's CPU under send load. JSON is unchanged. pilotctl - send-message exits non-zero when the receiver answers "ERR ...". - appstore install removes its unpack directory, on success and failure. - appstore call waits up to 15s for a just-installed app's socket, and reports a suspended app as such. - received --clear also removes interrupted transfers in .partial. - A dial that fails for lack of local ports says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Rebased onto current Re-run after the rebase with |
TeoSlayer
force-pushed
the
hosted-node-fixes
branch
from
October 1, 2026 15:17
c6e9b7c to
c70593d
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // The bundle was unpacked into a temporary directory for this | ||
| // install alone. It was never removed, so every install — failed or | ||
| // not — left a copy of the app behind in $TMPDIR. | ||
| removeUnpacked := func() { _ = os.RemoveAll(bundleDir) } |
| } | ||
| } | ||
| } | ||
| if _, serr := os.Stat(filepath.Join(appDir, ".suspended")); serr == nil { |
| func waitForAppSocket(appDir, sockPath string, wait time.Duration) error { | ||
| deadline := time.Now().Add(wait) | ||
| for { | ||
| _, err := os.Stat(sockPath) |
| if err == nil { | ||
| return nil | ||
| } | ||
| if _, merr := os.Stat(filepath.Join(appDir, "manifest.json")); merr != nil { |
| if _, merr := os.Stat(filepath.Join(appDir, "manifest.json")); merr != nil { | ||
| return err // not installed | ||
| } | ||
| if _, serr := os.Stat(filepath.Join(appDir, ".suspended")); serr == nil { |
`appstore audit` lists the supervisor's events (exit codes), not the reason; the app's stderr goes to the daemon's log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Fixes found by running a node in a 1 GB container under load and profiling it: a port leak that stops a node dialing after ~16k datagrams, two timers that stalled every new connection and every file chunk with both ends idle, and several
pilotctlpaths that reported success or left files behind.Changes
Daemon
DelayedACKTimeout40ms → 5ms. A write held by Nagle waits for the data before it to be ACKed, and a lone or odd segment's ACK waited 40ms: 40ms on the first exchange of every connection, and one stall per 48KB file chunk (~1.5 MB/s on any link).conn.DialCh) instead of on the next 10ms poll, which stays as a backstop.net.core.rmem_max/wmem_max).inforeply builds typed rows instead of a map per peer and connection.pilotctl send-messageasks for it on every send; it was 77% of daemon CPU under send load. A test asserts the JSON is byte-identical to the maps'.pilotctl
send-messageexits non-zero when the receiver answersERR ...(assend-filealready did).appstore installremoves its unpack directory on success, failure and fatal exit. Seven installs had left seven copies, 60 MB.appstore callwaits up to 15s for a just-installed app's socket; a suspended app is reported as suspended.received --clearalso removes interrupted transfers in.partial(newcleared_partialfield).Measured (two containers on one bridge, 1 CPU each)
rmem_maxephemeral ports exhaustedTest Plan
go build ./...succeedsgo vet ./...cleango test ./pkg/... ./cmd/... ./internal/... -shortandgo test -parallel 4 -count=1 ./tests/pass withGOWORK=offTestDatagramsDoNotExhaustEphemeralPorts,TestFirstExchangeOnAConnectionIsNotDelayed,TestLargeWritesDoNotStallOnDelayedACK(both timing tests run serially and judge the fastest run, so machine load does not flip them)TestInfoRowsMarshalLikeTheMapsTheyReplaced,TestWaitForAppSocketreceived --clearchanges (verified by hand in the lab), and behaviour over a lossy or high-latency link — the lab is a local bridgeChecklist
go.mod/go.sumunchanged🤖 Generated with Claude Code