Skip to content

quic: a reactor shutdown tells its peers with a CONNECTION_CLOSE instead of leaving them to time out - #380

Open
MDA2AV wants to merge 1 commit into
mainfrom
fix/195-quic-shutdown-close
Open

MDA2AV wants to merge 1 commit into
mainfrom
fix/195-quic-shutdown-close

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Fixes #195.

What was wrong. At a reactor shutdown, TeardownQuic removed each connection before evicting it. QuicRemoveConnection frees and zeroes the peer address, so Send was already a no-op when OnEvicted ran. QuicEngineConnection.OnEvicted only called Destroy() anyway. A farewell written in time would still have been lost: TeardownQuic runs in Reactor.Teardown, after the loop has exited, so an SQE queued there is never submitted, and CloseUdpFds closes the socket two lines later. Every peer was left to its own idle timeout.

What changed. Three parts, and each is needed (see Proof):

  1. Core, TeardownQuic: OnEvicted now runs before QuicRemoveConnection, so the peer address is still live for the farewell. The engine's own Teardown removes the connection first, which makes the second removal a no-op through the existing set-membership gate.
  2. QuicEngineConnection.OnEvicted: ReactorShutdown goes through the engine's existing Teardown(WriteTransportFarewell(0)). ReadTimeout (the idle sweep) and TimerFault still call Destroy() and stay silent (RFC 9000 10.1).
  3. Core, UdpSendTo: once _stopRequested is set, a datagram goes out with a direct sendmsg(MSG_DONTWAIT) instead of an SQE, because no loop pass may submit again. The slot is released through the existing OnUdpSendCompletion, which also logs a failure.

Supporting core hunks:

  • Reactor.Teardown sets _stopRequested = true first, because a faulted loop exit never sets it. Without this, a faulted exit would send the farewells to the ring. On a reactor that died with its SQ stuck full, GetSqeOrFlush would then throw out of Run()'s finally, skipping CloseUdpFds, the TCP closes and the ring dispose. Side effect: sibling reactors stop forwarding datagrams to a faulted reactor slightly earlier, since QuicTryForward checks the same flag.
  • Native.Socket.cs: sendmsg and MSG_DONTWAIT, both internal, so there is no public API change.
  • QuicConnection.OnEvicted doc: it no longer says the connection is always unregistered when this runs.

The core diff is 5 files, +19/-2.

Why a direct send rather than submit-and-wait. Entering the ring from Teardown has side effects:

  • With DEFER_TASKRUN, a GETEVENTS enter runs the deferred task work of armed multishots whose sockets became ready. An armed accept could take a new connection whose CQE teardown then drops, leaking its fd.
  • A submit also sends whatever the last loop pass queued. That can include accept re-arms for listeners that CloseTcpListeners has already closed.

A direct sendmsg touches only the UDP socket. It submits nothing and processes no CQEs, so the ring's state is untouched, and it works on a broken ring too.

The farewell code. Transport NO_ERROR. WriteTransportFarewell(0) calls iq_conn_close_liberr(0), and in ngtcp2 ngtcp2_ccerr_set_liberr(0) resolves to ngtcp2_err_infer_quic_transport_error_code(0), which is NGTCP2_NO_ERROR (checked at the pinned ngtcp2 7f7e24b). This is the engine's own immediate-close path, the one CloseFromEngine and the retention backstop use, with "no error" as the code.

  • The transport does not know the application protocol, so H3_NO_ERROR would be wrong on a raw QUIC (echo) connection.
  • RFC 9000 20.1 gives NO_ERROR for a connection "closed abruptly in the absence of any error".
  • To send an application code instead, change that one line to Close(code).

The wait bound. There is no wait. Each farewell is one non-blocking sendmsg; the flag is needed because the UDP sockets are blocking. Shutdown cannot hang on it.

  • Cost: one syscall per live connection, plus one per coalesced GSO batch that FlushGso sends first.
  • A socket whose send buffer is full drops the farewell (EAGAIN, logged as udp send error: -11), as a lost packet would be. The peer then falls back to its idle timeout, which is the behaviour today.

Proof. The pending test quic: a reactor shutdown tells its peers instead of leaving them to time out (QuicTeardownWireTests) is promoted to a Test. Each part was reverted in turn, with E2E registering only that file:

  • Part 1 reverted: the peer was left to time out: 2 datagrams arrived after the shutdown and none was a CONNECTION_CLOSE.
  • Part 2 reverted: the same failure, also 2 datagrams.
  • Part 3 reverted: the peer was left to time out: 0 datagrams arrived after the shutdown and none was a CONNECTION_CLOSE.
  • All three in place: 5/5 pass.

The file's other four tests pass in every run: application Close, retention push-back, retention backstop, and the idle sweep staying silent.

Suites. Main 71aab91 against this branch, same box, taskset -c 0-7,12-31:

  • E2E: 292 passed, 5 pending before; 293 passed, 4 pending after. Wall time 304 s before, 303 s after, so teardown is not slower.
  • Unit: 96 passed, 1 pending, unchanged.
  • Http: 88 passed, unchanged.
  • Chaos: 79 passed, 1 pending, unchanged.
  • No udp send error line in any run.
  • ioxide and ioxide.ngtcp2 build for net10.0 and net11.0 with no warnings.
  • UdpSendTo gains one volatile bool read per datagram, of the flag the loop already reads on every pass. Benched with the batch in http3: a request whose body does not match its content-length is refused with H3_MESSAGE_ERROR instead of being served #383 (Http3/ManagedBuffered): +0.5% req/s and -0.4% CPU/req against main, where main against itself was +0.4%. Flat.

Interactions checked.

  • reactor: a stop frees the native memory of its pooled connections and its spare write slabs #360's teardown frees pooled connections and spare slabs after the ring is disposed. Nothing here enters the ring after the loop, so that path is untouched.
  • Client connections: QuicClientEngine also makes a QuicEngineConnection, so these get the same farewell on their own UDP socket, which is still open in TeardownQuic. A client still in CLIENT_INITIAL sends nothing, because ngtcp2 refuses to write a close in that state.
  • User overrides of OnEvicted now run at shutdown while the connection is still in _quicConnSet and its CID routes. Nothing routes after the loop, so in practice this only means Send still works. The handler is now woken by MarkClosed after OnEvicted rather than before. The engine closes its send gate before that wake, as it does on every other Teardown path.
  • The idle sweep and timer-fault paths are unchanged: they still remove, then evict.

All eight together. This batch's PRs (#376 to #383) were merged onto main 71aab91 in one scratch tree. The only conflict is the binary libioxide_ngtcp2.so between #376 and #377, rebuilt there from the merged shim with the script's defaults. Everything builds, and Unit 102, E2E 309, Http 88, Chaos 80, Tls 169 (14 kTLS cases skipped) and File 5 pass, with 0 failed and 0 pending. The repo's only pending test left is the Redis suite's (#336).

Noticed, not changed.

  • An h3 server still sends no GOAWAY at a reactor shutdown. The graceful path is still the h3 layer's own shutdown.
  • TCP sends queued in a stopping reactor's last loop pass are still never submitted. Only UDP now leaves directly. This predates this PR.
  • QuicRemoveConnection still queues a SubmitCancel for a pinned (migrated) connection at teardown, through GetSqeOrFlush. On a reactor whose SQ is stuck full, that can still throw. This predates this PR too.

Overlap. Merges cleanly (git merge-tree) with #376, #377, #378 and #379 (issues #349, #200, #354 and #353). It touches no native code, so the libioxide_ngtcp2.so that #349 and #200 rebuild is unaffected.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quic: a reactor shutting down frees every connection without telling the peer

1 participant