Repository navigation
epoxide: ioxide's thread-per-core runtime on epoll, whose connections any thread can read and write - #384
Open
MDA2AV wants to merge 22 commits into
Open
epoxide: ioxide's thread-per-core runtime on epoll, whose connections any thread can read and write#384MDA2AV wants to merge 22 commits into
MDA2AV wants to merge 22 commits into
Conversation
… any thread can read and write directly
… raw and thread-pool samples
… chaos suites on it Epoxide.Tests.Harness is tests/Ioxide.Tests.Harness against src/epoxide only: Runner (plus a name filter and KnownCoreBug, for tests that reproduce an unfixed core bug and stay out of the green run unless EPOXIDE_KNOWN_CORE_BUGS=run), TestServer with the same per-process port windows, Client, TestCert (cached under epoxide-e2e-*), Handlers and Wire (partial, so the TLS suite adds its handlers beside them), TcpSockets, FdCount, Forwarder, FakeServer, ExecInheritance, ReactorAffinity and Externals. Dropped: H3TestClient, CurlH3, the UDP/QUIC/file-client starters. Internals reads what epoxide does not expose (recv-pool counts, generations, refs) by UnsafeAccessor and reflection, and Hop moves a handler to the thread pool. Epoxide.Tests.Chaos carries ChaosClient, ChaosServer, TcpChaosTests (13) and ReactorChaosTests (5; the incremental-ring case has no epoxide counterpart). 18 passed.
…d and write - each one serializes its own BoringSSL calls instead of posting them to the reactor
… samples line for line on epoll
…ssions used from the thread pool - two tests reproduce a core bug and are listed as known
…e, with epoxide's own thread-pool and connect tests E2E ports tests/Ioxide.Tests.E2E/Core (all but QuicTests and UdpTests). Where a test guards something only io_uring has, it is translated to what it protects on epoll: the recv-queue overflow cases become a reader that falls behind getting every byte, the op-table growth cases become a burst of timer waits and waits armed from other threads, the CQ-overflow case becomes more ready connections than MaxEvents, the send-in-flight cases drive the reactor's end with a reset, and the recv-buffer reclaim cases count the reactor's pool, since epoxide's pool grows instead of running dry. SEND_ZC, MSG_WAITALL, incremental buffer rings and ring setup have no counterpart. New: ThreadPoolTests (11) - a handler on the thread pool reads, writes and flushes there; parked reads and flushes awaited from the pool resume on it, an awaiter on the reactor resumes inline (counted through a wrapping SynchronizationContext installed in OnStart); full duplex on two pool threads; peer resets mid-flush and mid-read; DecRef from the pool in both orders; Stop() with pool-thread handlers holding connections; 48 clients served entirely from the pool. ConnectTests (10) - ConnectAsync from the pool and from the reactor, to epoxide and to a plain socket, refused and unreachable, cross-reactor resumption, and after Stop(). Unit: VersionTests, ClockTests, SyscallErrnoTests (on the epoll and socket wrappers, with a control showing an import without SetLastError loses the errno), ConfigValidationTests (the construction half of RingSetupMessageTests). Three tests reproduce core bugs and are registered with KnownCoreBug, out of the green run: E1 a ConnectAsync racing Stop() never settles (20 of 20), E2 RecvBufferSize/WriteSlabSize below 4 throw from inside BufferPool naming 'bufferSize', E3 MaxEvents has no upper bound. E2E: 97 passed, 0 failed, 1 known. Unit: 14 passed, 0 failed, 2 known.
…ex thread-pool test checks its two loops overlap in time rather than per operation
…stop strands what only its reactor would run, a listener-less reactor has no clocks, a 32 MiB buffer wraps the pool's chunk size E4 (two tests): a handler on its reactor parked on a continuation posted back to it, or on a ReactorTimer, never resumes once Stop() has run - the loop that would run either has exited - so it never lets go and its descriptor stays open for the life of the process. E5: ServerConfig.Tcp == null registers no TCP sweep, so ConnectAsync's connections get none of TcpOptions' clocks: a flush its peer stopped draining was still parked 90 s into the default 60 s send timeout. E2, the large end: BufferPool sizes a chunk as bufferSize << 6 in int, so a 32 MiB RecvBufferSize wraps and the first recv throws OutOfMemoryException. Each fails as described when run (EPOXIDE_KNOWN_CORE_BUGS=run), and passes against a scratch copy of epoxide with the patches proposed in the report, with every suite still green there (Unit 16/16, E2E 102/102, Chaos 18/18). The full-duplex thread-pool test no longer sends a separate go-byte: the first recv could carry it with the start of the stream, and the handler's drain of it swallowed stream bytes. The client now pauses now and then on each side, so both server directions park every run (it had once parked a single flush). Discrimination: the key assertion of each new or translated test (43 of them) was flipped to the defect's expectation in this tree, built and run - every one failed against the real values - and restored. Unit 14 passed, 2 known; E2E 97 passed, 5 known; Chaos 18 passed - twice each.
…st epoll's limit Probed: at INT_MAX / sizeof(struct epoll_event) Run works; one above, MaxEvents * the event size wraps in int and the event buffer's allocation throws OutOfMemoryException before any epoll_wait.
…t that covers what epoxide has, ported to tests-epoxide/, and epoxide's own thread-pool and connect tests
…oll - the same BoringSSL and kTLS - with its suite, its chaos tests and two samples # Conflicts: # bench/samples.tsv # ioxide.slnx
…ps at an urgent mark, no longer counts as drained - the next read ends at once instead of at the read timeout Linux's tcp_recvmsg returns the queued data and takes a FIN queued behind it in the same call, so the recv comes back short with n > 0, and the edge that reported both is already spent: the next ReadAsync parked until ReadTimeoutMs, and with 0 for good. An edge that carries EPOLLRDHUP, EPOLLHUP, EPOLLERR or EPOLLPRI - registered now, since a recv stops short at an urgent mark the same way - sets ReadUntilEmpty for the rest of the connection's life, with a full fence before OnReadable so the reader's next CAS-then-recv sees it. Found by both the TLS port and the core review: 128 to 197 of 200 connections waited out the read timeout, depending on the shape; none in over 50,000 since. The two TLS tests that reproduced it run now, beside a plaintext one - a request and the FIN back to back, inline and on the thread pool - which fails with the fix reverted (3 inline handlers still reading 3 s after the FIN).
… lock, so a release from another thread, or the reactor's reclaim at recycle, cannot meet a read A thread-pool handler that releases its adapter after DecRef - `using var`, or Complete() below a try/finally that lets go - did it while the reactor recycled the connection and reclaimed the same buffers. The pipe reader threw a NullReferenceException on the reactor, which ended it; the stream returned a buffer twice, linking the pool's free list into a cycle that hands one buffer to every connection. Complete or Dispose from another task while a read was parked or completing raced the read's ingest the same way. Each adapter takes its own lock where its state moves - ingest, advance, release - and not for a read that finds nothing held; the caller resumes outside it. A read already parked when the caller completes still gets what lands, as in ioxide; only the reclaim at recycle stops it, since what is left is the reactor's to return then. The recv queue's indices carry on across lives instead of restarting at 0, so a snapshot kept from a previous life finds nothing to take. Two tests. Released after DecRef on the pool, 5,000 connections per adapter: with the fix reverted, the exception kills the reactor and the rest are refused. Released by another task while a read loop runs, 400 per adapter: 395 of 400 finished with the fix reverted.
…at takes it back on close cannot own it at the same time OnWritable read WrParked and sent, while TryParkFlush's closed re-check could take the same flush back (Parked -> Idle) and run EndFlush: the reactor then went on sending from a reset slab and over overflow slabs already returned, and ran EndFlush and SetResult a second time. It now claims the flush (Parked -> Idle) before sending, and parks it again when the socket is still full or the send failed. Found by the core review, proven with ordering hooks in a scratch copy: writerReclaims=1 and reactorSentAfterWriterReclaimed=1 before, writerReclaims=0 after. The window is the few nanoseconds between the writer's two CASes, so no test reaches it unaided.
…ation queued for it, a continuation posted to it, its timers A ConnectAsync from another thread that raced Stop() stayed in _registerQ, which the teardown never drained: the connect never settled, and its socket and the live count leaked, so the pools were never freed (20 of 20 rounds). A handler parked on a continuation posted back to its reactor, or on a ReactorTimer, never resumed once the loop had exited, so it never let go and its descriptor stayed open for the life of the process. The teardown fences after setting _stopped - producers enqueue, then read the flag - and drains the registrations (refused, which settles the connect), the posts and the timers. A cancelled timer completes with ReactorTimer.ECanceled (-125), which Expired() refuses. Anything posted or armed after the stop runs on the thread pool. The three tests that reproduced these run now.
…so their read and send timeouts and the deferred FIN apply The TCP sweep was registered only when ServerConfig.Tcp was set, so ConnectAsync's connections on an outbound-only reactor got none of TcpOptions' clocks: a flush its peer stopped draining was still parked 90 s into the 60 s send timeout. Such a reactor registers the sweep with its first outbound connection. The test that reproduced it runs now; it takes the 60 s default.
… buffer constructs as it does in ioxide BufferPool sized a 64-buffer chunk as bufferSize << 6 in int: a 32 MiB RecvBufferSize wrapped, asked for about 16 EiB, and every recv threw OutOfMemoryException. MaxEvents times the event size wrapped the same way in Run; a MaxEvents above what epoll_wait accepts is refused at construction now, by name. A buffer smaller than the pool's four-byte free-list link is spaced four bytes apart instead of refused with the pool's own parameter name. The three tests that reproduced these run now.
…s unseen, the README lists the TLS suite, and CI runs all four suites RunGuarded checked ObservedDeaths and then recorded the failure, so a claim landing in between left the death reported as "a test reactor died and no test observed it" (once in 8 TLS runs). It records first and withdraws when claimed; the test claims before it removes, so one of the two sees the other. ioxide's harness has the same order.
…n a row, so a peer that keeps its buffer full cannot hold every other connection on that reactor A reader receives on its own thread while the socket has data, and a recv that fills its buffer leaves the read Ready, so the next ReadAsync reads again at once. On the reactor's thread - a handler running inline - a peer that sends faster than the handler consumes keeps every recv full, and the handler never gave the loop back: every other connection on the reactor, and this one's own parked flush, waited for good. ioxide cannot do this, since a reader there drains only what the ring has already delivered. With 4 KiB buffers, a flooding client left another connection's pings unanswered (1 of 100, then 3 s timeouts); with 32 KiB buffers it took a reader slower than loopback - TLS decrypt is - which is how a TLS test with a writer on another thread came to stall once in 40 runs: the reactor never got to the flush's writable edge. Past 16 reads in a row on the reactor's thread the reader parks and is queued to be attended on the next pass, where AttendNow reads for it as an edge would. The pings are then answered 100 of 100 (p99 3.4 ms under the flood). The new test fails with the budget off (0 and 2 of 20 answered) and passes with it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new project,
src/epoxide(packageepoxide, namespaceepoxide): ioxide's shared-nothing,thread-per-core runtime with epoll in place of io_uring. TCP and TLS - the same bundled BoringSSL,
the same optional kTLS - and no UDP, QUIC, h2 or h3.
SO_REUSEPORTlistener and connection table.Handlers start on the reactor and resume inline there, and the per-reactor
SynchronizationContextbringsTask.Delay/HttpClient/Task.Runcontinuations home - as in ioxide.Reactor,ServerConfig/TcpOptions,TcpConnection(rawReadAsync/TryGetItem/ReturnBuffer,GetSpan/Advance/FlushAsync),TcpConnectionPipeReader/Writer,TcpConnectionDualPipe,TcpConnectionStream,ReactorTimer(for
RingTimer),TlsService/TlsSession/TlsConnectionDualPipe.Playground/Epoxide/{Raw,Pipe,ThreadPool,TlsPipes,KtlsPipes}, registered inbench/samples.tsv.ThreadPoolruns its whole handler on the pool.The difference: any thread may use a connection
An io_uring ring has one issuer, so ioxide hands every off-reactor operation to the reactor. epoll
has no such rule: a read is a
recvand a flush is asend, made on the caller's own thread. So ahandler that moves to the thread pool (
Task.Run,ConfigureAwait(false)) keeps reading andwriting its connection from there, and nothing is posted to the reactor.
Reactor.ConnectAsyncworks from any thread too, so clients can live on the pool.
Only a read with nothing to read, or a flush the socket cannot take, waits on the reactor: it
parks, and the reactor finishes the
recv/sendat the next readiness edge and resumes thecaller - inline when it is on that reactor, on its own context or the thread pool when it is not.
How it holds together:
recv(and who maysendthe rest of aflush) is handed between the caller and the reactor by a CAS state machine (Idle / Ready / Parked),
so exactly one of them touches the socket at a time and no edge is lost.
recvcounts as drained, so a request costs onerecvand onesend- strace over 220Krequests: 220,341 recv, 220,324 send, 13,844 epoll_wait. Not once an edge has reported the peer's
FIN or urgent data, and not under kTLS RX: those read to EAGAIN.
that keeps its buffer full cannot hold the reactor. ioxide gets that bound for free, since a
reader there drains only what the ring has delivered.
handler that may be inside a syscall on it from another thread (the reactor's own end shuts the
socket down and wakes the handler instead).
with the kernel), and only bytes a reader has not taken yet hold one.
synchronous crypto, instead of posting them to the reactor. Removing the lock brings back
ioxide's "decrypt failed (error 5)" race; the tests that catch it are in the suite.
Benchmark (plaintext, both over pipes)
Each sample was run with
bench/any.sh: wrk with 8 threads and 64 connections for 10 s, against2 reactors pinned to CPUs 2 and 6, with wrk on CPUs 0-1 and 12-15. The samples alternated
ioxide, epoxide, epoxide, ioxide for 5 rounds, so drift cancels out. CPU per request is the
server's own utime+stime from
/proc, over the requests it served, with the reactors at 90% ormore.
Tcp/PipevsEpoxide/PipeTcp/RawvsEpoxide/RawEvery ioxide run beat every epoxide run, in both workloads, and repeats of one sample within a
round differ by at most 5.3%, so the gap is real. It is the syscalls: epoxide makes a
recvand asendper request (strace: one of each), while ioxide's multishot recv and batched submits foldboth into its ring.
The fixes below cost nothing measurable.
Epoxide/Pipebefore them (4ba8f02) against after,alternated over 3 rounds the same way, is 853K against 846K req/s and 2.37 against 2.38 µs/req,
inside the noise.
The TLS samples (
Epoxide/TlsPipes,Epoxide/KtlsPipes) serve, but are not part of thisbenchmark, which is plaintext as asked.
Tests
tests-epoxide/- a separate folder, as asked - holds every ioxide test that covers what epoxidehas, ported, plus tests for what is epoxide's own. Run twice on the final commit, pinned:
343 passed, 0 failed, 0 skipped, both times. The cross-thread TLS tests, where the last bug below
showed, then ran 100 more times without a failure (it had failed 1 run in 40 before its fix).
Epoxide.Tests.UnitRingSetupMessageTestsbecomesConfigValidationTests, without the ring-only cases)Epoxide.Tests.E2ECore/tests, less the io_uring-only ones; new:ThreadPoolTests(12),ConnectTests(11), and regression tests for the bugs belowEpoxide.Tests.TlsEpoxide.Tests.Chaosexample, "recv-queue overflow closes the connection" becomes "a reader that falls behind
back-pressures its peer": epoxide only reads when asked, so its queue cannot overflow.
rings, ring setup, UDP, QUIC, h2, h3, and the clients.
assertion flipped to the defect's expectation, built and run, and every one failed. Each fix below
was also reverted against the tests that pin it.
The new thread-pool tests cover:
DecRefon the pool, in both orders;Stop()while pool handlers hold connections;Bugs the port turned up, fixed here
Three agents ported TLS, ported the tests, and reviewed the core adversarially. A stress loop of
the cross-thread TLS tests found one more. Each bug below was reproduced, and each fix is pinned by
a test that fails with the fix reverted, except the flush race, which needs ordering hooks to reach.
epoxide: a recv that takes the peer's FIN ...DecRefon the pool, or by another task mid-read, met the reactor's reclaimNullReferenceExceptionthat ended the reactor, or a buffer returned twice: a free-list cycle handing one buffer to every connection... keep their read state under a lock ...EndFlush/SetResulttwice... claims a parked flush with a CAS ...Stop()never drained registrations, posted continuations or timersConnectAsyncnever settled (20 of 20 rounds); a handler parked on a post or aReactorTimernever let go, and kept its descriptor openStop() settles what only its loop would have run ...Tcp == nullregistered no TCP sweep... with no listener sweeps its outbound connections ...... gives the loop back after 16 reads in a row ...intRecvBufferSizefailed every recv withOutOfMemoryException; a hugeMaxEventsdied inRun... sizes are computed past int ...The harness also had a race, in ioxide's harness too: a reactor death a test had already claimed
could still be reported as unseen. Fixed in the epoxide copy only.
Not in this PR
dotnet packstep, so merging releases nothing.in
RunGuarded.reactor to a peer that keeps up never sees EAGAIN, so it yields only once the socket fills. The
response size bounds it, and only the handler can cause it, never a peer. A flush budget like
the read side's would close it.