Skip to content

Add portable public-source TCP churn experiment - #898

Draft
Michael Friesen (mtfriesen) wants to merge 218 commits into
mainfrom
user/mtfriesen/tcp-churn-public
Draft

Michael Friesen (mtfriesen) wants to merge 218 commits into
mainfrom
user/mtfriesen/tcp-churn-public

Conversation

@mtfriesen

@mtfriesen Michael Friesen (mtfriesen) commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Portable public-source TCP churn workload with separate Windows, Netperf fixture and lab-guest adapters.

Actual netperf-machine execution: verified

https://github.com/microsoft/netperf/actions/runs/40371149553

  • Successful management job on RR1-Netperf-25 using the documented parent=rr1-netperf-26\localadminuser label.
  • The workload ran INSIDE the existing netperf-windows-2022-client VM via PowerShell Direct, not on the management host or a GitHub-hosted runner.
  • Uses configured VM credentials without logging/publishing them; refuses an active Actions worker/CTS in the guest. No host networking or runner registration is changed.
  • The thin wrapper uses Windows PowerShell 5.1, matching the actual management host. The public workload executes under PowerShell 7 inside the VM.
  • 512 completed connections, 33,554,432 payload bytes, 33,754,112-byte TX and RX adapter deltas.
  • Owned fixture/process/firewall/device cleanup succeeded. Downloaded artifact contains ONLY numeric result.json.
  • CTS interval was 85 ms. This is deployment and data-path proof, NOT sustained performance, 80-CPU coverage or an optimization win.

The proven workload revision is 26c2f5f; subsequent documentation records this evidence.

Scheduling diagnosis and alternatives

Today's successful eBPF jobs used exactly [self-hosted,lab,os-windows-2022,x64], matching the initial dispatch. Other lab workloads were also queued, so labels were not assumed to be wrong.

The existing targeted reset procedure completed successfully for the documented Windows client slot, respecting its active-job safety gate. Neither the original labels nor an explicit image-independent Windows-lab target obtained a guest-runner assignment. The documented management runner did execute, enabling the successful guest adapter without modifying registration.

Obsolete unassigned smoke jobs were cancelled after hardware success. Normal guest-runner scheduling, hosted-smoke compatibility and management-to-guest execution remain explicitly separate targets.

Public/private and artifact contract

No private OS implementation/helper/internal feature identifier or unpublished binary is included. No new executable is introduced. CTS and Microsoft-signed DuoNIC are existing public assets, pinned by public commit and SHA256.

Native tools remain local to the execution guest. Only numeric allowlisted JSON is uploaded: no binaries, raw logs, traces, host inventory, credentials or adapter state. Workflow and publication scope are reviewed accordingly.

Portability

The core module owns plans, public CTS summary parsing, lifecycle and progress invariants. Five callbacks supply peer start/readiness, counters, client run and cleanup.

Windows execution owns processes/NIC observation. Netperf fixture provisioning and the management-to-guest transport are separate adapters. Manual/internal runners can use the same core without GitHub provisioning or publication.

Validation and remaining limits

45 public-source checks pass locally and in hosted CI; actual netperf guest provisioning/traffic/cleanup has now also passed. Workflow syntax and source-only publication checks pass.

Finite iterations cap the smoke at a port-budgeted connection count plus a deadline; no hidden port-reuse tuning. The result remains collection-only, cpuCoverage/targetBranchCoverage are not measured, and mechanismVerified=false. Selective PMTU faults, routed DGD, whole-system CPU/lock-site evidence and sustained comparative campaigns remain future work.

Separate workload and result invariants from Windows CTS and netperf provisioning. Pin pre-existing public tools by commit and hash; upload numeric allowlist only, never binaries or raw diagnostics. Keep lab execution explicit and collection claims bounded. Forty public-source checks pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use finite CTS iterations with an explicit aggregate connection budget as well as a wall-time ceiling. Keep sustained-load resource/reuse policy outside this deployment smoke.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep disposable lab provisioning separate from manual preconfigured adapters and reject pre-existing alias/subnet ownership before downloading or installing tooling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep hosted deployment/traffic smoke separate from self-hosted hardware performance and never reset unrelated lab machines for the hosted target.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep raw diagnostics local while distinguishing installation, adapter, address and firewall stages and recording bounded numeric state and tool exit codes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a source contract check for non-interactive rule creation after the actual hosted DuoNIC deployment reached this stage successfully.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mtfriesen

Copy link
Copy Markdown
Collaborator Author

Deployment verified on the real public DuoNIC fixture: https://github.com/microsoft/netperf/actions/runs/40013903384 . All 45 source checks, traffic, and owned-resource cleanup passed. The only published artifact contains allowlisted result.json: 512 completed connections, 33554432 payload bytes, matching adapter TX/RX deltas of 33754112 bytes. The finite-iteration traffic interval was 74 ms; this is a compatibility smoke, not a sustained performance claim. Hardware-lab retry https://github.com/microsoft/netperf/actions/runs/40339654787 remains queued after successful source checks. No new binaries were committed or uploaded.

Retain self-hosted Windows lab x64 requirements and distinct hosted validation. Existing 2022-only label combinations match recent successful eBPF jobs but are currently not assigning runners.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nt slot

Run the public smoke within the existing VM using configured credentials and PowerShell Direct. Refuse active guest workers, never install tooling on the management host, and leave runner configuration unchanged. Copy public text sources only and retain numeric-only publication.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep management-host execution manual only; branch pushes run hosted contract tests without VM credentials.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The documented management runner does not have pwsh. Keep only the thin PowerShell Direct wrapper PS5-compatible; the portable experiment still executes under pwsh inside the guest.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the working management-runner/PowerShell Direct configuration and numeric traffic evidence, distinct from hosted validation and unassigned direct-runner jobs. Keep short deployment smoke limits explicit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mtfriesen

Copy link
Copy Markdown
Collaborator Author

Verified execution on the actual netperf hardware: https://github.com/microsoft/netperf/actions/runs/40371149553 . The job ran on RR1-Netperf-25 and used PowerShell Direct to execute inside its existing netperf-windows-2022-client VM. Both source checks and guest smoke succeeded. Numeric artifact: 512 connections, 33554432 payload bytes, matching 33754112-byte adapter TX/RX deltas; cleanup succeeded. This is not the hosted fallback. The working configuration uses the documented management-runner label and Windows PowerShell on the host, with pwsh inside the guest. No runner-registration or host networking change, and no binary artifacts uploaded. The 85-ms finite smoke is not a sustained performance result.

Build a new public-source TraceEvent analyzer, resolve only public Microsoft system symbols and publish bounded aggregate frames, never raw ETLs. Separate idle/load intervals and repeated finite port-budgeted churn batches; preserve scoped WPR/fixture cleanup. Existing public CTS and OS binaries are not uploaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the hosted compiler dependency explicitly without changing the analysis contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…aw traces

Distinguish traffic/capture failures from symbol/stack analysis and retain aggregate progress for unattended iteration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain partial-window measurements and fixed public failure categories without turning resource failures into successful traffic results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TraceEvent exposes a typed event enumerator but intentionally throws from its nongeneric enumerator. Avoid LINQ OfType over that path; retain safe analyzer error-site evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…assuming defaults

Publish bounded full frame counts and spin-leaf stacks, and read only the public BH override and public event manifest. Keep registry state unchanged and distinguish missing override from disabled detection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Discover event identities/keywords from the running Windows public manifest and use an owned bounded session. Retain raw event and CPU traces in the guest; publish event counts only and do not equate generic path updates with BH.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ailure

Keep private paths and raw messages out of artifacts while locating pre-capture configuration failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ures

Stop WPR before analysis even when probe startup fails; do not dereference an interval that was never entered. Publish the actual bounded start-stage exit code instead of masking it with a missing-property error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rted limit

Replace overflowing keyword-only capture with a watchdog-bounded owned event-ID collector. Record actual RSS configuration and sampled network processors; the public DuoNIC driver caps this fixture at 16 RSS processors, not 80.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…measured batch

Do not mistake transient DAD state for workload failure. Allow profiled batches their explicit finite deadline rather than the short smoke deadline, while keeping campaign and batch caps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… budget

Use eight owned source addresses with scoped peer routes/firewall matching, without global TCP tuning. Retain bounded caller stacks per public network function so low-frequency path updates are not discarded by a global top-stack cutoff.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…king

Scope firewall loss to the disposable data fixture, verify failed attempt outcomes and public retry/timeout events, retain CPU samples and always remove the owned rule. Do not call this PMTU probing or a successful healthy measurement.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use ActiveStore for new measurement addresses/routes, verify matching configuration when an owned software-device instance returns with persisted state, and remove exact owned source entries from both stores before deleting devices.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tests

Use existing configured credentials and remoting without changing peer state; publish only fixed connectivity status and logical processor count. Same-host SYN-drop results are not treated as a valid outage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pter seam

Use the verified idle peer session and route-selected endpoints, keep client profiling unchanged, and own/remove peer server and port-scoped firewall rules. Pass credentials only through a same-machine DPAPI-protected ephemeral handoff that is immediately removed; never publish raw traces or credentials.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the idle-peer fault scoped to its test port, protect client-scoped remoting before temporarily enabling disabled profiles, and restore original profile states before removing owned rules. Record numeric control and outcome evidence; do not accept ignored block rules as an outage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The lab data path is roughly 1 GbE, so 16 KiB responses could not finish before the 150 s producer deadline and the aborted producer reset its connections. Hot mode now carries 512 exchanges of one 1460-byte response each, raising the exchange and segment rate within the same bandwidth. Failed receiver runs also record the remote producer's exit and error facts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Failed endpoints now report the UTC time, phase, remote address and socket error of their earliest connection failure, and receiver failures wait briefly for the remote producer so both sides can be ordered.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The first-error report showed the server resetting remote connections about 18 s after GC while ~119 connections stalled in retransmission backoff: offered load exceeded the ~1 GbE path and single-segment flights can only recover by RTO. Hot mode now uses 384 exchanges at 50-130 ms (about 0.7 Gbps).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Failed receiver runs without TCP probes now keep whole-interval CPU, spinlock and post-GC analysis, and record data-NIC discard and error deltas, so a mid-run stall can be attributed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Every hot failure was a client send timeout ~19 s after GC on connections still waiting un-accepted beyond the 4096-slot server. Hot mode caps the remote producer at 3072 in flight (plus 1024 loopback) and uses 256 exchanges so three waves fit the producer deadline.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Starts a 3 s, 2 ms dial at the receiver's pre-collection signal so invalidations land while the application is stopped and only timer-driven TCP sends run, matching the customer's timer-only contention signature after a long STW GC.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read-only netstat TCP counters (opens, failures, resets, segments, retransmissions) and data-NIC discards are captured before and after cluster traffic on the profiled node and peer, in both accepted and failed results, to locate post-GC segment loss.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… job timeout

A 2 GiB spinlock capture exceeded the 15 minute job timeout, and cancellation skipped the guest finally block. The guest job now allows 35 minutes. A separate dispatch-only workflow inventories owned resources and, when requested, stops orphaned owned processes and replays each orphaned run's own snapshot-based remove actions; only exclusively named rules and the owned dial route are removed by name.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tances by name

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A cancelled owned recording and a stop that failed on a full disk left 19.8 GB of WPR temporary ETL in the client temp root. With restore, top-level WPR-named .etl files written since a given UTC time are discarded only when no recording session is active and no process holds the file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The guest profile keeps one-segment responses; hosts with high-bandwidth private links can carry full 16 KiB responses at the hot exchange rate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Large-node experiments need more concurrently established connections than the ordinary 8192/4096 bounds; only the hot service descriptor may use up to 32768 connections and 16384 in flight, with a matching listen backlog hint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
At 4x scale the last few connections finished just after the fixed 150 s deadline; hot service mode now allows 330 s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explicit source binds with port 0 draw globally unique ports from the 16384-port dynamic range, which exhausted at ~16k concurrent cluster connections (WSAENOBUFS). Cluster-source sockets now set SO_REUSE_UNICASTPORT so the port is chosen at connect and shared by 4-tuple; no global TCP setting changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SO_REUSE_UNICASTPORT still reserved a unique dynamic port at bind without an auto-reuse range, capping the producer near 16k connections. Cluster sources now bind monotonically increasing explicit ports below the dynamic range; explicit binds conflict only on the same address and port, so each owned source address has an independent port space and no 4-tuple repeats within a run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Larger topologies need more distinct remote peers, and therefore more cached paths per node. The guest cluster helpers still own only the /22 subset.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
servicehold40:pipelined keeps a large population of long-lived connections established at once (one 512-byte request per 1-2 s for about a minute each), allowing up to 131072 connections in flight. It models pooled database connections whose timers stay active while mostly idle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Thousands of cluster source addresses exceed the Windows command-line limit; an @file argument supplies the same comma-separated list.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per-address cursors starting at the same port made every port number shared by thousands of source addresses, and explicit-bind cost grew to ~430 ms, starving the producer thread pool. A single global cursor over a 28999-port span (coprime to the source count) keeps only a few endpoints per port number and never repeats an address:port pair within a run; excluded ports are skipped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Large held populations reach their GC trigger more than two minutes after the dial starts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Held connections at 250-500 ms pacing and service wait, so delayed-ACK timer sends dominate while the application is paused.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NDIS R/W locks inline into the path query and wait in KxWaitForSpinLockAndAcquire rather than KxWaitForLockOwnerShip; results for RTL-lock traces are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant