Skip to content

GPU gates: A bench set that reports counters, bench timings and hot-path code of base against head - #207

Merged
jdrly merged 9 commits into
mainfrom
ci/198-bench-set
Nov 14, 2026
Merged

jdrly merged 9 commits into
mainfrom
ci/198-bench-set

Conversation

@jdrly

@jdrly jdrly commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

What changes

Every speed-relevant PR of 0.1.9 asks the same three questions: did a counter move, did the profile-off hot path change, and what do the benches say. Until now the answers came from bench_draws run by hand over SSH, because the hash suite's five frames never reach a PROF window. ci/gpu/run.sh bench answers them on the GPU gates, base against head, from d9mt's own synthetic benches only:

  • A, counters (exact). bench_draws 6000 600 present with D9MT_PROFILE=1, two runs per side, each side under its own bridge. scripts/bench-counters.py puts every counter of the last full PROF window and the last STATS line into counters.tsv, with the difference and a repeat column. That column says whether each side's second run printed the same value, so a counter that depends on timing shows itself. Times are left out.
  • B, timings (diagnostic). scripts/perf-abba.sh with base, basecopy (the control) and head, 10 rounds, for bench_draws 6000 600 and bindbench 1060, profiling off and on. perf-compare.py's report goes into summary.md as it is, under "diagnostic, no verdict", with the profile-on overhead per bench. perf-abba.sh swaps d3d9.dll only, so B runs only when base's and head's bridges are the same bytes. Otherwise B is skipped with a note.
  • C, hot-path code (exact). scripts/cmp-hot-code.sh compares the functions listed in ci/gpu/hot-functions.txt in base's and head's d3d9.dll, x64 and i686, as same, moved only or changed with the instruction-count difference. I added DrawPrimitive and MtlContextImpl::draw to the issue's starting list, because bench_draws draws with them.

The set fails only when a step doesn't run, or when a run's log shows d9mt didn't draw it or wasn't profiled. It never fails on a difference or a timing. Every bench call is a queue step of its own. Spec words: benches= (empty: no B), rounds=, profile=off,on. The workflow side (bench among the set choices, scratch folder outside the result folder) is a separate PR in the private gpu-ci repo.

Closes #198

  • ADR: none (ADR-0007: timings are diagnostics)

How to verify

  • python3 -I scripts/test-bench-counters.py and scripts/test-cmp-hot-code.sh (mingw-w64, no Wine, no GPU). Both run in Checks.
  • gh workflow run gates.yml --repo jdrly/gpu-ci -f set=bench -f sha=<head> -f base=<base>, then read summary.md, counters.tsv and hot-code.tsv in the result folder.

Evidence

Proof. Head f10d4d6, rebased onto main 9b64581 without conflicts. The rebase changed nothing of this PR's own; f10d4d6 adds one parser commit (review round 7).

  • Hosted Checks: green on f10d4d6 (run 43619523533), and before the rebase on a8b10b8 (43609576007). AIR codegen 25fc2491…7a6c2, unchanged: no codegen file is touched.

  • test-cmp-hot-code.sh builds small DLLs from one C++ source, x64 and i686, in 15 variants against a base:

    • moved only (or same) cases: code moved by a function in front; a string literal pushed further into .rdata; unnamed data in .gcc_except_table moved behind a longer block.
    • changed cases: a changed constant, the next element of the same array, a call to another function, an int3 in place of a nop, a lea into another register, a 32-bit lea on x64, a trap at a function's end that a branch reaches, a same-length string change, another double whose first bytes read like the same short string, an address-looking string in a build at another image base, and changed unnamed data in .gcc_except_table.
    • A lambda inside a listed function isn't taken for it.

    Controls, run by me on jd-ai-server: seven deliberate breaks of cmp-hot-code.sh each make the test fail, among them masked symbols, masked offsets, no target normalization, changed never reported and lambdas counted. Each earlier version of the script fails the variants added after it.

  • test-bench-counters.py covers main's slash-separated fields (abVs=a/b/c/d, tableVs, constVs), one counter per part; the parser before f10d4d6 fails that. It also covers one more CS command per draw, which gives csCmds/f +6000 at 6000 draws. It also checks that no time field gets through, the passClose/f: group, wait=<n>/<ms>ms, text fields, a base older than the CS fields, and a run without PROF or STATS, which fails.

  • On two existing unstripped d3d9.dll pairs (an older fix branch against its main), cmp-hot-code.sh reports the same changed functions as an earlier per-object analysis of the same builds: x64 StretchRect and GetMaxFlushType, x86 PrepareDraw and GetMaxFlushType. All other listed functions there are moved only.

  • Final acceptance run (Apple M1, macOS 27.0): gpu-ci 43619529644, main 9b64581 (this PR's base) against head f10d4d6, default spec. Verdict: pass. It wrote counters.tsv, summary.md and hot-code.tsv. C: same 22. A: 104 counters with main's new fields, for example PROF abVs.cbv 5996 per frame; only evRead/f and allocMiB differ, neither repeats. B ran; every report carries calibration: none and guard: pending. AIR codegen 25fc2491… on both sides. 0 IOGPUMetalError lines.

  • First acceptance run (Apple M1, macOS 27.0): gpu-ci 43591740999, base 10e71fc (this PR's base) against head 2bfa6af, default spec. Verdict: pass, 50 minutes including both builds. It wrote counters.tsv, summary.md and hot-code.tsv.

    • C: all 22 rows (11 functions × 2 archs) are same, since this PR doesn't touch the driver.
    • A: 73 counters. Two differ between base and head, PROF evRead/f (17.35 against 17.42) and STATS allocMiB (44 against 42). Neither repeats between two runs of the same DLL, and the repeat column says so. Everything else is equal and repeats, for example draws/f 6000.
    • B ran (the bridges are the same bytes from both checkouts). Each report carries calibration: none and guard: pending. Profile-on overhead: bench_draws +9.3 % (base) and +9.5 % (head), bindbench +7.5 % and +7.5 %. These are two blocks per number, a diagnostic.
    • 0 IOGPUMetalError lines.
  • Control A (one more CS command per draw, EmitCs([] (MtlContext*) { }); in PrepareDraw, never merged): gpu-ci 43610778322, on a8b10b8. PROF csCmds/f 24028 → 30028, +6000, the draw count. With it, csChunks/f +9, csKiB/f +140.6 and STATS flushChunk.refused +9 (fuller chunks). evRead/f and allocMiB differ again and don't repeat. C: only PrepareDraw is changed (x64 +63, i686 +82), the other 20 rows moved only.

  • Control C (one line in PrepareDraw, if (unlikely(PrimitiveType == D3DPT_FORCE_DWORD)) return;, never merged): gpu-ci 43609565168, on a8b10b8. PrepareDraw changed (x64 −2, i686 −9), the other 20 rows moved only. Counters: only evRead/f and allocMiB, which don't repeat.

  • The controls found two problems first, both fixed before the runs above. On 2bfa6af and on 9a8fec4, i686 DrawPrimitive and DrawIndexedPrimitive read changed +0. They reference a string and their exception tables, unnamed data whose offsets moved because PrepareDraw grew; on i686 the exception tables sit inside .text. Unnamed data is now compared by what it holds (see the review).

  • Three runs with base = head (a8b10b8 against itself, full set, 10 rounds), before anybody reads part B: 43611982347, 43614496034, 43616910711, all green, about 26 minutes each. Apple M1, macOS 27.0, a window visible throughout (no visibility change logged), no DIRTY run, no bimodal? flag.

    • C: same 22 in each. A: only evRead/f and allocMiB differ, and neither repeats, as in every other run.
    • Paired effect of identical binaries inside one block, head and basecopy against base, all 24 numbers: −0.80 % to +0.69 %, with up to 9/10 rounds on one side (bindbench, profile off, third run: basecopy +0.69 %). The final acceptance run's basecopy, a byte copy of base, reads +0.84 % on bindbench with profiling on.
    • Between blocks the same binary's median moves much more. bench_draws, profile off, base: 2.590, 2.713 and 2.713 ms (+4.7 %). bindbench, profile off, base: 2.894, 3.010 and 2.931 ms (+4.0 %).
    • Profile-on overhead (two blocks per number): bench_draws +7.7 to +8.8 %, bindbench +6.3 to +8.3 %.
    • So on this M1, a paired difference inside about ±0.85 % on these benches is what identical binaries show, and one block's numbers don't carry over to the next. I read B with that in mind. Nothing here is a guard: calibration stays none.

Performance (PERFORMANCE.md §1). Kind: CI tooling. No driver or bridge file changes, so nothing reaches a player.

Definition of Done

  • Clean builds: no driver or bridge source changes. The hosted Checks and the M1 run built both from scratch.
  • Hash suite: not affected (no driver change).
  • Tests: test-bench-counters.py and test-cmp-hot-code.sh, with controls (above).
  • Native tests: unchanged, green in Checks.
  • Performance evidence: CI tooling, no runtime change.
  • AIR codegen id: main's.
  • No new environment variable or dxvk.conf option. The new spec words are in ci/gpu/run.sh's header and docs/testing.md.
  • Terms follow CONTEXT.md.
  • Independent agent review: approved at f10d4d6 (below).
  • Controls A and C on a8b10b8.
  • The three base = head runs (above).

Review

Codex (gpt-6.1-sol, reasoning high), seven rounds.

  • Round 1, High, cmp-hot-code.sh dropped too much as padding. An interior int3 and a lea 0x0 into another register read as moved only. Fixed in 1168e6a. The same pattern had a third case, a 32-bit lea on x64, which clears the upper half; the fix covers it, and a test variant checks it. Only the nop forms and a lea of a register into itself count as padding now.
  • Round 2, High, partly open: an int3 at a function's end that a branch reaches was still dropped. Fixed in 2bfa6af: int3 is never padding. The real x64 build has none, so nothing else moves.
  • Round 3: no findings, APPROVE at 2bfa6af.
  • Round 4 reviewed a change the controls forced: unnamed data (string literals, constants, exception tables) compared by contents instead of by an offset that moves with anything in front of it. Two High findings. An 8-byte constant read as a short string, so two different doubles compared equal. And text put in for one operand was rewritten by a later pass, so the strings 0x10001000 and 0x20001000 of two builds at those image bases became one symbol. Fixed in 9a8fec4: data an instruction reads is compared by the bytes it reads, with the width taken from the mnemonic or register; only an address taken (lea, $) is read as a string; and every replacement waits in a placeholder until all passes are done.
  • Round 5: no findings, APPROVE at 9a8fec4.
  • Round 6 reviewed the fix control C forced: on i686 the linker puts .gcc_except_table into .text, so unnamed data is now told by its symbol, not by its output section (a8b10b8). No findings, APPROVE at a8b10b8.
  • Round 7, after the rebase onto main: bench-counters.py splits main's new slash-separated fields into one counter per part (f10d4d6). No findings. Verdict: APPROVE at f10d4d6.

Risks and what I didn't do

  • B's timings are a diagnostic under ADR-0007. The three base = head runs give a first spread on that M1 (±0.85 % paired inside a block, up to about 5 % between blocks), from three blocks only. PERFORMANCE.md §2.5's bimodal behaviour was measured on an M5 Max; these runs showed no bimodal? flag, which by §2.5 doesn't rule out a per-block mode.
  • B is skipped when the bridges differ. A PR that changes the bridge gets A and C, not B.
  • C compares instructions and the unnamed data they read (the bytes read, or the string or 16 bytes an address points to), not the contents of named data: a variable's value or a jump table shows only if where the function reads it from changes. Two unnamed objects with the same compared bytes are interchangeable, and a 16-byte fallback can reach a neighbour (a false changed, never a hidden one). cmp-hot-code.sh's header says so.
  • A uses DIRTY_CHECK=0: counts don't depend on another Wine server running beside them. B keeps the check.
  • The first bench run (gpu-ci 43581084789) ended its gates in 13 minutes, then hung in the result upload, because the bench calls' Wine prefixes, whose z: drive is /, were left in the result folder's work/. I cancelled it, and its log was lost. Each bench call now deletes its folder, and the gpu-ci PR moves the scratch folder out of the result folder.

How to revert

Revert the squash commit. Then remove bench from the gpu-ci workflow's set choices, or leave it: a dispatch of set=bench against a commit without the set ends with "unknown set". Nothing outside the two repos is written.

@jdrly jdrly self-assigned this Nov 14, 2026
…base against head)

ci/gpu/run.sh bench builds BASE and this checkout and reports three things:

- A, counters: bench_draws 6000 600 present with D9MT_PROFILE=1, two runs per side, each
  side under its own bridge. scripts/bench-counters.py takes every counter of the last full
  PROF window and the last STATS line (no times) into counters.tsv, with the difference and
  whether each side's second run repeated it.
- B, timings: scripts/perf-abba.sh (base, basecopy as the control, head) per bench and
  profile mode, perf-compare.py's report in summary.md under "diagnostic, no verdict", and the
  profile-on overhead per bench. It runs only when base's and head's bridges are the same
  bytes, since perf-abba.sh swaps d3d9.dll only.
- C, hot code: scripts/cmp-hot-code.sh compares the functions of ci/gpu/hot-functions.txt in
  both d3d9.dll builds, x64 and i686, as same, moved only or changed, into hot-code.tsv.

The set fails only when a step doesn't run or a run's log shows d9mt didn't draw it, never on
a difference or a timing. Every bench call is a queue step of its own. The two new tests run in
Checks: test-bench-counters.py on lines in d9mt's formats, test-cmp-hot-code.sh on small DLLs
that move, change a constant, read another array element, call another function or change a
lambda.
…r it

The first bench run (gpu-ci 43581084789) ended its gates in 13 minutes, then the result upload ran for two hours without an artifact: a-base and a-head were left in out/work with their Wine prefixes, whose z: drive is /. abba_step now copies the runs' stderr logs to <out>/<label>-logs/, deletes the folder, and the exit trap deletes the one running when the script is interrupted, as the hash set does with its suite folders.
Review (High): the padding pattern also dropped an int3 inside a function and a lea 0x0 whose destination isn't its source, so a nop that became a trap, or a lea that writes another register, read as moved only. Inside a function only the nop forms and a lea of a register into itself (the same register text, so a 32-bit lea on x64, which clears the upper half, isn't one) are padding now; int3 only after the last other instruction. test-cmp-hot-code.sh gets the three cases as raw bytes (the assembler drops a zero displacement); all three read moved only before this commit.
Review round 2 (High): the trailing strip still dropped an int3 at a function's end, where a branch inside the function can land, so a nop target that became a trap read as same. int3 is never padding now; the real x64 build has none and the x86 build five, so nothing else moves. test-cmp-hot-code.sh's tailfn (variant 9) is that case; it read same before this commit.
Control A (gpu-ci 43595813812: one more CS command in PrepareDraw) showed i686 DrawPrimitive and DrawIndexedPrimitive as changed +0. They don't change; the string literal and the exception table they point to moved in .rdata and .gcc_except_table, where the nearest symbol is the section's own, so the offset moved with anything that grew in front of them. The other side of it: a same-length change of such a string read as same.

A reference whose nearest symbol is a plain section symbol is now written as the section and what it holds: the string (up to 256 bytes), or 16 bytes with each pointer-sized word that is an image address written as its symbol. A section symbol with a $ suffix (.rdata$.refptr.<name>, .text$<function>) names its contents and stays symbol plus offset. test-cmp-hot-code.sh gets strmoved (moved only) and strchange (changed); the previous code read them as changed and same. On the two real build pairs the results are back to the per-object analysis's: x64 StretchRect, GetMaxFlushType; x86 PrepareDraw, GetMaxFlushType.
…nserted data

Review round 4 (2 High):
- An 8-byte constant whose bytes start 41 41 00 read as the string 'AA', so two different doubles compared equal. Unnamed data an instruction reads is now compared by the bytes it reads, the width from the mnemonic or a register operand (16 when neither tells), and is never taken for a string. Only an address taken (lea, $) is read as the string that starts there, or 16 bytes. Keeping 16 raw bytes beside the string, the first idea, failed the moved controls: past a short string they reach the next constant, whose padding moves.
- The text put in for one operand went through the later address passes again, so the strings '0x10001000' and '0x20001000' of builds at those image bases both became a symbol. Each replacement now waits in a placeholder until all passes are done.
test-cmp-hot-code.sh: double (a constant with the same first bytes, changed) and addrstr (built at another image base, changed); every variant now builds at a fixed image base. 2bfa6af's and ae5c13d's code fail them; the real build pairs still give the per-object analysis's functions.
Control C on the M1 (gpu-ci 43598088877: one line in PrepareDraw on 9a8fec4) still showed i686 DrawPrimitive and DrawIndexedPrimitive as changed +0: their references into .gcc_except_table moved from +0x81e to +0x832. On i686 the linker puts .gcc_except_table into .text, a code section, so the contents rule (non-code sections only) never applied. Unnamed data is now any reference whose nearest symbol is a plain section symbol other than .text's own, wherever the linker put it. test-cmp-hot-code.sh's ehmoved and ehchange put unnamed data under a local label in .gcc_except_table; 9a8fec4's code reads them as changed and same.
main now prints PROF abVs/abPs (layout/resources/cbv/keep) and tableVs/tablePs (flushes/slots), STATS constVs/constPs (writes/changed), all as slash-separated numbers (#202). bench-counters.py would have kept them as text: equal or not, but no difference. Every a/b/... field is now split into counters, named after the parts main's comments give, numbered otherwise. test-bench-counters.py has lines in main's format; the previous parser fails it.
@jdrly
jdrly marked this pull request as ready for review November 14, 2026 17:46
@jdrly
jdrly merged commit 002b1c6 into main Nov 14, 2026
5 checks passed
@jdrly
jdrly deleted the ci/198-bench-set branch November 14, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GPU gates: A bench set that reports counters, bench timings and hot-path code of base against head

1 participant