Repository navigation
GPU gates: A bench set that reports counters, bench timings and hot-path code of base against head - #207
Merged
Conversation
…base against head) ci/gpu/run.sh bench builds BASE and this checkout and reports three things: - A, counters: bench_draws 6000 600 present with D9MT_PROFILE=1, two runs per side, each side under its own bridge. scripts/bench-counters.py takes every counter of the last full PROF window and the last STATS line (no times) into counters.tsv, with the difference and whether each side's second run repeated it. - B, timings: scripts/perf-abba.sh (base, basecopy as the control, head) per bench and profile mode, perf-compare.py's report in summary.md under "diagnostic, no verdict", and the profile-on overhead per bench. It runs only when base's and head's bridges are the same bytes, since perf-abba.sh swaps d3d9.dll only. - C, hot code: scripts/cmp-hot-code.sh compares the functions of ci/gpu/hot-functions.txt in both d3d9.dll builds, x64 and i686, as same, moved only or changed, into hot-code.tsv. The set fails only when a step doesn't run or a run's log shows d9mt didn't draw it, never on a difference or a timing. Every bench call is a queue step of its own. The two new tests run in Checks: test-bench-counters.py on lines in d9mt's formats, test-cmp-hot-code.sh on small DLLs that move, change a constant, read another array element, call another function or change a lambda.
…OS pads wc's output)
…r it The first bench run (gpu-ci 43581084789) ended its gates in 13 minutes, then the result upload ran for two hours without an artifact: a-base and a-head were left in out/work with their Wine prefixes, whose z: drive is /. abba_step now copies the runs' stderr logs to <out>/<label>-logs/, deletes the folder, and the exit trap deletes the one running when the script is interrupted, as the hash set does with its suite folders.
Review (High): the padding pattern also dropped an int3 inside a function and a lea 0x0 whose destination isn't its source, so a nop that became a trap, or a lea that writes another register, read as moved only. Inside a function only the nop forms and a lea of a register into itself (the same register text, so a 32-bit lea on x64, which clears the upper half, isn't one) are padding now; int3 only after the last other instruction. test-cmp-hot-code.sh gets the three cases as raw bytes (the assembler drops a zero displacement); all three read moved only before this commit.
Review round 2 (High): the trailing strip still dropped an int3 at a function's end, where a branch inside the function can land, so a nop target that became a trap read as same. int3 is never padding now; the real x64 build has none and the x86 build five, so nothing else moves. test-cmp-hot-code.sh's tailfn (variant 9) is that case; it read same before this commit.
Control A (gpu-ci 43595813812: one more CS command in PrepareDraw) showed i686 DrawPrimitive and DrawIndexedPrimitive as changed +0. They don't change; the string literal and the exception table they point to moved in .rdata and .gcc_except_table, where the nearest symbol is the section's own, so the offset moved with anything that grew in front of them. The other side of it: a same-length change of such a string read as same. A reference whose nearest symbol is a plain section symbol is now written as the section and what it holds: the string (up to 256 bytes), or 16 bytes with each pointer-sized word that is an image address written as its symbol. A section symbol with a $ suffix (.rdata$.refptr.<name>, .text$<function>) names its contents and stays symbol plus offset. test-cmp-hot-code.sh gets strmoved (moved only) and strchange (changed); the previous code read them as changed and same. On the two real build pairs the results are back to the per-object analysis's: x64 StretchRect, GetMaxFlushType; x86 PrepareDraw, GetMaxFlushType.
…nserted data Review round 4 (2 High): - An 8-byte constant whose bytes start 41 41 00 read as the string 'AA', so two different doubles compared equal. Unnamed data an instruction reads is now compared by the bytes it reads, the width from the mnemonic or a register operand (16 when neither tells), and is never taken for a string. Only an address taken (lea, $) is read as the string that starts there, or 16 bytes. Keeping 16 raw bytes beside the string, the first idea, failed the moved controls: past a short string they reach the next constant, whose padding moves. - The text put in for one operand went through the later address passes again, so the strings '0x10001000' and '0x20001000' of builds at those image bases both became a symbol. Each replacement now waits in a placeholder until all passes are done. test-cmp-hot-code.sh: double (a constant with the same first bytes, changed) and addrstr (built at another image base, changed); every variant now builds at a fixed image base. 2bfa6af's and ae5c13d's code fail them; the real build pairs still give the per-object analysis's functions.
Control C on the M1 (gpu-ci 43598088877: one line in PrepareDraw on 9a8fec4) still showed i686 DrawPrimitive and DrawIndexedPrimitive as changed +0: their references into .gcc_except_table moved from +0x81e to +0x832. On i686 the linker puts .gcc_except_table into .text, a code section, so the contents rule (non-code sections only) never applied. Unnamed data is now any reference whose nearest symbol is a plain section symbol other than .text's own, wherever the linker put it. test-cmp-hot-code.sh's ehmoved and ehchange put unnamed data under a local label in .gcc_except_table; 9a8fec4's code reads them as changed and same.
main now prints PROF abVs/abPs (layout/resources/cbv/keep) and tableVs/tablePs (flushes/slots), STATS constVs/constPs (writes/changed), all as slash-separated numbers (#202). bench-counters.py would have kept them as text: equal or not, but no difference. Every a/b/... field is now split into counters, named after the parts main's comments give, numbered otherwise. test-bench-counters.py has lines in main's format; the previous parser fails it.
jdrly
force-pushed
the
ci/198-bench-set
branch
from
November 14, 2026 17:09
a8b10b8 to
f10d4d6
Compare
jdrly
marked this pull request as ready for review
November 14, 2026 17:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Every speed-relevant PR of 0.1.9 asks the same three questions: did a counter move, did the profile-off hot path change, and what do the benches say. Until now the answers came from
bench_drawsrun by hand over SSH, because the hash suite's five frames never reach aPROFwindow.ci/gpu/run.sh benchanswers them on the GPU gates, base against head, from d9mt's own synthetic benches only:bench_draws 6000 600 presentwithD9MT_PROFILE=1, two runs per side, each side under its own bridge.scripts/bench-counters.pyputs every counter of the last fullPROFwindow and the lastSTATSline intocounters.tsv, with the difference and arepeatcolumn. That column says whether each side's second run printed the same value, so a counter that depends on timing shows itself. Times are left out.scripts/perf-abba.shwith base, basecopy (the control) and head, 10 rounds, forbench_draws 6000 600andbindbench 1060, profiling off and on.perf-compare.py's report goes intosummary.mdas it is, under "diagnostic, no verdict", with the profile-on overhead per bench.perf-abba.shswapsd3d9.dllonly, so B runs only when base's and head's bridges are the same bytes. Otherwise B is skipped with a note.scripts/cmp-hot-code.shcompares the functions listed inci/gpu/hot-functions.txtin base's and head'sd3d9.dll, x64 and i686, assame,moved onlyorchangedwith the instruction-count difference. I addedDrawPrimitiveandMtlContextImpl::drawto the issue's starting list, becausebench_drawsdraws with them.The set fails only when a step doesn't run, or when a run's log shows d9mt didn't draw it or wasn't profiled. It never fails on a difference or a timing. Every bench call is a queue step of its own. Spec words:
benches=(empty: no B),rounds=,profile=off,on. The workflow side (benchamong thesetchoices, scratch folder outside the result folder) is a separate PR in the private gpu-ci repo.Closes #198
How to verify
python3 -I scripts/test-bench-counters.pyandscripts/test-cmp-hot-code.sh(mingw-w64, no Wine, no GPU). Both run inChecks.gh workflow run gates.yml --repo jdrly/gpu-ci -f set=bench -f sha=<head> -f base=<base>, then readsummary.md,counters.tsvandhot-code.tsvin the result folder.Evidence
Proof. Head
f10d4d6, rebased onto main9b64581without conflicts. The rebase changed nothing of this PR's own;f10d4d6adds one parser commit (review round 7).Hosted
Checks: green onf10d4d6(run 43619523533), and before the rebase ona8b10b8(43609576007). AIR codegen25fc2491…7a6c2, unchanged: no codegen file is touched.test-cmp-hot-code.shbuilds small DLLs from one C++ source, x64 and i686, in 15 variants against a base:moved only(orsame) cases: code moved by a function in front; a string literal pushed further into.rdata; unnamed data in.gcc_except_tablemoved behind a longer block.changedcases: a changed constant, the next element of the same array, a call to another function, anint3in place of anop, aleainto another register, a 32-bitleaon x64, a trap at a function's end that a branch reaches, a same-length string change, another double whose first bytes read like the same short string, an address-looking string in a build at another image base, and changed unnamed data in.gcc_except_table.Controls, run by me on jd-ai-server: seven deliberate breaks of
cmp-hot-code.sheach make the test fail, among them masked symbols, masked offsets, no target normalization,changednever reported and lambdas counted. Each earlier version of the script fails the variants added after it.test-bench-counters.pycovers main's slash-separated fields (abVs=a/b/c/d,tableVs,constVs), one counter per part; the parser beforef10d4d6fails that. It also covers one more CS command per draw, which givescsCmds/f+6000 at 6000 draws. It also checks that no time field gets through, thepassClose/f:group,wait=<n>/<ms>ms, text fields, a base older than the CS fields, and a run withoutPROForSTATS, which fails.On two existing unstripped d3d9.dll pairs (an older fix branch against its main),
cmp-hot-code.shreports the same changed functions as an earlier per-object analysis of the same builds: x64StretchRectandGetMaxFlushType, x86PrepareDrawandGetMaxFlushType. All other listed functions there aremoved only.Final acceptance run (Apple M1, macOS 27.0): gpu-ci 43619529644, main
9b64581(this PR's base) against headf10d4d6, default spec. Verdict: pass. It wrotecounters.tsv,summary.mdandhot-code.tsv. C:same 22. A: 104 counters with main's new fields, for examplePROF abVs.cbv5996 per frame; onlyevRead/fandallocMiBdiffer, neither repeats. B ran; every report carriescalibration: noneandguard: pending. AIR codegen25fc2491…on both sides. 0 IOGPUMetalError lines.First acceptance run (Apple M1, macOS 27.0): gpu-ci 43591740999, base
10e71fc(this PR's base) against head2bfa6af, default spec. Verdict: pass, 50 minutes including both builds. It wrotecounters.tsv,summary.mdandhot-code.tsv.same, since this PR doesn't touch the driver.PROF evRead/f(17.35 against 17.42) andSTATS allocMiB(44 against 42). Neither repeats between two runs of the same DLL, and therepeatcolumn says so. Everything else is equal and repeats, for exampledraws/f6000.calibration: noneandguard: pending. Profile-on overhead: bench_draws +9.3 % (base) and +9.5 % (head), bindbench +7.5 % and +7.5 %. These are two blocks per number, a diagnostic.Control A (one more CS command per draw,
EmitCs([] (MtlContext*) { });inPrepareDraw, never merged): gpu-ci 43610778322, ona8b10b8.PROF csCmds/f24028 → 30028, +6000, the draw count. With it,csChunks/f+9,csKiB/f+140.6 andSTATS flushChunk.refused+9 (fuller chunks).evRead/fandallocMiBdiffer again and don't repeat. C: onlyPrepareDrawischanged(x64 +63, i686 +82), the other 20 rowsmoved only.Control C (one line in
PrepareDraw,if (unlikely(PrimitiveType == D3DPT_FORCE_DWORD)) return;, never merged): gpu-ci 43609565168, ona8b10b8.PrepareDrawchanged(x64 −2, i686 −9), the other 20 rowsmoved only. Counters: onlyevRead/fandallocMiB, which don't repeat.The controls found two problems first, both fixed before the runs above. On
2bfa6afand on9a8fec4, i686DrawPrimitiveandDrawIndexedPrimitivereadchanged +0. They reference a string and their exception tables, unnamed data whose offsets moved becausePrepareDrawgrew; on i686 the exception tables sit inside.text. Unnamed data is now compared by what it holds (see the review).Three runs with base = head (
a8b10b8against itself, full set, 10 rounds), before anybody reads part B: 43611982347, 43614496034, 43616910711, all green, about 26 minutes each. Apple M1, macOS 27.0, a window visible throughout (no visibility change logged), no DIRTY run, nobimodal?flag.same 22in each. A: onlyevRead/fandallocMiBdiffer, and neither repeats, as in every other run.none.Performance (PERFORMANCE.md §1). Kind: CI tooling. No driver or bridge file changes, so nothing reaches a player.
Definition of Done
Checksand the M1 run built both from scratch.test-bench-counters.pyandtest-cmp-hot-code.sh, with controls (above).Checks.dxvk.confoption. The new spec words are inci/gpu/run.sh's header anddocs/testing.md.CONTEXT.md.f10d4d6(below).a8b10b8.Review
Codex (gpt-6.1-sol, reasoning high), seven rounds.
cmp-hot-code.shdropped too much as padding. An interiorint3and alea 0x0into another register read asmoved only. Fixed in1168e6a. The same pattern had a third case, a 32-bitleaon x64, which clears the upper half; the fix covers it, and a test variant checks it. Only the nop forms and a lea of a register into itself count as padding now.int3at a function's end that a branch reaches was still dropped. Fixed in2bfa6af:int3is never padding. The real x64 build has none, so nothing else moves.2bfa6af.0x10001000and0x20001000of two builds at those image bases became one symbol. Fixed in9a8fec4: data an instruction reads is compared by the bytes it reads, with the width taken from the mnemonic or register; only an address taken (lea,$) is read as a string; and every replacement waits in a placeholder until all passes are done.9a8fec4..gcc_except_tableinto.text, so unnamed data is now told by its symbol, not by its output section (a8b10b8). No findings, APPROVE ata8b10b8.bench-counters.pysplits main's new slash-separated fields into one counter per part (f10d4d6). No findings. Verdict: APPROVE atf10d4d6.Risks and what I didn't do
bimodal?flag, which by §2.5 doesn't rule out a per-block mode.changed, never a hidden one).cmp-hot-code.sh's header says so.DIRTY_CHECK=0: counts don't depend on another Wine server running beside them. B keeps the check.z:drive is/, were left in the result folder'swork/. I cancelled it, and its log was lost. Each bench call now deletes its folder, and the gpu-ci PR moves the scratch folder out of the result folder.How to revert
Revert the squash commit. Then remove
benchfrom the gpu-ci workflow'ssetchoices, or leave it: a dispatch ofset=benchagainst a commit without the set ends with "unknown set". Nothing outside the two repos is written.