perf: batched directory enumeration

[Illustar0]

Note

Closes #95.
While the diff stat shows +5,694 / -1,128 across 65 commits, over 3,600 of the added lines are unit tests, differential tests against native ntdll, and benchmarks (ntdll.cpp itself shrinks by nearly 600 lines, and the core implementation is cleanly isolated in directory_records.{h,cpp} and directory_scan.{h,cpp}).
Every commit is atomic, builds cleanly, and passes the test suite on its own. That said, if the MO2 team would prefer to review this as a series of smaller staged PRs, I'd be happy to split it right away!

Overview

Previously, when enumerating virtual entries in a mapped directory, hook_NtQueryDirectoryFile(Ex) operated one virtual file at a time: for every virtual match, it opened the source mod directory via CreateFileW (re-entering USVFS's own open and attribute hooks under the HookContext lock), called NtQueryDirectoryFile twice, closed the handle, and returned at most one virtual record per caller buffer. During game startup with hundreds of thousands of virtual candidates, this caused a massive storm of redundant syscalls, hook re-entrancy, and lock acquisitions.

ntdll.cpp hasn't seen any fundamental architectural rework since roughly 2016, making it increasingly painful to keep piling workarounds on top of it—and the existing design itself had become a ceiling on further optimization. So I had to first "make the change easy" before making the performance improvements.

In this PR, I reimplemented the NtQueryDirectoryFile(Ex) hooks around a per-handle DirectoryScan and achieved major performance gains:

• Normalizes both entry points into a unified DirectoryQuery.
• RegularEntryReader synchronously reads the physical directory on disk and filters out entries shadowed by virtual ones.
• VirtualEntryReader reads each mapping source directory at most once per scan.
• DirectoryOutput serves as the sole writer to the caller's buffer, handling record alignment, NextEntryOffset linking, and truncation in one place.

Along the way, this PR fixes a wide range of long-standing edge cases in the hooks to match native ntdll semantics, backed by differential tests against ntdll.

Performance

Test environment: JueLun v6.1 modlist on an i5-12600K with 529 active plugins, CRDW in ACCELERATE mode (to avoid Skyrim's redundant startup directory traversals), and the latest versions of OAR, KID, and SPID. Measured via interleaved, paired A/B startup benchmarks with a warm filesystem cache after initial warm-up.

Launch Time Overview	Kernel Syscall Volume
Critical Path & Lock Contention	Sampled CPU Attribution

Detailed Tables

Timing Metric (median of 3 runs)	Stock USVFS	Batched	Saved	Reduction
Main-thread hook_NtQueryDirectoryFileEx inclusive time1	23.976 s	1.606 s	22.370 s	93.30%
Process creation → DataLoader thread start	41.789 s	19.214 s	22.575 s	54.02%
DataLoader thread lifetime (plugin compilation & DataLoaded)	21.126 s	14.082 s	7.044 s	33.34%
↳ DataLoader blocked in WaitForDirectoryCache (waiting on OAR)	4.411 s	0.000 s	4.411 s	100.00%
OAR directory cache worker lifetime (parallel)	21.016 s	5.739 s	15.277 s	72.69%
Process creation → Load Time Profiler main-menu report export	63.440 s	33.372 s	30.068 s	47.40%

API / Internal Operation (main thread, median of 3 runs)	Stock USVFS	Batched	Reduction
Non-overlapping native NtQueryDirectoryFile / FileEx calls2	479,112	49,898	89.59%
Outer hook_NtQueryDirectoryFileEx calls	253,780	49,861	80.35%
Total native NtCreateFile / NtOpenFile calls3	248,955	43,359	82.58%
Hooked NtClose calls4	257,269	25,425	90.12%
Hooked GetFileAttributesW calls	469,569	5,787	98.77%
Hooked NtQueryAttributesFile calls	470,731	6,950	98.52%
HookContext::readAccess lock acquisitions	1,233,697	51,500	95.83%
Virtual directory scan constructions (VirtualEntryReader)	11,624	11,624	0%
Total virtual candidates	220,268	220,268	0%

Cumulative Native Call Time / Sampled CPU (main thread)5	Stock USVFS	Batched	Reduction
Native NtQueryDirectoryFile / FileEx	3.627 s	0.404 s	88.86%
Native NtCreateFile / NtOpenFile (incl. openScanDirectory)	4.586 s	1.135 s	75.25%
Native NtClose (via hook path)	2.049 s	0.216 s	89.46%
Native GetFileAttributesW	5.892 s	0.238 s	95.96%
Native NtQueryAttributesFile	10.567 s	0.257 s	97.57%
Outermost HookContext lock hold time	13.162 s	0.391 s	97.03%
Sampled CPU time under USVFS call stacks (incl. kernel & minifilters)6	28.266 CPU s	4.007 CPU s	85.82%
usvfs_x64.dll exclusive leaf CPU time6	2.245 CPU s	0.488 CPU s	78.26%

Main Breaking Changes

• (a6a514e) Callers querying asynchronous handles, or passing an Event, ApcRoutine, or ApcContext (completion packet), will no longer see virtual entries. The legacy implementation only partially handled these and even forwarded the caller's Event and APC directly to internal reads backed by a stack-allocated IO_STATUS_BLOCK, which is undefined behavior. (Full emulation of asynchronous notifications is feasible if ever needed, though it would require additionally hooking NtSetInformationFile.) Frankly, I'm not sure whether anything in the ecosystem relies on asynchronous directory queries, but it should be an exceedingly rare pattern—only applications calling the low-level NT API directly with an asynchronous handle, Event, or APC are affected. Personally, I'd much rather make a clean breaking change than keep a broken implementation for the illusion of compatibility.

• (6656902) Chained "virtual-to-virtual" mappings are no longer listed. Because the file-open hooks only perform a single-hop reroute, those files could never be opened in the first place, and re-entering the hooks to resolve them hurt performance.

Expected Behavior Changes

Below are the intended externally visible behavior changes beyond the main breaking changes above.
There may still be unintended behavior changes I introduced inadvertently, though the existing test suite and the new tests added here should minimize both their likelihood and their impact.

Expected Behavior Changes

Native Forwarding Rules

The following queries are now forwarded straight to ntdll; forwarded queries see only the physical directory on disk:

• Non-directory-enumeration FileInformationClass values: return ntdll's status code instead of bogusly returning STATUS_NO_SUCH_FILE (fa76500).
• Undersized or misaligned buffers: rejected just as native ntdll does (67c38e1).
• SL_RETURN_ON_DISK_ENTRIES_ONLY and SL_NO_CURSOR_UPDATE (c874455):
  • FindFirstFileEx(FIND_FIRST_EX_ON_DISK_ENTRIES_ONLY) now honors the flag's contract and enumerates only the physical directory.
  • KernelBase keeps passing this flag ((flags & 4) * 2) on every subsequent FindNextFile call on that handle, so Win32 enumeration never mixes modes mid-scan.
  • Like the kernel I/O manager, SL_INDEX_SPECIFIED and undefined flag bits are ignored.

Buffer & Status Output

• Status block (IO_STATUS_BLOCK) (ccedf5f):
  • Information is set to 0 when enumeration ends.
  • On query failure, the status block is left untouched (the stock implementation overwrote it unconditionally).
• Records:
  • Each record starts at an 8-byte aligned offset and is linked via NextEntryOffset the same way ntdll does.
  • Information ends at the exact end of the last record, without trailing alignment padding (beb05c5).
  • Renamed records are sized to their full virtual name, without zeroing bytes past the name (06f2997).
• Multiple virtual entries are packed into a single call until the caller's buffer is full (8e145ae).
• Partial records:
  • Physical directory enumeration continues on subsequent calls after returning a partial record with STATUS_BUFFER_OVERFLOW (59f1d6e).
  • A virtual entry that does not fit into an empty buffer returns a partial record with STATUS_BUFFER_OVERFLOW and is repeated on subsequent calls until a buffer large enough to hold it is provided (e0bb294, 4bbf3d3).
  • A virtual entry is still listed whole when its virtual name fits the caller's buffer, even if its source file name on disk would be too long for that buffer (58ab031).
• STATUS_NO_SUCH_FILE is returned only when the very first query on a handle yields no results. An empty result after a restart ends with STATUS_NO_MORE_FILES (d34800b).

Directory Enumeration & Matching

• Errors from reading the physical directory are propagated to the caller rather than silently swallowed (b6f2cc3).
• Restarting a scan:
  • SL_RESTART_SCAN also restarts the physical portion of the scan (c7f09a8).
  • A restart without FileName keeps the pattern latched by the first query, matching NTFS and ReFS (fdd30ff).
• Case-sensitive directories list names that differ only in case as distinct entries (00977a0).
• Stale scan state left behind under a handle value is dropped when NtOpenFile/NtCreateFile reissues that handle value (50bd6a6).
• Wildcard pattern matching now uses RtlIsNameInExpression (cdd1a4a):
  • Supports DOS wildcards and non-ASCII case folding.
  • An empty pattern lists every entry.
  • foo no longer falsely matches .foo, and a trailing .* is no longer treated as *.
• The physical portion of a scan always lists the real on-disk directory at the virtual path, regardless of how the caller opened the directory (fa8e67a). Previously this depended on FILE_OPEN_FOR_BACKUP_INTENT. Caller handles lacking FILE_LIST_DIRECTORY access fail just as in native ntdll.
• Case folding matches NTFS and ReFS by folding one UTF-16 code unit at a time via RtlUpcaseUnicodeChar (dfe9dfa), no longer merging case variants outside the Basic Multilingual Plane (BMP).
• Directories in the redirection tree without a linkTarget are left to the on-disk listing and returned in the same query batch as ntdll (96a46f5).
• Purely virtual directories (directories that exist only in mods and not on disk, e.g., meshes) list only . and .. from the rerouted caller handle, followed by the virtual entries (87d91b6).
  • Previously, the entire directory of the first mod was listed, leaking entries that were excluded from the virtual tree: .mohidden files, skipped .git directories, and files added to the mod after mapping. Those files were listed even though they could not be opened.
• Internal directory opens enable delete sharing (FILE_SHARE_DELETE), matching FindFirstFileExW (19887de):
  • Directories can now be renamed or deleted while being enumerated.
  • Directories already held open by another handle with DELETE access can still be enumerated from disk.

Changes Beyond Enumeration

• Hooks & locking:
  • Handles opened internally by a scan bypass the open and close hooks (d736e6b).
  • Removed SearchHandleMap (7ad82c0). FindFirstFileExW, NtOpenFile, and NtCreateFile no longer write state to HookContext.
  • NtClose no longer acquires the HookContext lock or calls GetFileType (e4fa037).
• Logging:
  • Logs one line per directory scan instead of one per query call (92fcba8).
  • Unmapped directories are no longer logged (454adab).
  • Logs the actual NTSTATUS when opening a mapping source directory fails, rather than an unrelated GetLastError (c4f6473).
  • Logs QueryFlags on both entry points (6fdfc61).
• Build system:
  • Fixed Clang standard-conformance compilation errors (e52de37).
  • Added Google Benchmark 1.9.5 under the testing feature in vcpkg.json for a benchmark target not run by default under ctest (53695da).

Known Limitations & Trade-offs

• Status block on ReFS when no files match:
  • When no files match, ReFS writes {STATUS_NO_SUCH_FILE, 0} to the status block, whereas NTFS leaves it untouched; the hook follows NTFS.
  • As a result, the status block on ReFS differs from native ntdll. The stock implementation wrote the status block unconditionally, matching ReFS but diverging from NTFS.
  • Differential tests compare only the returned NTSTATUS on failed queries.
• Physical directory renamed or deleted before the first query:
  • Whether the scan falls back to "dot entries (. and ..) only" depends on why opening the physical directory at the virtual path failed.
  • If the caller's handle itself points to the physical directory (NtCreateFile does not reroute when the path is a directory both on disk and in the virtual tree) and that directory is renamed or marked for deletion before the first query, the hook lists only ., .., and virtual entries, whereas ntdll would still list the entire physical directory.
  • Enabling FILE_SHARE_DELETE in 19887de makes such renames easier to trigger. This case is not currently covered by tests.
• Mixing forwarded and virtualized queries on the same handle:
  • If a low-level NT caller alternates between the two on the same handle (e.g., passing an Event on one call and omitting it on the next, or toggling flags mid-scan), the first virtualized call will start a fresh scan, which may return physical directory entries a second time.
  • This cannot happen through FindFirstFileEx/FindNextFile.
• When a scan cannot open an existing physical directory at the virtual path for other reasons (such as an ACL denying directory listing), it falls back to reading the caller's handle in full. If that handle was opened via hook_NtOpenFile, it points to the first mapped mod directory—matching what the stock implementation listed.
• Shadowing remains case-insensitive even in case-sensitive directories: a virtual a.txt hides both a.txt and A.txt from the physical directory, consistent with how the redirection tree resolves names.
• Virtual entries carry no 8.3 short name, as on a volume with 8.3 names disabled (c5f7ef2). Previously they carried the short name of their source file, which is unique only in the source directory, so GetShortPathNameW could build paths that did not open or that led to another file.
• Unhandled exceptions inside the hook still terminate enumeration with STATUS_NO_MORE_FILES, as in the stock implementation.
• For mapped directories containing a large number of physical files, the physical portion takes roughly 1.16–1.52× native time due to filtering and copying overhead. Reading ahead into a larger buffer would improve throughput, but introduces cursor and information-class hazards that aren't worth the trade-off right now.
• Verified only on NTFS and ReFS. (Though honestly, does anyone actually play Skyrim on ReFS or another filesystem?)

Verification

cmake --preset vs2022-windows-x64
cmake --build --preset vs2022-windows-x64 --config Release
cmake --preset vs2022-windows-x86
cmake --build --preset vs2022-windows-x86 --config Release

ctest --test-dir vsbuild64 -C Release --output-on-failure

Both unit and differential tests for this PR live in tvfs_test, so you can also run tvfs_test directly.
The differential tests run against C:\Windows and generated directories on the volume hosting the test binary. To cover ReFS, clone and run the repository on a ReFS volume.

Benchmarks (Optional)

cmake --build --preset vs2022-windows-x64 --config Release --target tvfs_directory_bench
.\test\bin\tvfs_directory_bench_x64.exe

Compares NativeListing / VirtualListing / UnmappedListing across mods ∈ {1, 8} and files ∈ {64, 512}. Use --benchmark_filter= to filter specific cases.

Notes for Reviewers

• Suggested reading order:

  • Although the overall diff is large, the vast majority of added lines are unit and differential tests in test/tvfs_test/directory_enumeration.cpp. The core production code lives in directory_records.{h,cpp} (record layout parsing and DirectoryOutput buffer writing) and directory_scan.{h,cpp} (RegularEntryReader, VirtualEntryReader, and DirectoryScan state management)—you may find it easiest to read the final state of those two pairs of files directly.

• Reviewing commit-by-commit: every commit builds cleanly and passes the test suite on its own, structured around "First make the change easy, then make the easy change":

  1. e52de37 – 6656902: Split directory enumeration out of ntdll.cpp and establish the DirectoryScan / DirectoryOutput skeleton, while fixing UB, resource leaks, dropped partial records, and hook re-entrancy when opening source directories that stood in the way of batching.
  2. 53695da – bc60b58 (plus 097ebff, a41b9a0, etc.): Implement the core batched reading optimizations.
  3. cdd1a4a – 3c2fa5d: Align remaining ntdll semantics orthogonal to batching (pattern matching and latching, SL_* flags, physical directory source, dot entries for purely virtual directories, FILE_SHARE_DELETE, etc.), along with follow-up performance work around the HookContext lock, SearchHandleMap, NtClose, and the name hash set.

• Commits labeled refactor that also change externally visible details:

  • beb05c5: Drops trailing alignment padding after the last record.
  • 06f2997: Sizes renamed records to the virtual name and stops zeroing bytes past it.
  • 92fcba8: Logs once per directory scan instead of once per query call.
  • 7ad82c0: Removes SearchHandleMap along with the associated lock acquisition and logging in NtClose.
  • All other refactor commits are purely structural and should not change external behavior.

Footnotes

1. Inclusive wall-clock time on the main thread inside usvfs::hook_NtQueryDirectoryFileEx prior to DataLoader thread creation. Because inner hook/native calls and HookContext lock hold times nest and overlap, rows cannot be summed directly. ↩

2. Counts non-overlapping low-level Original_NtQueryDirectoryFile / FileEx call sites (including forwardToNative), avoiding double-counting across outer addNtSearchData spans. ↩

3. Combines native ntdll_mess_NtCreateFile / NtOpenFile calls with the 26,294 internal openScanDirectory NtOpenFile calls added by the batched reader (which bypass outer hooks via InternalIoScope). ↩

4. Reflects only calls going through usvfs::hook_NtClose; scan directory handles opened internally on the batched path are closed directly under InternalIoScope. ↩

5. Native call times and lock hold durations nest and overlap, so rows cannot be summed directly. ↩

6. Exact ETW SampledProfile CPU sample weights from two lossless paired runs (n=2 per group). "USVFS call stacks" includes kernel (ntoskrnl.exe, Ntfs.sys, FltMgr.sys) and filter driver work (e.g., AV minifilter sysdiag.sys) triggered by USVFS hooks; "exclusive leaf CPU" counts only instructions executed directly inside usvfs_x64.dll. ↩ ↩²