Repository navigation
Conversation
1001Bits
force-pushed
the
perf/missing-directory-record
branch
from
October 12, 2026 14:15
0134516 to
caab0e8
Compare
Games probe many paths below directories that exist nowhere: Fallout 4 looks up "Data\DATA\Sound\Voice\..." twice for every loose sound file, and engines check for loose files that only exist in archives. Each such miss costs usvfs several path canonicalizations, redirection tree walks and file system calls - 80-180 us of CPU per lookup in a synthetic benchmark, against 15-30 us without usvfs. Keep a per-process record of real directories inside mapped folders that were found missing on disk. Below such a directory nothing exists physically, so a path there exists only if the redirection tree has an entry for it. NtCreateFile (FILE_OPEN by an absolute drive path) and GetFileAttributes(Ex)W answer those lookups from the tree alone and return the same error the slow path returns. - the tree is always consulted, so files and directories added through usvfs (by this or any other hooked process) are found immediately - directory creation or moves through the hooks clear the record - entries expire after 2 seconds, so a directory created some other way (an unhooked process, a rename through a handle) shows up shortly - only directories below a mapped tree node (one with a link target) are recorded MissingDirectoryTest checks that repeated lookups return the same errors as the first one, that virtual files below a recorded directory are still found and that new files and directories are visible at once; it passes on the unpatched DLL as well.
1001Bits
force-pushed
the
perf/missing-directory-record
branch
from
October 12, 2026 14:34
caab0e8 to
daf8d8e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When I looked into speeding up the startup of the game, I came across this, might be a good optimisation for MO2.
Games probe a lot of paths that exist nowhere. Fallout 4, for example, looks up
Data\DATA\Sound\Voice\...twice for every loose sound file while it registers them at startup; in a profile with 339k loose files that was 251,250 failed lookups. Under usvfs each miss runs the full hook path (several path canonicalizations, redirection tree walks,GetFileAttributesWplus the realNtCreateFile), which costs several times what the miss costs without usvfs.This PR keeps a small per-process record of real directories inside mapped folders that were found missing on disk. Below such a directory nothing exists physically, so a path there can only exist through the redirection tree.
NtCreateFile(FILE_OPEN, absolute drive path) andGetFileAttributes(Ex)Wthen answer from the tree alone, without touching the file system, and return the same error the existing code returns.Measurements
Synthetic benchmark: 5 mods with 20,000 loose
.fuzfiles linked ontogame\Data(plus an overwrite withLINKFLAG_CREATETARGET, like MO2), child process started withusvfsCreateProcessHooked. Stock = this repository'smaster(57f1ea5) built with the same configuration; median of 5 interleaved rounds, CPU time per call in µs (the machine was shared, so wall time is noisier):CreateFileofData\DATA\Sound\Voice\...\x.fuz(folder missing everywhere)CreateFileof a missing file in a folder that only exists in a modGetFileAttributesof theData\DATA\...pathCreateFileof existing virtual filesCreateFileof files outside the virtual foldersThe last two rows are within run-to-run noise (±8% here); measured in isolation, the record check costs about 1 µs per lookup while the record is non-empty, and nothing while it is empty. Every arm found the same files (identical hit counts).
Why answers cannot change
CreateDirectoryW,NtCreateFilewithFILE_DIRECTORY_FILEthat reportsFILE_CREATED) and moves (MoveFile*W) through the hooks clear the record; a generation counter keeps a probe that raced with such a change from re-adding a stale entry.NtCreateFilereturnsSTATUS_OBJECT_PATH_NOT_FOUND/STATUS_OBJECT_NAME_NOT_FOUNDexactly as the real call on the unredirected path would, andGetFileAttributes(Ex)Wkeeps the existing "file not found when the parent exists virtually" fix-up.Tests
New
UsvfsGlobalTest.MissingDirectoryTest: repeated lookups (first from the file system, later from the record) return the same errors for a folder missing everywhere, for the missing folder itself and for a missing file in a mod-only folder; mod files below a recorded directory are still found; files and directories created afterwards are visible immediately. The test also passes against the unpatched DLL.Run locally (x64, Release):
shared_test,thooklib_test,tinjectlib_test,tvfs_test,usvfs_global_test_runner(all pass) andusvfs_test_runner(basic_x64passes; the x86 scenarios were not run because I only built x64). I have not measured a full game startup with this build.Related to #95, which batches directory enumeration; this PR covers lookups of paths that do not exist, so the two are independent.