Skip to content

Answer lookups below directories missing on disk without the file system - #98

Open
1001Bits wants to merge 1 commit into
ModOrganizer2:masterfrom
1001Bits:perf/missing-directory-record
Open

1001Bits wants to merge 1 commit into
ModOrganizer2:masterfrom
1001Bits:perf/missing-directory-record

Conversation

@1001Bits

@1001Bits 1001Bits commented Oct 12, 2026 •

Copy link
Copy Markdown

When I looked into speeding up the startup of the game, I came across this, might be a good optimisation for MO2.

Games probe a lot of paths that exist nowhere. Fallout 4, for example, looks up Data\DATA\Sound\Voice\... twice for every loose sound file while it registers them at startup; in a profile with 339k loose files that was 251,250 failed lookups. Under usvfs each miss runs the full hook path (several path canonicalizations, redirection tree walks, GetFileAttributesW plus the real NtCreateFile), which costs several times what the miss costs without usvfs.

This PR keeps a small per-process record of real directories inside mapped folders that were found missing on disk. Below such a directory nothing exists physically, so a path there can only exist through the redirection tree. NtCreateFile (FILE_OPEN, absolute drive path) and GetFileAttributes(Ex)W then answer from the tree alone, without touching the file system, and return the same error the existing code returns.

Measurements

Synthetic benchmark: 5 mods with 20,000 loose .fuz files linked onto game\Data (plus an overwrite with LINKFLAG_CREATETARGET, like MO2), child process started with usvfsCreateProcessHooked. Stock = this repository's master (57f1ea5) built with the same configuration; median of 5 interleaved rounds, CPU time per call in µs (the machine was shared, so wall time is noisier):

lookup no usvfs stock this PR
CreateFile of Data\DATA\Sound\Voice\...\x.fuz (folder missing everywhere) 16.8 78.9 6.2
CreateFile of a missing file in a folder that only exists in a mod 15.2 175.8 8.6
GetFileAttributes of the Data\DATA\... path 28.9 52.0 7.0
CreateFile of existing virtual files – 180.5 175.0
CreateFile of files outside the virtual folders – 182.0 199.2

The last two rows are within run-to-run noise (±8% here); measured in isolation, the record check costs about 1 µs per lookup while the record is non-empty, and nothing while it is empty. Every arm found the same files (identical hit counts).

Why answers cannot change

  • The redirection tree is always consulted: a path with a tree entry (a mod file, or anything created through usvfs by this or another hooked process) takes the normal path.
  • Only directories directly below a mapped tree node (one with a link target) are recorded, so paths outside the virtual folders (saves, documents, temp, ...) never use the record.
  • Directory creation (CreateDirectoryW, NtCreateFile with FILE_DIRECTORY_FILE that reports FILE_CREATED) and moves (MoveFile*W) through the hooks clear the record; a generation counter keeps a probe that raced with such a change from re-adding a stale entry.
  • Entries expire after 2 seconds, so a directory that appears some other way (an unhooked process writing into the game folder, a rename through a handle) is visible again shortly after.
  • Error codes match the existing behaviour: NtCreateFile returns STATUS_OBJECT_PATH_NOT_FOUND/STATUS_OBJECT_NAME_NOT_FOUND exactly as the real call on the unredirected path would, and GetFileAttributes(Ex)W keeps the existing "file not found when the parent exists virtually" fix-up.

Tests

New UsvfsGlobalTest.MissingDirectoryTest: repeated lookups (first from the file system, later from the record) return the same errors for a folder missing everywhere, for the missing folder itself and for a missing file in a mod-only folder; mod files below a recorded directory are still found; files and directories created afterwards are visible immediately. The test also passes against the unpatched DLL.

Run locally (x64, Release): shared_test, thooklib_test, tinjectlib_test, tvfs_test, usvfs_global_test_runner (all pass) and usvfs_test_runner (basic_x64 passes; the x86 scenarios were not run because I only built x64). I have not measured a full game startup with this build.

Related to #95, which batches directory enumeration; this PR covers lookups of paths that do not exist, so the two are independent.

@1001Bits
1001Bits force-pushed the perf/missing-directory-record branch from 0134516 to caab0e8 Compare October 12, 2026 14:15
Games probe many paths below directories that exist nowhere: Fallout 4
looks up "Data\DATA\Sound\Voice\..." twice for every loose sound file,
and engines check for loose files that only exist in archives. Each such
miss costs usvfs several path canonicalizations, redirection tree walks
and file system calls - 80-180 us of CPU per lookup in a synthetic
benchmark, against 15-30 us without usvfs.

Keep a per-process record of real directories inside mapped folders that
were found missing on disk. Below such a directory nothing exists
physically, so a path there exists only if the redirection tree has an
entry for it. NtCreateFile (FILE_OPEN by an absolute drive path) and
GetFileAttributes(Ex)W answer those lookups from the tree alone and
return the same error the slow path returns.

- the tree is always consulted, so files and directories added through
  usvfs (by this or any other hooked process) are found immediately
- directory creation or moves through the hooks clear the record
- entries expire after 2 seconds, so a directory created some other way
  (an unhooked process, a rename through a handle) shows up shortly
- only directories below a mapped tree node (one with a link target)
  are recorded

MissingDirectoryTest checks that repeated lookups return the same errors
as the first one, that virtual files below a recorded directory are
still found and that new files and directories are visible at once; it
passes on the unpatched DLL as well.
@1001Bits
1001Bits force-pushed the perf/missing-directory-record branch from caab0e8 to daf8d8e Compare October 12, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant