Skip to content

fix(rvc,rvc-bin): give each slashing database its own fallback marker (W4-D2a) - #1013

Draft
rootwarp wants to merge 13 commits into
developfrom
fix/w4-d2a-888
Draft

rootwarp wants to merge 13 commits into
developfrom
fix/w4-d2a-888

Conversation

@rootwarp

@rootwarp rootwarp commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Refs #746 (item 1b; #746 stays open and closes with W4-D3e)
Closes #888

Summary

If the keystore directory is not the parent of the slashing database, the shutdown drain can write a fallback detection marker beside the database. Earlier releases used one shared name, .rvc.doppelganger_detected, in that directory. A marker from one instance could stop the other instance at boot. A clear for one instance could remove the only detection record of the other instance.

The drain now writes .rvc.doppelganger_detected.<db file name>. The name keeps the native bytes of the database file name. The file records the keystore path, which the writer makes absolute at write time. The writer does not resolve a symlink in that path. Boot checks this name and the legacy name.

The fallback directory is the parent of the database, or . for a bare file name such as slashing.sqlite. Thus slashing.sqlite and ./slashing.sqlite select one directory and one marker. One walk opens the fallback directory for the selection, the read, the write and the clear. The walk follows no symlink, and nothing resolves the path before the walk. The selection opens the database directory and the keystore directory with that walk and compares (st_dev, st_ino) of the two handles. It selects no fallback only when the two handles are one directory. When a walk or an fstat fails, the selection gives the database directory, so boot reads the fallback and fails closed. No step of the fallback path calls canonicalize or compares paths. A missing directory is not absent: boot refuses it, the drain write fails, and the clear stops.

Unless the walk shows that the database directory is the keystore directory, the name of the database file must be the only name of that file. Boot refuses a second name of a database before SQLite opens it (exit code 1). A second name is a symlink to the database file, a symlink or a .. in the directory path, or a database file with more than one link. Boot also refuses a missing database directory. Because of this, boot, the drain and the clear use one name of each database.

The fallback writes and clears in one database directory hold one lock and use one directory handle. A write does not replace an entry that is not a marker of the same keystore. clear-detection --slashing-db-path removes the owned file only when its recorded owner is --datadir. It keeps an owned file of another keystore and prints its path and owner. It keeps an owned file that records no owner and a legacy file, and it stops with the path and the recovery step. Boot refuses a database file name in the reserved name space .rvc.doppelganger* and a name whose marker name has more than 255 bytes.

Changes

  • crates/rvc/src/detection_marker.rs
    • FallbackMarkerPath { dir, slashing_db_path }. for_config calls fallback_marker_dir. owned_path returns the path of the owned marker.
    • fallback_marker_dir(keystore_dir, slashing_db_path) uses the parent of the database, or . for an empty parent (a bare file name). It returns None only when same_dir shows one directory.
    • same_dir(left, right) (private) opens the two directories with MarkerDir::open and compares (st_dev, st_ino) from File::metadata of the two handles (fstat on the descriptor). An error of a walk or of a metadata read gives false. On a platform other than Unix it is always false. It does not call canonicalize and does not compare paths.
    • fallback_marker_name(slashing_db_path) -> OsString: .rvc.doppelganger_detected. and the native bytes of the database file name.
    • check_slashing_db_name(keystore_dir, slashing_db_path) returns InvalidInput for a file name that starts with .rvc.doppelganger or ..rvc.doppelganger. When the database directory gets a fallback marker, it also returns InvalidInput for these paths:
      • a marker name of more than 255 bytes;
      • a database directory that the walk cannot open (a missing directory, or a symlink, a file or a .. in the path);
      • a database file that is a symlink, is not a regular file, or has more than one link.
    • MarkerDocument::keystore_dir: Option<String> with #[serde(default, skip_serializing_if = "Option::is_none")]. deny_unknown_fields stays. The keystore writer omits keystore_dir.
    • resolved_owner(keystore_dir) (private) returns std::path::absolute(keystore_dir) with no . component and no trailing slash. It does not call canonicalize, so it keeps a symlink and a ... It reads no file system entry. It returns None for an empty path or a path that is not valid UTF-8.
    • MarkerDir::open (private, Unix) opens the database directory, and for the selection the keystore directory, with a walk. The walk starts at / for an absolute path and at the working directory for a relative path. It opens each component with O_NOFOLLOW | O_DIRECTORY (O_PATH on Linux). It does not call canonicalize. A missing name, a symlink, a file or a .. stops the walk with an error. read, lock, replace, rename and unlink use the handle (openat, flock, renameat, unlinkat, and fsync of the handle).
    • MarkerDir::check_db_entry (private, Unix) opens the database file entry with O_PATH | O_NOFOLLOW through the handle. It refuses a symlink, an entry that is not a regular file, and a file with more than one link. A missing file (a new database) passes.
    • MarkerDir::replace is the one marker writer. It writes a temp file .rvc.doppelganger_tmp.<pid>.<nanos>.tmp (at most 57 bytes) with O_EXCL | O_NOFOLLOW and mode 0600. Then it does an fsync, renameat, and an fsync of the directory. A failed step removes only a temp file that this call created and did not rename.
    • atomic_write (keystore marker, force-window file) opens the keystore directory by its path, as boot does for its reads of that directory. Then it calls MarkerDir::replace.
    • write_fallback_marker(&FallbackMarkerPath, keystore_dir, &DetectionMarker) takes the fallback lock and reads the owned path. It writes only when the path is absent or holds a marker with the same owner. Else it returns AlreadyExists with the path.
    • inspect_fallback(&FallbackMarkerPath) -> FallbackInspect { owned, legacy } reads the two names through one directory handle. A directory that the walk cannot open, a missing one too, gives two Unreadable results.
    • clear_fallback_marker(&FallbackMarkerPath, keystore_dir) -> io::Result<Vec<FallbackClear>> checks the database file as check_slashing_db_name does, and takes the fallback lock. It returns Removed or OtherOwner. It returns InvalidData with the path and the recovery step for these entries:
      • an owned marker that cannot be read, does not parse, or records no owner;
      • a legacy marker.
        The checks occur before a removal.
    • FALLBACK_LOCK_FILENAME = ".rvc.doppelganger_lock".
    • Test-only (#[cfg(test)]): arm_walk_start(dir) makes each walk of a relative path on the test thread start at dir in place of the working directory.
  • crates/doppelganger/src/marker_file.rs: read_open_marker(file) holds the fstat check and the bounded read. read_marker_file and MarkerDir::read use it.
  • crates/rvc/src/bootstrap/run.rs
    • run_with_clock calls check_slashing_db_name before refuse_if_detection_records and before SQLite opens the database. A refusal is exit code 1 (InvalidConfig).
    • refuse_if_detection_records refuses (exit code 12) when inspect_fallback(..).owned or .legacy is not Absent. The log names the path. bootstrap/slashing.rs calls the same function after the keystore lock.
    • DetectionShutdown.fallback: Option<FallbackMarkerPath>. The two fallback writes in persist_latched_detection call write_fallback_marker.
  • bin/rvc/src/commands/doppelganger.rs: execute_clear_detection calls clear_fallback_marker when --slashing-db-path is given and FallbackMarkerPath::for_config gives a fallback. It prints one line for each result. An error stops the command before the keystore marker step. Fix round 3 changes only its doc comment.
  • crates/rvc/src/liveness_loop.rs (tests only): the drain tests use FallbackMarkerPath and inspect the owned marker. The two-instance test and the drain test for three failed writes are here.

State ownership

State Owner Writers Readers Clearers Lock / order Invariant after this PR
Fallback directory (selection; fix round 3) none: fallback_marker_dir computes it each time none boot check_slashing_db_name and refuse_if_detection_records; the drain (built at boot); CLI execute_clear_detection none the walk opens the database directory (. for a bare file name) and the keystore directory; then fstat of the two handles No fallback only when the two handles have one (st_dev, st_ino). A walk error or a metadata error gives the database directory, so the fallback steps run and fail closed. No path comparison and no canonicalize.
Owned fallback marker <db dir>/.rvc.doppelganger_detected.<db file name> the database file name (native bytes), which must be the only name of the database file; keystore_dir in the document records the keystore path that the writer made absolute, with no symlink resolution drain persist_latched_detection -> write_fallback_marker (both fallback branches) boot refuse_if_detection_records (before signer setup, and again after the keystore lock) CLI clear_fallback_marker the selection; then the walk with no symlink opens the database directory; then the fallback lock; then one directory handle for the read, the replace check, the write and the unlink A write replaces only a marker with the same owner. Only the clear for the recorded owner removes it. A marker that records no owner stays (spec change that Ash approved). Boot of another database does not read it. A second name of the database is refused, so it cannot select a different marker.
Legacy fallback marker <db dir>/.rvc.doppelganger_detected none none after this PR (an older release can write it) boot refuse_if_detection_records none: the operator removes it by hand after the recovery step (spec change that Ash approved) as above for the read Boot refuses while it exists. The CLI keeps it and stops with its path and the recovery step.
Fallback lock <db dir>/.rvc.doppelganger_lock (new) each process that writes or clears a fallback marker in that directory MarkerDir::lock creates it (0600) none none (the file stays) `flock(LOCK_EX LOCK_NB)` through the directory handle; the drain tries again in 1 s, the CLI stops
Reserved name space .rvc.doppelganger*, ..rvc.doppelganger* (new) rvc detection files rvc only (markers, record, temp files, probe, lock) boot check_slashing_db_name none before refuse_if_detection_records and before SQLite opens the database No slashing database has a name in it (exit code 1). A marker name has at most 255 bytes.
Database name (new rule) the configured --slashing-db-path, unless the walk shows that its directory is the keystore directory none boot check_slashing_db_name; CLI clear_fallback_marker none before SQLite opens the database; in the clear, before the lock The directory path has no symlink and no .., and the directory exists. The database file is absent or a regular file with one link. Else boot returns exit code 1, and the clear stops.
Keystore marker <keystore>/.rvc.doppelganger_detected keystore directory liveness loop, drain (write_marker, no keystore_dir) boot refusal, restart-skip CLI clear_marker (last step) .rvc.lock The writer is now MarkerDir::replace, on a handle that atomic_write opens by path; the file format does not change
Detection record <keystore>/.rvc.doppelganger_record keystore directory boot preallocate; loop and drain commit; CLI reset or replace boot refusal, preallocate, CLI CLI reset_or_replace_detection_record .rvc.lock, after the force-window file Unchanged

Boot checks four locations for evidence of detection. These are the detection record, the keystore marker, the owned fallback, and the legacy fallback. Boot refuses while one of them holds evidence or cannot be read. clear-detection removes an owned fallback only when it records --datadir as its owner. It never removes a fallback that records another owner or no owner.

On-disk format

  • The owned fallback marker has the name .rvc.doppelganger_detected.<db file name> in the directory of the slashing database, unless the walk shows that this directory is the keystore directory. The name keeps the native bytes of the database file name. Example: --slashing-db-path /data/db/a.sqlite gives /data/db/.rvc.doppelganger_detected.a.sqlite. For a bare file name, the directory is the working directory: slashing.sqlite and ./slashing.sqlite give ./.rvc.doppelganger_detected.slashing.sqlite.
  • The <db file name> is the only name of the database file. The directory path has no symlink and no .., and the database file is a regular file with one link. Boot refuses another path (exit code 1). Because of this, each database has one marker name. Fix rounds 2 and 3 do not change the marker name or the document.
  • The marker document can have the optional field keystore_dir (string). Only the owned fallback has it. Its value is the keystore path that the writer made absolute at write time, with no . component and no trailing slash. The writer does not resolve a symlink or a .. (fix round 3). For a keystore path with no symlink and no .., the value is the same as in fix round 2. A keystore path that is not valid UTF-8 is not recorded.
  • The fallback lock .rvc.doppelganger_lock is a new file. Each marker write also makes the temp file .rvc.doppelganger_tmp.<pid>.<nanos>.tmp. The temp file exists only during a write.
  • A slashing database file name must not start with .rvc.doppelganger or ..rvc.doppelganger.
  • The document keeps format version 1. The reader accepts a version-1 document without keystore_dir. An unknown field causes the reader to return Corrupt. Boot then returns exit code 12.

Tests

Test cases

The red commits keep the previous behavior in temporary implementations. The next three entries give the test results with the previous behavior.

  • Round 1, commit ae9b8ba3 (60ea92b6 before the rebase onto 49d4cacd): run the 13 named tests on that commit. Five tests fail on assertions. Eight tests pass.
  • Fix round 1, commit b3c05934 (4b032d4c before the rebase): run the 17 named tests on that commit. Fourteen tests fail on assertions. Three tests pass. Commit 4f6f7cfa (4f4bccc0) later merges three pairs of these tests. Each assertion stays. Commit 20da6d14 (0145a66a) adds two tests that kill mutants; they pass on the fix.
  • Fix round 2, commit 1d86c608 (d71cd908 before the rebase), changes only tests. The production code keeps the fix1 behavior. Run the five named tests on that commit. The five tests fail on assertions. Commit 39cc4149 (23d23701) removes the test fallback_write_refuses_a_directory_that_changes_after_the_resolve, because no resolve step occurs before the walk now. The row "a symlink at the last component" of fallback_paths_with_a_symlink_or_a_parent_name_are_refused checks a symlink at the directory.

Fix round 3 has a separate record. Commit 7c45722e changes only tests and adds the test-only seam arm_walk_start. The production code keeps the fix2 selector. Run the six named tests on that commit. Five tests fail on assertions. The sixth test, clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore, has the new Harper N1 messages, and it passes. Commit 804257da has the fix.

Test Type With the previous behavior Failing assertion text
liveness_loop::tests::fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir round 1 Assertion failure The boot of instance B must not exit 12 for the detection of instance A.
bootstrap::run::tests::boot_refuses_an_owned_fallback_marker round 1 Assertion failure The boot must exit 12 for the owned fallback marker.
commands::doppelganger::tests::clear_detection_keeps_an_owned_fallback_marker_of_another_keystore round 1 Assertion failure The command for keystore B must keep the owned fallback marker of keystore A.
detection_marker::tests::clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore round 1; changed in fix1 (B5); changed in fix3 (Harper N1 messages, the absolute owner path) Assertion failure (round 1 and fix1). Fix3: Pass Fix1: The function must return InvalidData with the legacy path and the recovery step.
detection_marker::tests::fallback_marker_name_appends_the_db_file_name round 1 Assertion failure The owned path must be the legacy name, a dot, and the database file name.
detection_marker::tests::fallback_names_keep_the_native_bytes_of_the_db_file_name fix1, group 1 Assertion failure Databases A and B must have different owned paths.
detection_marker::tests::fallback_owner_of_a_relative_keystore_is_resolved_at_write_time fix1, group 1 Assertion failure A clear from a different working directory must return the absolute keystore path that the writer resolved.
detection_marker::tests::fallback_owner_that_is_not_utf8_keeps_the_marker fix1, group 1 Assertion failure A clear for keys-� must return InvalidData with the marker path, because the marker records no owner.
detection_marker::tests::fallback_write_replaces_only_a_marker_of_the_same_keystore fix1, groups 2 and 4 Assertion failure The write must return AlreadyExists with the owned path for a slashing database file.
detection_marker::tests::fallback_write_keeps_an_entry_that_it_did_not_create fix1, group 2 Assertion failure The entry at the temp path must keep its bytes.
detection_marker::tests::fallback_write_and_clear_hold_the_lock_and_one_directory_handle fix1, groups 3 and 4 Assertion failure (two earlier tests) The lock try in the hook must fail.
detection_marker::tests::fallback_name_limits_hold_for_the_check_and_the_write fix1, group 6; changed in fix2 (a row for a file name with a NUL byte) Assertion failure (two earlier tests, and fix2) The write must succeed for a database file name of 228 bytes. The check must refuse a name of 229 bytes. Fix2: The check must refuse a name with a NUL byte and give the reason "nul byte".
bootstrap::run::tests::boot_refuses_a_db_name_that_cannot_get_a_fallback_marker fix1, groups 2 and 6 Assertion failure The error must give the reason "name space" for a name in the reserved name space.
commands::doppelganger::tests::clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner fix1, groups 5 and 7 Assertion failure (two earlier tests) The command must fail with an error that names the legacy marker and gives the recovery step.
detection_marker::tests::clear_fallback_marker_keeps_an_unreadable_owned_entry fix1, group 7 Pass Kills RSA M11.
bootstrap::run::tests::boot_refuses_an_unreadable_owned_fallback_marker fix1, group 7 Pass Kills RSA M05.
liveness_loop::tests::drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds fix1, group 7 Pass Kills RSA M07.
detection_marker::tests::fallback_reads_fail_closed_for_each_directory_state fix1, group 3; changed in fix2 (RSA B3: a missing directory) Fix2: Assertion failure The read must give Unreadable and the clear must fail for a missing directory. The results are Absent and Ok([]).
detection_marker::tests::marker_write_reports_a_failed_rename_and_removes_its_temp_file fix1, group 2 (added after the mutant list) Pass Kills cvt -> Ok(()).
detection_marker::tests::fallback_paths_with_a_symlink_or_a_parent_name_are_refused fix2, RSA B3 Assertion failure The read must give Unreadable for the two markers for a symlink in an intermediate component.
detection_marker::tests::second_names_of_a_database_are_refused_by_the_check_and_the_clear fix2, RSA B1 Assertion failure The check must refuse <root>/other/link.sqlite and give the reason "one link".
bootstrap::run::tests::boot_refuses_a_second_name_of_a_database_with_a_fallback_marker fix2, RSA B1 and B3; changed in fix3 (a row for a symlink to the keystore directory) Assertion failure (fix2 and fix3) Boot must return exit code 1 and an error with the text "one link" for a symlink to the database file. Fix3: Boot must return exit code 1 and an error with the text "no symlink" for a symlink to the keystore directory.
detection_marker::tests::fallback_dir_is_the_slashing_parent_only_when_it_differs develop (FU-A2a); changed in fix3 (RSA B1) Fix3: Assertion failure The names slashing.sqlite and ./slashing.sqlite must select the working directory and the same marker.
bootstrap::run::tests::boot_refuses_an_owned_fallback_marker_of_a_bare_db_name fix3, RSA B1 Assertion failure Boot must return exit code 12 for the database name slashing.sqlite.
detection_marker::tests::fallback_selection_does_not_skip_a_directory_behind_a_symlink fix3, RSA B3 Assertion failure The selection must give the database directory for a database directory that is a symlink to the keystore. The result is None.
detection_marker::tests::fallback_owner_keeps_a_symlink_in_the_keystore_path fix3, RSA B3 (no canonicalize in the write and the clear) Assertion failure A clear for keys must keep the marker and return the path of the symlink as the owner.
sigterm_during_marker_retry_exits_12_and_boot_is_refused, token_cancel_during_marker_retry_exits_12, panic_while_latched_exits_12_with_a_fallback_marker, in_place_record_survives_marker_and_fallback_write_failure round 1 The four tests cannot compile on the round-1 base 920a7afa. They pass with the temporary implementations in ae9b8ba3 (60ea92b6). The listed hand mutants cause their assertions to fail.
marker_without_keystore_dir_still_parses, marker_with_an_unknown_field_is_corrupt, open_slashing_db_rechecks_a_fallback_marker_beside_the_slashing_db round 1 Pass

The round-1 CLI test clear_removes_a_fallback_marker_beside_the_slashing_db pinned the removal of the legacy marker. Fix1 replaces it with the row "a legacy marker" of clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner. That row pins the new behavior: the command keeps the legacy marker and the keystore marker, and the error names the legacy path and gives the recovery step (RSA B5, a spec change that Ash approved).

Group 8: the two-instance test and in_place_record_survives_marker_and_fallback_write_failure run with paused Tokio time. The join handle of the drain task is the completion signal. join_in_paused_time gives a limit of 30 s of paused time. No wall-clock thread is in these two tests.

The working directory in the fix3 tests: fallback_dir_is_the_slashing_parent_only_when_it_differs and boot_refuses_an_owned_fallback_marker_of_a_bare_db_name use the test-only seam arm_walk_start(dir). On the test thread, each walk of a relative path starts at dir in place of the working directory. The selection, the check, the read, the write and the clear use that walk, so the result is the same as with the working directory dir. The tests do not call set_current_dir, so they do not change the working directory of the process for other tests. The boot test uses a current_thread runtime, so boot does its checks on the test thread.

Edge-case coverage:

Case Test
Two instances share a database directory; A has a detection fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir
Owned marker of this database at boot boot_refuses_an_owned_fallback_marker
Owned marker that cannot be read, at boot boot_refuses_an_unreadable_owned_fallback_marker
The owned marker names another keystore clear_detection_keeps_an_owned_fallback_marker_of_another_keystore, clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore
The selected keystore owns the fallback marker the same two tests
Owned marker with no owner, that does not parse, or that cannot be read clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore, clear_fallback_marker_keeps_an_unreadable_owned_entry, clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner
Legacy fallback present open_slashing_db_rechecks_a_fallback_marker_beside_the_slashing_db (boot), clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner (CLI)
Two database file names with one lossy string fallback_names_keep_the_native_bytes_of_the_db_file_name
Relative keystore path; keystore path that is not valid UTF-8 fallback_owner_of_a_relative_keystore_is_resolved_at_write_time, fallback_owner_that_is_not_utf8_keeps_the_marker
An entry at the owned path that is not a marker of this keystore fallback_write_replaces_only_a_marker_of_the_same_keystore
An entry at the temp path fallback_write_keeps_an_entry_that_it_did_not_create
Lock and directory handle; directory moved after the open fallback_write_and_clear_hold_the_lock_and_one_directory_handle
Missing database directory; directory path under a file; symlink at the owned path fallback_reads_fail_closed_for_each_directory_state
Symlink in an intermediate component; symlink at the last component; .. component fallback_paths_with_a_symlink_or_a_parent_name_are_refused
Symlink to the database file; hard link of the database file (both names) second_names_of_a_database_are_refused_by_the_check_and_the_clear, boot_refuses_a_second_name_of_a_database_with_a_fallback_marker
Boot through a second name of a database with a marker; boot with a missing database directory; a . component selects the same marker (exit code 12) boot_refuses_a_second_name_of_a_database_with_a_fallback_marker
A bare file name and ./<file name> select . and one marker; a bare name in the keystore directory and a relative path of the keystore directory select no fallback fallback_dir_is_the_slashing_parent_only_when_it_differs
Boot with a bare database name and an owned marker in the working directory (exit code 12, as for ./slashing.sqlite) boot_refuses_an_owned_fallback_marker_of_a_bare_db_name
A database directory that is a symlink to the keystore directory; a keystore path that is a symlink to the database directory fallback_selection_does_not_skip_a_directory_behind_a_symlink, boot_refuses_a_second_name_of_a_database_with_a_fallback_marker
A keystore path with a symlink as the recorded owner; a clear with a trailing slash fallback_owner_keeps_a_symlink_in_the_keystore_path
The rename of a marker write fails marker_write_reports_a_failed_rename_and_removes_its_temp_file
Name limits; reserved name space; a file name with a NUL byte fallback_name_limits_hold_for_the_check_and_the_write, boot_refuses_a_db_name_that_cannot_get_a_fallback_marker
Document without the new field; unknown field marker_without_keystore_dir_still_parses, marker_with_an_unknown_field_is_corrupt

Drain fixtures:

  • Three tests use a missing keystore directory. They check that the drain writes the owned fallback.
  • One test injects failures in both marker writes. It checks that the durable keystore record prevents boot.
  • drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds: The test uses a missing keystore directory and a hook that fails both marker writes. The drain waits. It returns exit code 12 after the fallback write succeeds.

All tests pass on 804257da. Each of the six new or changed tests of fix round 3 passes 50 repeat runs. The fix round 2 tests passed 50 repeat runs on 23d23701.

Mutants

Fix round 3: cargo mutants --in-diff <git diff 39cc4149..804257da> -p rvc -p rvc-bin --test-tool nextest --all-features, on the final head.

Caught Missed Unviable Timeout
12 0 1 0
  • Each caught mutant makes a test fail on an assertion. No mutant is a timeout.
  • One generated mutant does not compile (Default::default() for MarkerDir, which has no Default): MarkerDir::open. Its hand mutant U7 fails on assertions on the final head.
  • The test-only seam arm_walk_start (#[cfg(test)]) gives no mutant.
  • Fix round 3 hand mutants T1 to T9 change the B1 and B3 fixes. Each one fails on assertions on the final head:
    • T1: a bare file name gives no directory again. "The names slashing.sqlite and ./slashing.sqlite must select the working directory and the same marker." Also "Boot must return exit code 12 for the database name slashing.sqlite."
    • T2: the selection compares the canonicalize results again. "The selection must give the database directory for a database directory that is a symlink to the keystore. The result is None."
    • T3: the selection compares path strings. "A bare file name must select no directory when the working directory is the keystore directory."
    • T4: a walk error counts as one directory. "The selection must give the database directory for a database directory that is a symlink to the keystore." Also the boot row "a symlink to the keystore directory".
    • T5: the walk follows a symlink. The same two texts as T4.
    • T6: the selection reads the keystore directory by path. "The selection must give the database directory for a keystore path that is a symlink to the database directory."
    • T7: the selection reads the database directory by path. The same two texts as T4.
    • T8: resolved_owner calls canonicalize again. "A clear for keys must keep the marker and return the path of the symlink as the owner."
    • T9: the owner keeps a trailing slash. "A clear for the path of the symlink with a trailing slash must remove the marker."
  • The RSA fix2 expectations fail on the fix3 code. Probe P1 is the RSA B1 assertion fallback_marker_dir(&keys, Path::new("slashing.sqlite")) == None: the result is Some("."). Probe P2 is the RSA B3 claim that equal resolved paths give None: the result is Some("<root>/slash-link").
  • The fix2 hand mutants K1 to K9, the reviewer mutants (RSA M01 to M13, Harper C4, D1 and D2), the hand mutants U1 to U10 and C1, and the guards G29, G33 and G45 fail on assertions again on the final head. The hand mutant C1 now causes 21 boot error checks to fail. The new boot test of fix round 3 adds one. C1 also causes two rvc-bin integration tests to fail their exit code checks.

Fix round 2: cargo mutants --in-diff <git diff 0145a66a..23d23701> -p rvc -p rvc-bin --test-tool nextest --all-features, on the fix2 head.

Caught Missed Unviable Timeout
14 0 3 0
  • Each caught mutant makes a test fail on an assertion or on a named check. No mutant is a timeout.
  • Three generated mutants do not compile (Default::default() for a type without Default): inspect_fallback, clear_fallback_marker (Ok(vec![Default::default()])) and MarkerDir::open. Their hand mutants are U2, U3 and U7 below. Each one fails on assertions on the final head.
  • The guard mutant err.kind() == NotFound with true in MarkerDir::check_db_entry fails on "The check must refuse a name with a NUL byte and give the reason "nul byte"."

Fix round 1: cargo mutants --in-diff <git diff 920a7afa..0145a66a>, in two shards, on 0145a66a.

Caught Missed Unviable Timeout
50 0 10 0
  • Each caught mutant makes a test fail. No mutant is a timeout.
  • Ten generated mutants do not compile (Default::default() for a type without Default). Each corresponding hand mutant fails an assertion. Fix round 2 runs them again on the final head:
    • U1: for_config returns empty paths.
    • U2: inspect_fallback returns two Absent values.
    • U3: clear_fallback_marker returns one empty Removed.
    • U4: inspect returns Absent.
    • U5: inspect_at returns Absent.
    • U6: inspect_file returns Absent. "The reader must return Present for a version-1 marker without keystore_dir. The result is Absent."
    • U7: MarkerDir::open returns a handle of /. "The read must give Unreadable and the clear must fail for a missing directory."
    • U8: MarkerDir::read returns Absent. "The read must give Unreadable and the clear must fail for a symlink at the owned path."
    • U9: MarkerDir::lock returns a file without flock. "The lock try in the write must fail."
    • U10: open_at returns /dev/null. "The check must accept a name of 228 bytes."
  • The hand mutant C1 replaces the body of run_with_clock with Ok(()). The hand mutant causes 20 boot error checks to fail. It also causes two rvc-bin integration tests to fail their exit code checks. Fix round 1 reported 19 boot error checks. The new boot test of fix round 2 adds one.
  • The reviewer mutants of fix round 1 (RSA M03, M05, M07, M10, M11 and Harper C4) fail on assertions on the final head. The earlier hand mutants and the reviewers' killed mutants fail on assertions again.
  • Fix round 2 hand mutants K1 to K9 change the B1 and B3 fixes. Each one fails on assertions:
    • K1: MarkerDir::open runs canonicalize again. "The read must give Unreadable for the two markers for a symlink in an intermediate component."
    • K2: a missing directory reads as absent in inspect_fallback and in the clear. "The read must give Unreadable and the clear must fail for a missing directory."
    • K3: boot does not check the database file. "The check must refuse <root>/other/link.sqlite and give the reason "one link"."
    • K4: no check of the number of links. "The check must refuse <root>/other/hard.sqlite and give the reason "one link"."
    • K5: the walk opens each name without O_NOFOLLOW. "Boot must return exit code 1 and an error with the text "no symlink" for a symlink in the directory path."
    • K6: the walk accepts a ... "The read must give Unreadable for the two markers for a .. component."
    • K7: the clear does not check the database file. "The clear for <root>/other/link.sqlite must fail."
    • K8: the check of the database file follows a symlink. "Boot must return exit code 1 and an error with the text "one link" for a symlink to the database file."
    • K9: boot ignores an error of the walk. "Boot must return exit code 1 and an error with the text "no symlink" for a symlink in the directory path."
  • Fix1 also moves the fstat check and the bounded read of doppelganger::marker_file::read_marker_file into read_open_marker, with no change to the logic. The mutant command covers rvc and rvc-bin only. The rvc tests reach that code through read_marker_file and MarkerDir::read.

Gates

The gate results apply to 804257da724d4586401934a9a8e30b99b035ffd0 with rustc 1.100.0. The builds use private target directories. All gates ran on the final head. The branch is on develop 49d4cacdafe597490dd736da466eda660722c57a (fix round 2 moved it there; #1014 changes only docs/releases/UNRELEASED.md).

Gate Result
cargo fmt --all -- --check PASS
cargo build --workspace --all-features PASS (one warning on the base: unused dependency toml in rvc-config)
cargo clippy --workspace --all-targets --all-features -- -D warnings PASS
cargo nextest run --workspace --all-features --no-fail-fast PASS: 7807 tests run, 7807 passed (1 slow), 10 skipped. This includes rvc (1905 passed), rvc-bin (169 passed) and rvc-architecture-tests (409 passed).
cargo build --release -p rvc-bin PASS (one warning on the base: first_ok_with is never used in rvc-bn-manager)
stress, 50 iterations for each of the six new or changed tests of fix round 3 PASS (each run: Starting 1 test, 50 passed)
cargo deny check advisories sources PASS: advisories ok, sources ok
cargo mutants --in-diff <git diff 39cc4149..804257da> -p rvc -p rvc-bin PASS: 13 mutants, 12 caught, 0 missed, 0 timeout, 1 unviable (with a hand mutant that fails on assertions)
gitleaks tree scan (strict config, 8.21.2) PASS: no leaks found
scripts/gitleaks_range_scan.sh <strict> 49d4cacdafe597490dd736da466eda660722c57a PASS: 13 commits scanned, no leaks found

Pre-PR self-check

  • Red check: each new acceptance test shown failing on base or with the fix removed. The five fix3 tests fail on assertions on 7c45722e, which keeps the fix2 selector. Hand mutants T1 to T9 of the fix3 code fail on assertions on the final head. The five fix2 tests failed on 1d86c608 (d71cd908), and hand mutants K1 to K9 fail on assertions on the final head.
  • Measurement claims ("no regression") come from a scenario where unfixed code scores worse. Not applicable: this PR makes no measurement claim.
  • Repo-wide grep for every changed symbol/pattern the issue targets: fallback_marker_dir, same_dir, resolved_owner, fallback_marker_name, FallbackClear, LegacyRemoved, atomic_write, write_marker(, clear_marker(, inspect_fallback, MarkerDir::open, check_slashing_db_name, arm_after_marker_dir_resolve, arm_walk_start, canonicalize, realpath, .rvc.doppelganger. No production path writes the legacy name beside the database. No caller uses LegacyRemoved. MarkerDir::open has five callers: same_dir (the selection), check_slashing_db_name, inspect_fallback, write_fallback_marker and clear_fallback_marker. None of them resolves the path first. The selection callers are check_slashing_db_name, refuse_if_detection_records, the drain setup in run_with_clock and execute_clear_detection, each through FallbackMarkerPath::for_config. In git diff 49d4cacd...HEAD, canonicalize is only in two test lines, and the diff removes the canonicalize of the develop same_dir. No line has realpath. No code uses arm_after_marker_dir_resolve.
  • Signing-root / hash_tree_root tests are KAT-anchored or kat_exempt with a reason. No new or renamed test name matches .*(tree_hash|signing_root|_root)$.
  • The format check, Clippy, and the tests passed on toolchain 1.100.0 on 804257da before the PR update.

Decisions (intentional)

From the issue:

  • Boot selects the fallback marker by its database file name. The clear also compares the recorded keystore path.
  • Downgrade is not fail-closed for the owned name, and the release note says so. An older release does not read .rvc.doppelganger_detected.<db file name>. It still reads the record, the keystore marker and the legacy name. If both keystore writes fail, the owned fallback can be the only detection record. Writing the legacy name too would bring back the shared-name hazard.
  • The fallback of one instance does not stop the boot of another instance. The evidence of the second instance (its record, its keystore marker, its owned fallback) still stops it.
  • The keystore marker keeps no owner field. It is in the keystore directory that it belongs to.

Spec change that Ash approved (RSA B5):

  • clear-detection keeps an owned fallback that records no owner and a legacy fallback. Issue [W4-D2a] Fallback marker: one owned file per slashing database, refused at boot and cleared only by its own keystore #888 said that the command removes a legacy fallback with a note. RSA B5 showed that such a file can be the only detection evidence of another instance. The command now keeps the file, returns an error with its path, and stops before the keystore marker. The recovery step has four parts:
    1. Stop each validator client whose slashing database is in that directory.
    2. Make sure that its validators do not run in another place.
    3. Remove the file.
    4. Run clear-detection for each of those clients before it starts again.

Choices in this PR (details in the run's findings.md):

  • The database identity is the file name in the directory that the walk opens, and that name must be the only name of the file (fix round 2, RSA B1). Boot refuses a symlink to the database file, a symlink or a .. in the directory path, a missing database directory, and a database file with more than one link (exit code 1). The clear refuses the same paths. Because of this, two names of one database select one marker, or boot refuses the second name. The marker name does not change. A marker name from the inode of the file was not chosen: a new database has no inode before SQLite creates it, and a restore from a backup gives a new inode.
  • MarkerDir::open walks the configured path with no resolve (fix round 2, RSA B3). The walk starts at / or at the working directory and opens each name with O_NOFOLLOW | O_DIRECTORY. It does not call canonicalize. A symlink, a .. or a missing name stops the walk. A missing directory is not absent. Boot, the drain and the clear each open their own handle.
  • One walk for the selection, the read, the write and the clear (fix round 3, RSA B1 and B3). The fallback directory is the parent of the database, or . for a bare file name. The selection opens that directory and the keystore directory with the walk of MarkerDir::open and compares (st_dev, st_ino) of the two handles. It does not compare or resolve paths. It selects no fallback only when the two handles are one directory. A walk error or a metadata error gives the database directory, so boot reads the fallback and fails closed. Each step opens its own handle with the same walk, so a later change to a path cannot make a step skip the fallback.
  • The keystore marker writer opens the keystore directory by its path (fix round 2). Boot reads the keystore marker, the record and .rvc.lock by that path too. The walk with no symlink is for the fallback steps and for the selection. Fix round 3 removes the canonicalize comparison that fallback_marker_dir kept from develop.
  • New limit (fix round 2). When the database directory is not the keystore directory, a path with a symlink is refused. Examples are a systemd StateDirectory= with DynamicUser=yes, and the macOS /tmp. A backup tool that keeps a hard link of the database file also causes a refusal. Use the real path, or remove the second link. The error names the path and the rule.
  • New limits (fix round 3).
    • Boot also refuses (exit code 1) a database directory with a symlink in its path when that directory is the keystore directory. The walk cannot show that the two are one directory. Use the real path.
    • A keystore path with a symlink: the selection cannot show that the database directory is the keystore directory, so boot reads the fallback in the database directory. If that directory is the real keystore directory, its legacy name is the keystore marker. clear-detection --slashing-db-path then stops at the fallback step while that marker exists. Give the real keystore path in --datadir.
    • A bare database name gets its fallback in the working directory, unless the working directory is the keystore directory.
    • A clear with another path of the same keystore (a symlink or a ..) keeps the marker and prints the recorded owner. Run the clear again with that path in --datadir.
  • The owner is the keystore path that the writer makes absolute at write time (fix round 3: no canonicalize). The working directory joins a relative path. The result has no . component and no trailing slash. The writer does not resolve a symlink or a .., and it reads no file system entry, so a missing keystore directory still gives an owner (the drain writes the fallback when the keystore write fails). A path that is not valid UTF-8 is not recorded, so the clear keeps the marker.
  • A bare database name (fix round 3). Issue [W4-D2a] Fallback marker: one owned file per slashing database, refused at boot and cleared only by its own keystore #888 left 746-1a ("A bare database file name getting no fallback") to W4-D2b. RSA B1 and the fix round 3 brief require now that a bare name means .. W4-D2b keeps the resolved clear command and the fallback-dir probe (746-1c).
  • One lock for each database directory. The lock does not wait. The drain tries again in 1 s. The CLI stops with an error that names the lock path.
  • MarkerDir::replace is the one marker writer. The keystore marker and the force-window file also use it. Their file format and fsync order do not change.
  • The C8 limit is closed. The temp name has at most 57 bytes. Boot refuses a database file name whose marker name has more than 255 bytes (exit code 1), before the client can sign. A database file name of 228 bytes works.
  • fallback_marker_name returns OsString, not the String of the issue. A lossy string can give two databases one name.
  • Unix only. The directory handle uses openat, renameat, unlinkat and flock through libc, which is already a direct dependency. No new crate. On another platform, the marker writes and the clear return Unsupported, and boot reads the fallback paths by path. There, the selection always gives the database directory. CI builds Linux only.
  • Commit b3c05934 (4b032d4c) adds test-only hooks (arm_after_marker_dir_resolve, arm_before_marker_temp_open, arm_before_fallback_unlink), FALLBACK_LOCK_FILENAME, and a check_slashing_db_name that accepts every name, so that each new test compiles and fails on an assertion. Fix round 2 removes arm_after_marker_dir_resolve.
  • The test-only seam arm_walk_start (fix round 3). A test with a bare name needs a working directory that holds a marker. set_current_dir changes the working directory of the process, and a different test in the same process can see it. The seam changes the start of each walk of a relative path on the test thread only. It has separate #[cfg(test)] and #[cfg(unix)] attributes, so cargo mutants does not mutate it.
  • The test hook for marker-write failures applies to both writers. The in-place test checks that the fallback write also fails.
  • RSA N3 and Harper N1. The first two sentences of "Test cases" are the sentences that RSA N3 gives (fix round 1 and fix round 2). In the first sentence, "red commits" names the commits. The three entries after them are the three earlier rounds. Fix round 3 has a separate paragraph after them. The table uses "With the previous behavior" for the results of the earlier tests.

Release note

Text for W4-R4 (docs/releases/UNRELEASED.md; this PR does not edit it):

Fallback detection marker: one file for each slashing database. If the keystore and database directories differ, the drain can write the marker beside the database. The file name is .rvc.doppelganger_detected.<db file name>. The file records the keystore path. Boot stops with exit code 12 if it finds the owned file or the legacy file .rvc.doppelganger_detected. The writes and clears of these files in one directory hold the lock file .rvc.doppelganger_lock. rvc doppelganger clear-detection --datadir <keys> --slashing-db-path <db> --confirm removes the owned file only when it belongs to --datadir. It keeps a file that belongs to another keystore and prints its path and owner. If the owned file cannot be read, does not parse, or records no owner, or if a legacy file exists, the command stops and prints the path. Before you remove that file, stop each validator client whose slashing database is in that directory. Make sure that its validators do not run in another place. Remove the file. Run the clear command for each of those clients before it starts again. Boot refuses a slashing database file name that starts with .rvc.doppelganger (exit code 1). Unless boot can show that the database directory is the keystore directory, boot also refuses these paths (exit code 1): a file name longer than 228 bytes, a missing database directory, a symlink or a .. in the directory path, a database file that is a symlink, and a database file with more than one hard link. A symlink in the directory path is refused also when it points to the keystore directory. Use the real path of the database. For a bare file name such as slashing.sqlite, the marker is in the working directory. The file records the keystore path as given, made absolute, so give --datadir with the same path as the validator client.

Migration: Do not install an older release until you clear the detection. An older release does not read the owned fallback name. If both keystore writes fail, the owned fallback can be the only detection record.

Running-guide delta

For docs/running-guide.md §Shutdown (W4-R4 copies it; this PR does not edit it):

If the keystore and database directories differ, the drain also writes the marker beside the database as .rvc.doppelganger_detected.<db file name>. The file records the keystore path. Earlier releases used the name .rvc.doppelganger_detected there. Nothing writes that name now. Boot stops while either file exists. The database path must be the only name of the database: no symlink and no .. in the directory path, and one hard link for the database file. Else boot stops with exit code 1. For a bare file name, the marker is in the working directory. clear-detection --slashing-db-path <db> removes only the owned file that belongs to --datadir. Give --datadir with the same path as the validator client, because the file records that path. It keeps a file that belongs to another keystore and prints its path and owner. If the command cannot read or parse the owned file, if the file records no owner, or if a legacy file exists, the command stops and prints the path. Stop each validator client whose slashing database is in that directory. Make sure that its validators do not run in another place. Remove the file. Run the clear command for each of those clients before it starts again. Do not install an older release until you clear the detection.

Fix round 1

Reviews of 58ed07d1: Harper (6 BLOCKING, 1 NIT) and RSA (11 BLOCKING, 3 NIT). One line for each finding:

  • Harper B1 (a marker write can replace another database): boot refuses a database file name in the reserved name space .rvc.doppelganger* before SQLite opens it, and write_fallback_marker does not replace an entry that is not a marker of the same keystore.
  • Harper B2 (two Unix names can select one marker): fallback_marker_name returns an OsString with the native bytes of the database file name.
  • Harper B3 (the recorded owner can name another keystore): the writer records the absolute keystore path that it resolves at write time, or no owner if the path is not valid UTF-8. The clear compares that string and does not resolve it again. A marker with no owner stays.
  • Harper B4 (a valid name can prevent a durable write): the temp name has at most 57 bytes. Boot refuses a database file name whose marker name has more than 255 bytes (exit code 1).
  • Harper B5 (wall-clock sleep): the two-instance test and the in-place test use paused Tokio time and the join handle of the drain.
  • Harper B6 (the CLI tests pass when the command discards a fallback error): clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner checks an owned file that does not parse and an owned directory first. Harper C4 fails on its assertion.
  • Harper N1 (STE): the new commit messages, this PR body, and commit 4f4bccc0 for the round-1 test prose.
  • RSA B1 (database and keystore identities): as Harper B2 and B3. Fix round 2 completes the database identity.
  • RSA B2 (protect other files in the database directory): the reserved name space, the no-replace rule, O_EXCL | O_NOFOLLOW for the temp file, and a cleanup that removes only a temp file that the writer created.
  • RSA B3 (keep file operations in the selected directory): MarkerDir uses one handle for the reads, the temp create, the rename, the unlink, the lock and the directory fsync. Fix round 2 removes the resolve before the walk.
  • RSA B4 (serialize fallback updates and clear): flock on .rvc.doppelganger_lock in the database directory. The owner read, the replace check and the unlink occur under that lock. The drain does not replace a marker of another owner.
  • RSA B5 (preserve fallback evidence with unknown ownership): the clear keeps an owned marker with no owner and a legacy marker, and returns the path and the recovery step. This is a spec change that Ash approved. The round-1 test clear_removes_a_fallback_marker_beside_the_slashing_db is now the row "a legacy marker" of clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner.
  • RSA B6 (check both filename limits before startup): as Harper B4.
  • RSA B7 (CLI fallback error): as Harper B6. RSA M03 fails on the same assertion.
  • RSA B8 (clear result for an unreadable owned entry): clear_fallback_marker_keeps_an_unreadable_owned_entry. RSA M11 fails on its assertion.
  • RSA B9 (boot refusal for an unreadable owned fallback): boot_refuses_an_unreadable_owned_fallback_marker. RSA M05 fails on its assertion.
  • RSA B10 (latch when every durable write fails): drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds. RSA M07 fails on its assertion.
  • RSA B11 (exclusive creation in the shared marker writer): MarkerDir::replace is the one marker writer. fallback_write_keeps_an_entry_that_it_did_not_create checks it. RSA M10 fails on its assertion.
  • RSA N1 (direct words in the test requirements): commit 4f4bccc0 changes the five locations.
  • RSA N2 (drain fixtures): the heading "Test cases", the label "The selected keystore owns the fallback marker.", and the drain fixture text.
  • RSA N3 (test results): the texts of N3 in "Test cases" and "Mutants".

Fix round 1 also fixes two architecture gates that the first workspace run found (commit caf6c0f2, 74f078c8 before the rebase). A list of the in-diff mutants then showed four kinds of mutant that no test could kill. Commit 20da6d14 (0145a66a) adds two tests for them: a missing database directory, a directory path under a file, a symlink at the owned path, and a failed rename of a marker write. The behavior does not change.

Fix round 2

Reviews of 0145a66a: Harper CLEARED (N1 NIT open) and RSA BLOCK (B1 and B3 NOT FIXED, N2 and N3 NOT FIXED). One line for each finding:

  • RSA B1 (stable database identity): check_slashing_db_name and clear_fallback_marker refuse a second name of a database (a symlink to the file, a symlink or .. in the directory path, a missing directory, a file with more than one link), so boot, the drain and the clear use one name of each database. Tests: second_names_of_a_database_are_refused_by_the_check_and_the_clear, boot_refuses_a_second_name_of_a_database_with_a_fallback_marker. Hand mutants K3, K4, K7, K8 and K9 fail on assertions.
  • RSA B3 (no symlink resolution before the walk): MarkerDir::open does not call canonicalize. It walks from / or the working directory with O_NOFOLLOW | O_DIRECTORY and refuses a ... A missing directory gives Unreadable and an error, not absent. Tests: fallback_paths_with_a_symlink_or_a_parent_name_are_refused, fallback_reads_fail_closed_for_each_directory_state (row "a missing directory"). Hand mutants K1, K2, K5 and K6 fail on assertions.
  • RSA N2 (drain fixtures): each fixture group is a separate list item, and the third item uses the sentences of the review.
  • RSA N3 (test results): "Test cases" starts with the sentence of the review, and "Mutants" uses the sentence of the review for the hand mutant C1.
  • Harper N1 (PR body wording): "State ownership" replaces "Shared invariant" with direct sentences, and the test table uses "With the previous behavior" in place of "red". The fix2 commit message 23d23701 gives the corrected wording. The earlier commit messages do not change.

Fix round 3

Reviews of 39cc4149: Harper CLEARED (N1 NIT open) and RSA BLOCK (B1 and B3 NOT FIXED for the second time, N3 NOT FIXED). One line for each finding:

  • RSA B1 (stable database identity): fallback_marker_dir gives . for a bare file name, so slashing.sqlite and ./slashing.sqlite select one directory and one marker, and boot reads it. Tests: fallback_dir_is_the_slashing_parent_only_when_it_differs (changed: the assertion that expected None for a bare name is gone), boot_refuses_an_owned_fallback_marker_of_a_bare_db_name. Hand mutant T1 and the RSA probe P1 fail on assertions.
  • RSA B3 (no symlink resolution before the walk): the selection opens the database directory and the keystore directory with the walk and compares (st_dev, st_ino) of the two handles; an error gives the database directory; no step calls canonicalize (also not resolved_owner). Tests: fallback_selection_does_not_skip_a_directory_behind_a_symlink, fallback_owner_keeps_a_symlink_in_the_keystore_path, the row "a symlink to the keystore directory" of boot_refuses_a_second_name_of_a_database_with_a_fallback_marker. Hand mutants T2 to T9 and the RSA probe P2 fail on assertions.
  • RSA N3 (test results): "Test cases" uses "The next three entries give the test results with the previous behavior.", and the three entries follow it.
  • Harper N1 (STE): clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore uses "The function must return InvalidData with the path of the marker with invalid JSON." and "The function must keep the marker with invalid JSON."

Size

  • Fix round 3: git diff --shortstat 39cc4149ffc3e8de7f9a1032bc3fae5eb20df2b1..HEAD: 3 files changed, 297 insertions(+), 61 deletions(-). The total is 358 lines, below the limit of about 600 for this round. Cargo.lock has no change.
  • Fix round 2: git diff --shortstat 20da6d14..39cc4149: 3 files changed, 262 insertions(+), 117 deletions(-). The total is 379 lines.
  • Fix round 1: git diff --shortstat 343f32d6..20da6d14: 5 files changed, 1303 insertions(+), 197 deletions(-). The total is 1,500 lines.
  • Whole PR: git diff --shortstat 49d4cacdafe597490dd736da466eda660722c57a...HEAD: 5 files changed, 2110 insertions(+), 121 deletions(-). The total is 2,231 lines. Cargo.lock has no change. This is 431 lines above the line of 1,800 at which issue [W4-D2a] Fallback marker: one owned file per slashing database, refused at boot and cleared only by its own keystore #888 says to stop and report. Fix round 2 reported 1,965 lines. This section reports the new total.
  • The branch is on develop 49d4cacdafe597490dd736da466eda660722c57a. Fix round 2 moved the 11 earlier commits onto it with no change (the range-diff shows = for each commit). The SHAs before that move are in parentheses in "Test cases". Fix round 3 adds two commits on 39cc4149.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report

Total: 94.71% (141755 / 149666 lines)

File Coverage Lines
src/bls_to_execution.rs 92.5% 246/266
src/deposit.rs 100.0% 295/295
src/existing_mnemonic.rs 86.3% 120/139
src/exit.rs 94.2% 259/275
src/fs_util.rs 98.5% 64/65
src/main.rs 0.0% 0/51
src/network.rs 100.0% 138/138
src/new_mnemonic.rs 92.7% 460/496
src/password.rs 92.6% 212/229
src/verify.rs 97.6% 82/84
src/main.rs 95.1% 887/933
src/cli.rs 90.2% 542/601
commands/doppelganger.rs 96.4% 370/384
commands/health.rs 100.0% 5/5
commands/prepare_exit.rs 90.1% 182/202
commands/signed_exit.rs 97.5% 1101/1129
commands/slashing.rs 93.5% 144/154
commands/submit_exit.rs 75.3% 73/97
commands/voluntary_exit.rs 63.1% 41/65
src/logging.rs 90.4% 641/709
src/main.rs 100.0% 44/44
bin/generate_architecture_md.rs 0.0% 0/8
src/lib.rs 83.5% 192/230
src/rust_source.rs 93.9% 1381/1471
src/admission.rs 86.4% 51/59
src/client.rs 88.3% 1884/2133
src/error.rs 100.0% 57/57
src/http_caps.rs 89.6% 43/48
src/prepare_check.rs 85.0% 306/360
src/retry.rs 100.0% 53/53
src/shape.rs 100.0% 30/30
src/ssz_deser.rs 95.3% 631/662
src/types.rs 95.7% 1882/1967
src/v4_wire.rs 100.0% 87/87
src/error.rs 100.0% 22/22
src/metrics.rs 98.9% 89/90
service/mod.rs 88.4% 1305/1477
src/traits.rs 73.1% 68/93
src/types.rs 100.0% 7/7
src/validation.rs 99.0% 191/193
src/admission.rs 98.5% 194/197
src/broadcast.rs 98.0% 146/149
src/error.rs 100.0% 13/13
src/fan_out.rs 94.8% 2581/2723
src/header_quorum.rs 100.0% 238/238
src/health.rs 86.9% 292/336
src/manager.rs 95.5% 8800/9214
src/metrics.rs 98.4% 186/189
src/mock.rs 93.6% 1132/1210
src/revocation_commit.rs 86.4% 51/59
src/sse.rs 87.3% 1440/1649
sse/frame.rs 98.1% 363/370
src/submit.rs 96.9% 557/575
src/sync_status.rs 98.3% 920/936
src/traits.rs 96.1% 395/411
src/types.rs 99.2% 245/247
src/service.rs 98.9% 2411/2439
src/traits.rs 73.9% 17/23
src/bls.rs 90.2% 220/244
src/composite_signer.rs 92.9% 469/505
src/eip2333.rs 97.1% 233/240
src/error.rs 100.0% 63/63
src/insecure.rs 100.0% 131/131
src/key_manager.rs 94.7% 791/835
src/keystore.rs 98.2% 1471/1498
src/mnemonic.rs 100.0% 91/91
src/signer_trait.rs 97.8% 89/91
src/signing.rs 97.7% 126/129
src/signing_root.rs 98.3% 456/464
src/test_utils.rs 100.0% 41/41
src/typed_signer.rs 94.4% 656/695
src/voluntary_exit_signing.rs 100.0% 129/129
src/enablement.rs 100.0% 26/26
src/epoch_clock.rs 97.7% 84/86
src/forward_window.rs 97.7% 1052/1077
src/lib.rs 100.0% 34/34
src/marker_file.rs 97.1% 132/136
src/restart_skip.rs 100.0% 34/34
src/duty.rs 100.0% 47/47
src/metrics.rs 100.0% 26/26
src/tracker.rs 98.6% 3401/3450
src/aggregation.rs 100.0% 307/307
src/aggregator.rs 96.4% 81/84
src/block.rs 99.4% 831/836
src/block_body.rs 94.3% 1508/1599
src/block_body_json.rs 96.4% 397/412
src/builder.rs 99.0% 96/97
src/builder_request_auth.rs 90.8% 108/119
canonical/gvr_hex.rs 100.0% 25/25
canonical/pubkey_hex.rs 98.4% 63/64
canonical/signing_root_hex.rs 100.0% 13/13
src/deposit.rs 98.2% 161/164
src/domains.rs 100.0% 64/64
src/duties.rs 98.4% 63/64
src/fork.rs 99.3% 435/438
src/hex_fixed.rs 100.0% 108/108
src/lib.rs 96.3% 131/136
src/networks.rs 100.0% 91/91
src/payload_attestation.rs 98.5% 64/65
src/proposer_preferences.rs 86.4% 19/22
src/serde_signature.rs 97.1% 68/70
src/signed_block_contents_json.rs 100.0% 154/154
src/ssz_helpers.rs 95.9% 396/413
src/sync_committee.rs 99.0% 307/310
src/tree_hash_utils.rs 93.4% 253/271
src/client.rs 91.0% 998/1097
src/lib.rs 100.0% 212/212
src/auth.rs 95.4% 334/350
src/error.rs 95.2% 258/271
src/handlers.rs 81.6% 678/831
src/lifecycle.rs 93.5% 1551/1658
src/server.rs 84.7% 400/472
src/traits.rs 97.6% 40/41
src/types.rs 96.4% 80/83
src/url_validator.rs 91.5% 193/211
src/definitions.rs 93.7% 177/189
src/lib.rs 95.3% 241/253
src/server.rs 95.3% 306/321
src/hex.rs 100.0% 39/39
src/logging.rs 99.3% 288/290
src/pubkey.rs 100.0% 62/62
src/span_names.rs 100.0% 70/70
src/client.rs 80.5% 124/154
src/wire.rs 95.1% 705/741
src/error.rs 100.0% 6/6
src/lib.rs 100.0% 10/10
src/network.rs 100.0% 43/43
sections/beacon.rs 0.0% 0/12
sections/builder.rs 98.2% 109/111
sections/builder_limits.rs 54.5% 12/22
sections/duties.rs 100.0% 55/55
sections/fork_schedule.rs 79.3% 46/58
sections/grpc_signer.rs 0.0% 0/8
sections/keymanager.rs 95.8% 114/119
sections/keys.rs 0.0% 0/9
sections/logfile.rs 62.5% 15/24
sections/mod.rs 99.3% 139/140
sections/monitoring.rs 46.2% 6/13
sections/network.rs 0.0% 0/8
sections/proposer_config.rs 62.5% 20/32
sections/safety.rs 61.9% 13/21
sections/secret_provider.rs 24.0% 6/25
sections/server.rs 0.0% 0/3
sections/slashing.rs 0.0% 0/6
sections/timing.rs 100.0% 94/94
sections/tracing.rs 61.0% 25/41
containers/attestation.rs 93.9% 123/131
containers/block_body.rs 97.9% 94/96
containers/body_leaves.rs 95.1% 214/225
containers/builder_requests.rs 95.6% 65/68
containers/envelope.rs 95.9% 94/98
containers/execution_payload.rs 94.4% 51/54
containers/leaves.rs 95.9% 162/169
containers/payload.rs 93.6% 103/110
containers/requests.rs 97.8% 44/45
src/kat.rs 100.0% 49/49
src/lib.rs 100.0% 71/71
src/merkle.rs 94.6% 88/93
src/roots.rs 96.5% 390/404
bin/gen_spec_kat.rs 81.3% 1091/1342
src/loader.rs 87.1% 196/225
src/lib.rs 92.0% 150/163
background_tasks/config_url.rs 95.8% 2917/3046
background_tasks/monitoring.rs 93.3% 280/300
src/beacon_adapter.rs 98.0% 484/494
bootstrap/beacon.rs 97.9% 512/523
bootstrap/enablement.rs 98.2% 1222/1244
bootstrap/executor.rs 95.6% 739/773
bootstrap/keys.rs 78.4% 333/425
bootstrap/mod.rs 100.0% 60/60
bootstrap/run.rs 94.1% 1792/1905
bootstrap/services.rs 97.8% 750/767
bootstrap/signer_probe.rs 90.7% 603/665
bootstrap/slashing.rs 91.1% 205/225
bootstrap/tasks.rs 94.6% 930/983
config/builder.rs 93.2% 2739/2940
config/error.rs 100.0% 136/136
config/network.rs 100.0% 118/118
config/types.rs 96.1% 3752/3906
src/deletion_denylist.rs 92.3% 372/403
src/detection_marker.rs 97.0% 2776/2863
src/detection_stop.rs 98.8% 251/254
src/index_resolver.rs 97.4% 1216/1248
src/key_admission.rs 92.7% 612/660
keymanager_adapters/config.rs 100.0% 86/86
keymanager_adapters/doppelganger.rs 56.5% 78/138
keymanager_adapters/import_window.rs 96.6% 1004/1039
keymanager_adapters/kdf_budget.rs 100.0% 173/173
keymanager_adapters/keystore.rs 82.4% 742/901
keymanager_adapters/notifier.rs 84.0% 21/25
keymanager_adapters/remote_keys.rs 96.3% 105/109
keymanager_adapters/slashing.rs 83.1% 143/172
keymanager_adapters/spawn.rs 91.5% 182/199
keymanager_adapters/validator.rs 52.9% 46/87
keymanager_adapters/voluntary_exit.rs 99.3% 398/401
src/liveness_loop.rs 98.3% 3017/3070
src/metrics.rs 99.6% 242/243
orchestrator/aggregation.rs 92.4% 2341/2534
orchestrator/attestation.rs 86.4% 1216/1407
block_proposal/mod.rs 98.2% 56/57
coordinator/mod.rs 93.7% 947/1011
orchestrator/dispatch.rs 89.8% 150/167
orchestrator/duty_management.rs 98.5% 705/716
orchestrator/epoch_boundary.rs 99.0% 3822/3862
orchestrator/error.rs 100.0% 31/31
orchestrator/head_events.rs 99.4% 334/336
orchestrator/payload_attestation.rs 97.1% 2311/2380
orchestrator/slot_anchor.rs 94.9% 111/117
orchestrator/slot_context.rs 94.7% 1297/1369
orchestrator/sync_committee.rs 94.9% 2311/2436
orchestrator/utils.rs 97.6% 486/498
validation/attestation_data.rs 100.0% 149/149
src/pubkey_index.rs 98.1% 414/422
src/quiesce.rs 98.7% 298/302
src/slashing_monitor.rs 94.3% 280/297
src/startup.rs 97.2% 650/669
src/test_metrics.rs 100.0% 49/49
src/format.rs 82.8% 82/99
src/key_source_manager.rs 96.4% 1363/1414
src/lib.rs 97.6% 82/84
src/metrics.rs 100.0% 74/74
src/refresh.rs 97.7% 1231/1260
src/lib.rs 78.9% 310/393
src/lib.rs 100.0% 20/20
audit/cn.rs 83.6% 127/152
audit/log.rs 63.3% 31/49
audit/mod.rs 100.0% 41/41
backend/basic.rs 93.7% 311/332
backend/signer_adapter.rs 50.0% 6/12
src/config.rs 99.8% 823/825
src/error.rs 56.2% 27/48
src/gate_shared_across_transports.rs 100.0% 97/97
src/grpc_common.rs 99.5% 370/372
grpc_tls/mod.rs 96.7% 118/122
grpc_tls/server_builder.rs 100.0% 12/12
accept_loop/mod.rs 94.4% 117/124
http_api/dispatch.rs 98.9% 366/370
http_api/mod.rs 97.8% 132/135
http_api/pubkey.rs 98.7% 77/78
http_api/request.rs 95.1% 77/81
http_api/response.rs 98.8% 241/244
routes/mod.rs 96.5% 193/200
tls_config/mod.rs 93.8% 61/65
src/insecure_startup.rs 100.0% 65/65
src/metrics.rs 98.5% 658/668
src/reload.rs 78.8% 216/274
server/backend.rs 97.9% 94/96
server/grpc.rs 88.2% 150/170
server/http.rs 90.0% 235/261
server/mod.rs 92.9% 1168/1257
server/slashing.rs 84.9% 225/265
src/service.rs 95.4% 1879/1969
src/sign_plan.rs 98.7% 833/844
slashing/config.rs 97.3% 73/75
src/tls_io.rs 100.0% 11/11
src/trace_ctx.rs 100.0% 41/41
src/core.rs 92.5% 1853/2004
src/error.rs 98.1% 154/157
src/fail_closed.rs 100.0% 3/3
src/gate.rs 92.1% 314/341
src/lib.rs 96.5% 4228/4381
src/locks.rs 85.1% 40/47
src/metrics.rs 92.6% 225/243
service_util/circuit_breaker.rs 100.0% 129/129
src/test_utils.rs 100.0% 112/112
src/traits.rs 75.0% 12/16
src/audit.rs 91.7% 22/24
db/interchange.rs 96.5% 1285/1331
db/migrations.rs 80.6% 412/511
db/mod.rs 98.3% 788/802
db/open.rs 87.9% 326/371
db/records.rs 99.2% 492/496
db/watermarks.rs 97.1% 726/748
src/error.rs 100.0% 7/7
src/evaluate.rs 97.9% 94/96
src/group_commit.rs 89.4% 388/434
src/history.rs 97.9% 466/476
src/metrics.rs 95.7% 90/94
src/reader.rs 76.4% 42/55
src/rules.rs 92.6% 390/421
src/scoped.rs 100.0% 203/203
src/stage.rs 83.4% 267/320
src/types.rs 92.0% 380/413
src/config.rs 100.0% 190/190
src/file_appender.rs 98.3% 119/121
src/format.rs 92.6% 386/417
src/health_probe.rs 97.7% 128/131
src/init.rs 96.1% 370/385
src/propagation.rs 94.1% 256/272
src/rolling.rs 99.4% 330/332
src/shutdown.rs 90.6% 29/32
src/trace_id.rs 98.1% 158/161
src/clock.rs 96.5% 556/576
src/error.rs 100.0% 12/12
src/lib.rs 100.0% 19/19
src/block_selection.rs 100.0% 84/84
src/config.rs 100.0% 15/15
src/error.rs 94.7% 198/209
src/store.rs 97.0% 5647/5823
src/hex_serde.rs 90.2% 46/51
src/lib.rs 97.9% 568/580

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 R1 security review

Blake (RSA) reviews W4-D2a, issue #888, Class S. The review ends at 2026-11-19 06:37:30 KST (Asia/Seoul).

Verdict for 58ed07d1 — BLOCK

The review has 11 BLOCKING and 3 NIT findings.

Pins and lineage

Ref Commit SHA Tree SHA
Assigned head 58ed07d105e1affae6835e91972e1e6d442f021e 115fbda8135f18a0f23d892d82e3ee0845090d3a
Head parent and test commit 60ea92b6277f0167e81daf672a33ef77a3cbb566 54f11b104621e2e112b35acf4add3c5eb9f7e60f
Parent of the test commit and PR base 920a7afacce843426a54f719a198c5bc886a07e4 911477cb3bf504b2ac541df3cfe621c4fefa43c0

The head branch is fix/w4-d2a-888. The base branch is develop. The PR is open and in draft at both checks.
The local and live pins match at the start and before this report. The detached worktree is clean at both checks.

Git verifies both ED25519 commit signatures for rootwarp@gmail.com.
The two-commit range begins directly on develop. The symmetric range count is 0 2. This history permits a fast-forward.

Commit Stable patch ID
60ea92b6 1789d6fc28fed204fd27a16d67d9b93db5a25343
58ed07d1 9e35a75198b733232bd62f9e92e95ab8295d627d

The diff has four files, 554 insertions, and 52 deletions. Cargo.lock has no changes.
The assignment has no earlier review head or rebase pair. Start pins and signatures, tree and parent map, live start pins, final local pins, final live pins, assigned diff.

B1 — BLOCKING. Preserve database and keystore identities

crates/rvc/src/detection_marker.rs:131–154 derives the fallback name from the configured database basename. A database alias can select a different fallback for the same database.

The name converts non-UTF-8 bytes with to_string_lossy(). Distinct database names can select the same fallback. The pinned rusqlite accepts raw Unix path bytes.

Lines 188–192 use the same conversion for the owner. A lossy owner string can match an unrelated directory with a valid Unicode name.

The code also stores an unresolved owner path as supplied. Line 241 resolves that string at clear time. A relative path or a replaced directory alias can change the result.

Boot can miss the durable fallback for a database alias. Clear can accept the wrong keystore as the owner. The PR body's C7 note covers only the mismatch case.

Use a stable database identity for boot, write, and clear. Use a lossless keystore identity and compare it without re-resolving an unchecked owner string. Reject ambiguous identities. Source, SQLite path handling, live PR body.

B2 — BLOCKING. Protect other files in the database directory

crates/rvc/src/detection_marker.rs:179–185 passes a name in the database directory to atomic_write. Line 375 renames over an existing destination.

The destination name can be the filename of a different slashing database. crates/rvc/src/config/builder.rs:744–777 accepts this filename. The write can replace that database and destroy its signing history.

The error cleanup at line 379 can also unlink an entry after the exclusive open fails. It does not check whether this process created that entry.

Use a dedicated marker namespace. Reject database paths in that namespace. Refuse to replace an existing entry unless its identity and type permit the update. Clean up only entries that the writer created. Source.

B3 — BLOCKING. Keep file operations in the selected directory

crates/rvc/src/detection_marker.rs:110–123 compares canonical paths but returns the configured parent. Lines 149 and 154 keep and join that path.

The read at line 222 and unlink at line 245 resolve the parent separately. The write and directory sync at lines 358–376 also resolve the path again.

The leaf checks do not prevent resolution of an intermediate symlink. A change to the parent can redirect the operation or leave detection evidence in a different directory.

Resolve the directory beneath a trusted root without following symlinks. Use its descriptor for lookup, temporary creation, rename, unlink, and sync. Refuse operations if the required directory identity cannot be established. Source.

B4 — BLOCKING. Serialize fallback updates and clear

bin/rvc/src/commands/doppelganger.rs:43–47 locks one keystore. crates/rvc/src/bootstrap/slashing.rs:86–102 also uses a lock for one keystore.

Two keystores can refer to the same database. The slashing database open at crates/slashing/src/db/open.rs:48–74 does not hold an exclusive process lock for that database.

crates/rvc/src/detection_marker.rs:222–247 separates the owner read from the unlink. A different writer can replace the entry between them. The unlink can remove the new detection.

The drain also replaces an existing fallback without checking its owner at lines 179–185 and 375. This can discard the first detection without a concurrent clear.

Use a lock shared by all users of the same fallback. Verify ownership under that lock. Preserve unresolved detections when another keystore owns them. Source, live PR body.

B5 — BLOCKING. Preserve fallback evidence with unknown ownership

crates/rvc/src/detection_marker.rs:244–251 removes a valid owned file with no owner. It also removes the shared legacy file when one exists.

bin/rvc/src/commands/doppelganger.rs:43–64 locks only the selected keystore. It writes the force-window file only there.

A legacy fallback can still be the only durable detection for a different database. The command can erase that evidence without writing a force-window file for its owner.

The issue and the PR body specify automatic legacy removal. This behavior conflicts with the review requirement to preserve another instance's detection.

Keep ownerless evidence until its owner is known. Require a separate recovery step that checks all affected validators and preserves their startup protection. Source, issue #888, live PR body.

B6 — BLOCKING. Check both filename limits before startup

crates/rvc/src/detection_marker.rs:131–134 adds 27 bytes to the database name. Lines 358–366 add more bytes to the temporary name.

The filesystem has a 255-byte component limit. A 228-byte database name has a readable final name, but the temporary create fails with ENAMETOOLONG.

With no fallback file, crates/doppelganger/src/marker_file.rs:40–42 reports Absent. Boot passes the fallback gate at crates/rvc/src/bootstrap/run.rs:731–741.

The drain keeps the latch and retries when all writes fail. A restart after loss of both keystore paths can lack any detection evidence.

From 229 database bytes, the final read fails and boot returns exit 12. That limit does not protect the range where only temporary creation fails.

Use bounded names for both the destination and temporary file. Validate the fallback before startup permits signing. Add assertions for both filename limits. Source, filesystem results.

B7 — BLOCKING. Check how the CLI handles a fallback error

M03 discards the fallback error at bin/rvc/src/commands/doppelganger.rs:64. This change lets the command continue to the keystore marker unlink. The selected tests pass.

Add a CLI assertion for a fallback clear error. Require these results.

  • The command returns an error that identifies the fallback path.
  • The fallback entry stays.
  • The keystore marker stays.

Include an unreadable owned entry in the fixture. M03 log.

B8 — BLOCKING. Check the clear result for an unreadable owned entry

M11 treats an unreadable fallback as absent at crates/rvc/src/detection_marker.rs:225. The selected tests pass.

This change skips the error that protects evidence with an unknown owner. The test at lines 1904–1919 checks a document that fails to parse.

Add a unit test for an unreadable owned entry. Require these results.

  • The function returns InvalidData and gives the entry's path.
  • The owned entry stays.
  • The legacy fallback stays.

M11 log.

B9 — BLOCKING. Check boot refusal for an unreadable owned fallback

M05 skips the owned fallback refusal at crates/rvc/src/bootstrap/run.rs:735 when the entry cannot be read. The selected tests pass.

The owned fallback fixture at lines 1385–1424 contains valid JSON. The unreadable fixture at lines 1310–1313 uses the keystore marker.

Add a boot assertion for an unreadable owned entry beside the database. Require exit 12 before signer setup. M05 log.

B10 — BLOCKING. Check the latch when every durable write fails

M07 marks the detection durable after the fallback write fails at crates/rvc/src/bootstrap/run.rs:702–708. This change lets shutdown return exit 12 without durable detection evidence.

The selected tests pass. crates/rvc/src/liveness_loop.rs:3857–3907 starts with a record that can become durable. The fsync retry test at lines 4020–4074 has no fallback path.

Inject failures in all three writes. Assert that the drain stays active and the durability flag remains clear. Permit a successful write and require exit 12. M07 log.

B11 — BLOCKING. Check exclusive creation in the shared marker writer

M10 removes exclusive creation at crates/rvc/src/detection_marker.rs:366. The selected tests pass.

With the flag removed, a pre-existing temporary entry can be opened for write. The change weakens the protection against leaf symlinks and file collisions.

Add a test that supplies an existing temporary entry and verifies that its contents and the contents of its target stay intact. M10 log.

N1 — NIT. Describe the test requirements with direct words

Location Rewrite
crates/rvc/src/detection_marker.rs:1826 “The path must be in the database directory. The filename must contain the legacy name, a dot, and the database filename.”
crates/rvc/src/detection_marker.rs:1853, bin/rvc/src/commands/doppelganger.rs:183, and crates/rvc/src/liveness_loop.rs:3925 “The test must find a fallback path beside the database.”
crates/rvc/src/detection_marker.rs:1866 “The function must keep the other keystore's marker and return that keystore's canonical path.”
crates/rvc/src/bootstrap/run.rs:1382–1383 “Boot must refuse the fallback marker for its database. Write the JSON fixture directly.”
crates/rvc/src/liveness_loop.rs:3958–3959 “Instance B must not exit with code 12 when only instance A has a fallback marker. The error is {err}.”

At crates/rvc/src/detection_marker.rs:1833–1842, use this list.

  • When a different keystore owns the marker, keep the marker. Return its path and the keystore's canonical path.
  • When this keystore owns the marker and a legacy marker exists, remove both markers.
  • When the marker has no owner, remove it under the current test contract.
  • When the marker does not parse, keep it. Return InvalidData and include its path.

N2 — NIT. Describe the drain fixtures in the PR body

Replace the row at PR body line 74 with these two cases.

  • Three tests use a missing keystore directory. They check that the drain writes the owned fallback.
  • One test injects failures in both marker writes. It checks that the durable keystore record prevents boot.

Replace the case label at line 69 with “The selected keystore owns the fallback marker.” Replace the heading at line 62 with “Test cases.”

N3 — NIT. State the test results in the PR body

Lines Rewrite
47 and 51–58 “Commit 60ea92b6 keeps the previous behavior in temporary implementations. Run the 13 named tests on that commit. Five tests fail on assertions. Eight tests pass.” Use “Assertion failure” on lines 51–55. Use “Pass” on lines 57–58.
56 and 107 “Four tests cannot compile on 920a7afa. They pass with the temporary implementations in 60ea92b6. The listed hand mutants cause their assertions to fail.”
60 and 99 “All 13 tests pass on 58ed07d1. Each of the 11 new or changed tests passes 50 repeat runs.”
85–86 “Five generated mutants do not compile. Each corresponding hand mutant fails an assertion. The replacement for run_with_clock triggers 17 checks that require a boot error.”

The assertions and the named tests give these statements a clear meaning. These findings apply to new or changed prose. Live PR body, assigned diff.

Scoped checks

The first refusal check runs before signer setup at crates/rvc/src/bootstrap/run.rs:119. The next check runs after the keystore lock at crates/rvc/src/bootstrap/slashing.rs:102.

Boot returns exit 12 for a present, corrupt, or unreadable marker. They check the in-place record, the keystore marker, and both fallback names. The tests exercise the owned marker with detection disabled.

The process latch blocks new signing admissions at crates/doppelganger/src/forward_window.rs:816–825. The drain stops duty tasks before retrying marker writes.

The drain can acknowledge a durable record, a durable keystore marker, or a durable fallback. It retries if all writes fail. See crates/rvc/src/bootstrap/run.rs:634–715.

C4. The clear function returns InvalidData when it cannot read or parse an owned fallback. It keeps that entry. The CLI returns the error before the keystore marker unlink. The call search finds no alternative clear path after that error.

C7. The database suffix uses the final filename component. It cannot insert a separator. The lossy conversion can still select the wrong identity. See B1.

C8. The final name and temporary name have different limits. The write fails in a range where a missing final marker reads as absent. See B6.

On Unix, the reader rejects a leaf symlink before it opens the entry. Its open uses O_NOFOLLOW, O_NONBLOCK, and O_CLOEXEC.

The writer creates a temporary entry with exclusive creation. Its rename replaces the final directory entry. The clear unlinks the leaf entry. See B2 and B3 for the remaining path risks.

The owner string does not supply a write path. The clear passes it to canonicalize through same_dir. See B1 for the identity risks.

The start-up and clear checks apply to this head. The PR documents a downgrade gap when an older binary cannot see the owned name. Source, call sites, test map, live PR body.

Test-only code and secrets

The encoder at crates/rvc/src/detection_marker.rs:334–338 gates the fault injection with cfg(test). Lines 1263 and 1303 also gate the hook and its setup.

The shared encoder propagates normal JSON serialization errors in production. crates/rvc/src/liveness_loop.rs changes only the test module.

The new fixtures contain synthetic marker data and public key strings. Gitleaks 8.21.2 finds no leaks in the two-commit range or the four changed files. Both scans use the strict CI rules. Source, Gitleaks log.

Hand mutants

Each run changes one source file. The runner restores the full file from HEAD and checks git status --porcelain after each run.

8 mutants fail an assertion. 5 non-equivalent mutants survive. The time for each command includes the wait for the lock.

M01 — ASSERTION_KILL

Remove the check for a different keystore. The command takes 156.635 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • detection_marker::tests::clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore fails at crates/rvc/src/detection_marker.rs:1863:9 after 0.009 seconds.

    assert_eq!( clear_fallback_marker(&fallback, &keys).map_err(|err| err.to_string()), Ok(vec![FallbackClear::OtherOwner { path: owned.clone(), owner: canonical_other }]), "The function must keep the marker of another keystore and give its canonical owner." );

Log, applied diff, JSON result, restore check.

M02 — ASSERTION_KILL

Treat a document that does not parse as a document with no owner. The command takes 190.832 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • detection_marker::tests::clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore fails at crates/rvc/src/detection_marker.rs:1906:9 after 0.010 seconds.

    assert!( matches!( &result, Err((io::ErrorKind::InvalidData, text)) if text.contains(&owned.display().to_string()) ), "The function must return InvalidData with the path for a marker that does not \ parse. The result is {result:?}." );

Log, applied diff, JSON result, restore check.

M03 — SURVIVED

Continue to the keystore marker when the fallback clear returns an error. The command takes 14.816 seconds.

The selected tests pass. The result gives no assertion kill. The regression discards a returned error and can clear the keystore marker.

Log, applied diff, JSON result, restore check.

M04 — ASSERTION_KILL

Skip the owned fallback check at boot. The command takes 50.919 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • bootstrap::run::tests::boot_refuses_an_owned_fallback_marker fails at crates/rvc/src/bootstrap/run.rs:1419:9 after 0.008 seconds.

    assert_eq!( err.exit_code(), EXIT_DOPPELGANGER_DETECTED, "The boot must exit 12 for the owned fallback marker {}. The error is: {err}", owned.display() );
  • liveness_loop::tests::sigterm_during_marker_retry_exits_12_and_boot_is_refused fails at crates/rvc/src/liveness_loop.rs:3764:9 after 0.011 seconds.

    assert_eq!(err.exit_code(), EXIT_DOPPELGANGER_DETECTED);
  • liveness_loop::tests::fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir fails at crates/rvc/src/liveness_loop.rs:3963:9 after 0.011 seconds.

    assert_eq!( err.exit_code(), EXIT_DOPPELGANGER_DETECTED, "The boot of instance A must exit 12. The error is: {err}" );

Log, applied diff, JSON result, restore check.

M05 — SURVIVED

Skip boot refusal only when the owned fallback cannot be read. The command takes 52.492 seconds.

The selected tests pass. The result gives no assertion kill. The regression skips refusal for an unreadable owned marker.

Log, applied diff, JSON result, restore check.

M06 — ASSERTION_KILL

In the drain, use the legacy name when both keystore writes fail. The command takes 136.004 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • liveness_loop::tests::panic_while_latched_exits_12_with_a_fallback_marker fails at crates/rvc/src/liveness_loop.rs:3830:9 after 0.010 seconds.

    assert!(matches!( crate::detection_marker::inspect_fallback(&fallback_in(fallback.path())).owned, crate::detection_marker::MarkerInspect::Present(_) )); }  /// A failed marker write increments this test's view of the counter by at least one. #[tokio::test(start_paused = true)] async fn failed_marker_write_increments_the_failure_counter() { let before = crate::metrics::RVC_DOPPELGANGER_MARKER_WRITE_FAILURES_TOTAL.get(); let missing = missing_marker_dir("metric"); let (_m, task, _executor, _shutdown_rx, token, _clock) = latched_loop_with_failing_marker(&missing).await; let after = crate::metrics::RVC_DOPPELGANGER_MARKER_WRITE_FAILURES_TOTAL.get(); assert!( after.saturating_sub(before) >= 1, "rvc_doppelganger_marker_write_failures_total must grow on a failed write \ (before {before}, after {after})" );
    
  • liveness_loop::tests::token_cancel_during_marker_retry_exits_12 fails at crates/rvc/src/liveness_loop.rs:3790:9 after 0.013 seconds.

    assert!(matches!( crate::detection_marker::inspect_fallback(&fallback_in(fallback.path())).owned, crate::detection_marker::MarkerInspect::Present(_) )); task.abort(); }  /// A panicking registered task while the latch is set must not exit 0 or 16. #[tokio::test] async fn panic_while_latched_exits_12_with_a_fallback_marker() { let fallback = tempfile::tempdir().expect("temp"); let missing = missing_marker_dir("panic"); let m = machine(); let pk = new_pk(); m.register(&pk, 0); let samples = vec![ValidatorLivenessData { index: hex::encode(pk.to_bytes()), is_live: true }]; let _ = m.apply_liveness_tick(&[EpochLiveness { epoch: 0, samples: Some(samples) }], 0, 0); assert!(m.detected_latch()); let token = CancellationToken::new();
    
  • liveness_loop::tests::sigterm_during_marker_retry_exits_12_and_boot_is_refused fails at crates/rvc/src/liveness_loop.rs:3755:9 after 0.013 seconds.

    assert!( matches!( crate::detection_marker::inspect_fallback(&fallback_in(fallback.path())).owned, crate::detection_marker::MarkerInspect::Present(_) ), "the fallback marker must be durable" );
  • liveness_loop::tests::fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir fails at crates/rvc/src/liveness_loop.rs:3955:9 after 0.013 seconds.

    assert_ne!( err.exit_code(), EXIT_DOPPELGANGER_DETECTED, "The boot of instance B must not exit 12 for the detection of instance A. The \ error is: {err}" );

Log, applied diff, JSON result, restore check.

M07 — SURVIVED

Record durability even when the last fallback write fails. The command takes 132.807 seconds.

The selected tests pass. The result gives no assertion kill. The regression treats a failed write as durable.

Log, applied diff, JSON result, restore check.

M08 — ASSERTION_KILL

Remove the file extension from the fallback name. The command takes 107.129 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • bootstrap::run::tests::boot_refuses_an_owned_fallback_marker fails at crates/rvc/src/bootstrap/run.rs:1419:9 after 0.008 seconds.

    assert_eq!( err.exit_code(), EXIT_DOPPELGANGER_DETECTED, "The boot must exit 12 for the owned fallback marker {}. The error is: {err}", owned.display() );
  • detection_marker::tests::fallback_marker_name_appends_the_db_file_name fails at crates/rvc/src/detection_marker.rs:1826:9 after 0.009 seconds.

    assert_eq!( fallback.owned_path(), PathBuf::from("/srv/db/.rvc.doppelganger_detected.a.sqlite"), "The owned path must be the legacy name, a dot, and the database file name." );

Log, applied diff, JSON result, restore check.

M09 — ASSERTION_KILL

Report an unreadable leaf symlink as absent. The command takes 80.834 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • detection_marker::tests::symlink_at_the_marker_path_is_not_absent fails at crates/rvc/src/detection_marker.rs:1719:9 after 0.007 seconds.

    assert!( matches!(inspect(dir.path()), MarkerInspect::Unreadable { .. }), "a dangling symlink must not look absent" );

Log, applied diff, JSON result, restore check.

M10 — SURVIVED

Remove the exclusive create flag from the marker temporary file open. The command takes 129.737 seconds.

The selected tests pass. The result gives no assertion kill. The regression can open an existing temporary entry.

Log, applied diff, JSON result, restore check.

M11 — SURVIVED

Treat a fallback that cannot be read as absent during clear. The command takes 136.782 seconds.

The selected tests pass. The result gives no assertion kill. The regression reports an unreadable owned marker as absent.

Log, applied diff, JSON result, restore check.

M12 — ASSERTION_KILL

Suppress a directory fsync failure after the marker rename. The command takes 137.110 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • detection_marker::tests::dir_fsync_failure_after_rename_is_not_durable fails at crates/rvc/src/detection_marker.rs:2647:9 after 0.008 seconds.

    assert!( result.is_err(), "persist_detection must return Err when the directory fsync fails." );

Log, applied diff, JSON result, restore check.

M13 — ASSERTION_KILL

Accept unknown fields in the marker document. The command takes 136.085 seconds.

The following failures are assertion kills. The test times come from Nextest.

  • detection_marker::tests::marker_with_an_unknown_field_is_corrupt fails at crates/rvc/src/detection_marker.rs:1811:9 after 0.007 seconds.

    assert!( matches!(inspected, MarkerInspect::Corrupt { .. }), "A marker with the unknown field foo must be Corrupt. The result is {inspected:?}." );

Log, applied diff, JSON result, restore check.

The review counts only assertions that fail. It excludes build errors, time limits, and the shutdown guard. The review checks panic lines against the applied diff. This corrects the initial M09 parser result. The M09 assertion moves from line 1718 at the assigned head to line 1719 in the mutant. The review skips cargo mutants --in-diff to limit build time. Mutation plan, log reconciliation, progress log.

Clean runs and tools

All 23 selected rvc tests and 5 selected rvc-bin tests pass after the last restore. Final rvc log, final rvc-bin log.

Before the mutants, the main filters pass 16 rvc tests and 2 rvc-bin tests.
The additional filters pass 7 and 3 tests. First rvc log, first rvc-bin log, additional rvc log, additional rvc-bin log.

The toolchain is rustc 1.99.0 (b940084d7 2026-09-28). The workspace declares Rust 1.92 and edition 2021. CI selects stable Rust.

The Cargo runs use nextest run --all-features --no-fail-fast for the two affected packages. Hand mutants select the relevant unit binary and tests.
Each Cargo run uses the shared lock and the review's private target.
The runner checks disk space and inodes before each call. The first call waits for the shared lock.
This review excludes full Nextest suites, Clippy, and Docker. Main filters, additional filters, locked command, build wait, dependency inventory.

The format check for the two affected packages passes. Format log.

cargo deny check advisories sources passes with the repository policy and a private advisory database. Deny log.

The isolated CLI checks keep an unreadable file, a directory, and a leaf symlink in C4. The keystore marker stays in each case. Boot refuses the unreadable entry with exit 12.
For C8, the 228-byte case reaches the database check. It exits 1 because the fixture has no database. The 229-byte case refuses at the marker check with exit 12. CLI result.

CI at the final pin check

GitHub reports these workflow runs for 58ed07d105e1affae6835e91972e1e6d442f021e.

Run ID Workflow Status Conclusion
44388603193 CI completed success
Check or context Status Conclusion
Check COMPLETED SUCCESS
Test (all features) COMPLETED SUCCESS
Secret Scan COMPLETED SUCCESS
Coverage COMPLETED SUCCESS
Architecture Gates COMPLETED SUCCESS
Spec Vectors COMPLETED SUCCESS
Scripts COMPLETED SUCCESS
Trace continuity (end to end) COMPLETED SUCCESS
GitGuardian Security Checks COMPLETED SUCCESS

CI is green on this head. Exact-head workflow runs, PR rollup.

Model: gpt-6.2-sol, effort high

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 (W4-D2a / #888) — Harper Formal R1

Overall verdict: REQUEST_CHANGES (6 BLOCKING, 1 NIT). The selected tests pass on the pinned head. Generated mutants and the five unviable hand twins fail by assertion after a focused retry. Six defects remain: a marker write can replace a second database, two Unix names can share one marker, a recorded owner can name a different keystore, a valid database name can prevent a durable fallback write, a new test uses a wall-clock sleep, and the submitted CLI tests pass when the command discards a fallback error.

  • Head: 58ed07d105e1affae6835e91972e1e6d442f021e (re-checked at FINAL against gh pr view 1013 and the review worktree; unchanged)
  • Tree: 115fbda8135f18a0f23d892d82e3ee0845090d3a
  • Develop / parent: 920a7afacce843426a54f719a198c5bc886a07e4 — 2 ahead / 0 behind (rev-list --left-right --count = 0 2). Live develop stayed at this SHA during the review.
  • Red tests commit: 60ea92b6277f0167e81daf672a33ef77a3cbb566
  • Branch fix/w4-d2a-888. DRAFT, MERGEABLE, OPEN. Two signed commits sit directly on develop (red, then fix). Ash pinned this two-commit fast-forward shape. It is not a finding. Author = committer = Joonkyo Kim <rootwarp@gmail.com>; no trailers. Both commits have a Good "git" signature. Authored and committed 2026-11-19 04:09:22 and 04:14:30 KST (Asia/Seoul).

Diff versus the assigned parent: 4 files, +554/−52. Files: bin/rvc/src/commands/doppelganger.rs, crates/rvc/src/bootstrap/run.rs, crates/rvc/src/detection_marker.rs, crates/rvc/src/liveness_loop.rs. Production has 243 added and 40 removed lines. Tests have 311 added and 12 removed lines. The production total is below the split threshold of about 1,000 lines. Cargo.lock is unchanged.

This review used one Codex run: gpt-6.2-sol with model_reasoning_effort=high. The wrapper started at 2026-11-19 05:17:46 KST (Asia/Seoul) and exited 0 at 06:39:11 KST.

BLOCKING

B1 — An owned marker can replace a different slashing database

fallback_marker_name builds .rvc.doppelganger_detected.<db file name> in the database directory. atomic_write then renames the temporary file over that path. If database A is other.sqlite, its marker path is also a legal path for database B, .rvc.doppelganger_detected.other.sqlite. An independent probe confirmed that a later marker write by A replaced the SQLite file of B with marker JSON.

Reserve the marker name space before SQLite opens a database. Protect an existing non-marker file at the destination. Add a test that keeps the second database.

Evidence: crates/rvc/src/detection_marker.rs:131 and :358 (rename at :375). Probe record: pr1013-r1/scratch/harper-logs/filesystem-probes.log.

B2 — Two distinct Unix database names can select one owned marker

The name function uses to_string_lossy(). Different invalid UTF-8 bytes can map to the same replacement character. SQLite can open the two paths as distinct files. Boot for database B can then read the marker from database A and return exit code 12. An independent probe opened two distinct paths that both identified .rvc.doppelganger_detected.a�.sqlite.

Preserve the native basename bytes in the marker name. Add a test with two different basenames that produce the same lossy string.

Evidence: detection_marker.rs:131. Probe record: filesystem-probes.log.

B3 — The recorded owner can identify a different keystore

If canonicalization fails, marker_owner_of records the path as given, which can be a relative path such as keys. A clear from a different working directory can resolve that owner to a different keystore and remove the marker. The lossy conversion can also map a native owner path to a distinct valid UTF-8 path. Independent probes confirmed both cases: a clear from a different working directory removed the marker of the relative owner, and a different keystore matched the lossy owner path and removed its marker.

Record an owner that retains its reference directory and distinguishes supported paths. If the owner cannot be represented, the owner check must fail closed. Add tests that keep the marker in both cases.

Evidence: detection_marker.rs:188 and the same_dir comparison at :241. Probe record: filesystem-probes.log.

B4 — A valid database basename can prevent a durable fallback write

The temporary basename includes the full marker basename, the process id, a time suffix, and .tmp. A 210-byte database basename fits on a file system with a 255-byte component limit, and the final marker basename also fits, but the temporary basename does not. The write returns File name too long. The boot reader then returns Absent. An independent CLI boot of that path continued past the marker check and returned exit code 1 for a later beacon error. If both keystore writes fail and the process stops before a write succeeds, the next boot sees no marker for this path.

A 229-byte database basename also makes the final marker basename too long. The boot reader returns Unreadable, and boot returns exit code 12.

Use a shorter temporary basename. Check the maximum supported database basename before the client signs.

Evidence: detection_marker.rs:358 and the drain retry in bootstrap/run.rs:690. Probe record: filesystem-probes.log.

B5 — The new two-instance test uses a wall-clock sleep

The new test fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir calls finish_within. That helper starts a thread that sleeps for two real seconds and panics if the drain is still running. The elapsed time can cause a test failure on a busy runner. The re-pointed in-place test retains the same helper. The 50-run stress set passed, but a pass count does not remove the wall-clock wait.

Use a deterministic drain completion check. Use paused Tokio time if the test needs a time limit.

Evidence: call at liveness_loop.rs:3947. Sleep at :4173–4181.

B6 — The CLI tests pass when the command discards a fallback error

The C4 hand mutant discards the error from clear_fallback_marker at doppelganger.rs:64. All 10 submitted CLI unit tests passed (exit 0). This is a claim-bearing, non-equivalent survivor. The PR requires fail-closed behavior: if the command cannot read an owned fallback file, it must keep the file, return an error, and stop before it clears the keystore marker.

An independent fixture with invalid JSON at the owned path confirms the production path. The mutant returned code 0 and removed the keystore marker. The same fixture on the pinned code returned code 1 and kept the marker. Add a CLI assertion that an unreadable or invalid owned marker causes an error and that the keystore marker remains.

Evidence: pr1013-r1/scratch/harper-logs/hand/C4-cli-ignores-fallback-error.log and C4-cli-ignores-fallback-error.probe. Production fail-closed probes are in filesystem-probes.log.

NIT

N1 — Rewrite the commit messages, the PR body, and the test prose

The two commit messages, the PR body, and some new or changed test prose use words or sentence forms outside ASD-STE100 Issue 9. The meaning is clear. The ownership table must also state that a valid marker with no owner can be cleared. Concrete replacements are in pr1013-r1/scratch/harper-logs/ste-recheck.md.

Tests, mutants, and CI

The unmutated head passes the selected tests under rustc 1.99.1 (58cf8f9c3 2026-10-13): 13 passed, 2,049 skipped, exit 0 (11 rvc tests and 2 rvc-bin tests). The shared 1.99.1 rustup path was missing, so Harper used a private official 1.99.1 toolchain (commit 58cf8f9c35b355199727627d07ae34a4e2846d67) and did not change /home/box/.rustup. Each of 11 new or changed tests passed 50 of 50 stress runs (550 passed, exit 0). Finding B5 still records the wall-clock wait.

A separate checkout at 60ea92b6 ran the same 13 tests: 8 passed, 5 failed, exit 100. All five failures were assertions (separate-database boot, owned-marker boot refusal, CLI with a different keystore, clear owner report, and owned marker path). The four re-pointed drain tests passed with the legacy-name stubs. Harper removed that checkout. The review worktree stayed at the pinned head.

Generated cargo mutants --in-diff on the assigned parent-to-head diff, one package at a time, under rustc 1.99.1:

  • First pass, rvc: 28 tested, 22 caught, 0 missed, 5 unviable, 1 timeout, 22 strict assertion kills, exit 3. A timeout is not a kill.
  • First pass, rvc-bin: 1 tested, 1 caught, 0 missed, 0 unviable, 0 timeout, 1 strict assertion kill, exit 0.
  • First pass total: 29 tested, 23 caught, 0 missed, 5 unviable, 1 timeout, 23 strict assertion kills.
  • Focused retry of fallback_marker_name -> String::new(): 1 tested, 1 caught, 0 timeout. fallback_marker_name_appends_the_db_file_name failed the path assertion at detection_marker.rs:1826. Exit 0.

Across the first pass and the retry, named assertions caught all 24 distinct viable generated mutants. The five unviable mutants each fail with E0277 because their return types lack Default. Those compile failures are justified non-kills. The first-pass timeout is not an assertion kill; the retry is.

Hand mutants, after each restore left a clean worktree:

  • U1–U5 (unviable twins): assertion kills, exit 100, on the second run after a stale signer-proto artifact was cleared. The first U1–U3 attempts returned 101 and are outside the assertion count.
  • C1 (run_with_clock -> Ok(())): assertion kill, exit 100, at integration_test.rs:1125 and :1166.
  • C4 (CLI discards the fallback error): submitted tests 10/10 passed, exit 0. See B6.
  • D1 (permit unknown marker fields): assertion kill, exit 100, at detection_marker.rs:1811.
  • D2 (require keystore_dir): assertion kill, exit 100, at detection_marker.rs:1792.

Plan conflicts C1 through C8: Harper agrees with C1, C3, C4, and C6. Harper agrees with the C5 path setup, and records the wait in B5. Harper agrees in part with C2 and C8; B3 and B4 record the remaining defects. Harper disagrees with C7; B2 and B3 record the lossy database and owner paths. The on-disk format adds only .rvc.doppelganger_detected.<db file name> and the optional keystore_dir field. Version-1 documents still parse. A document with an unknown field is Corrupt.

Same-head CI run 44388603193 is success on 58ed07d105e1affae6835e91972e1e6d442f021e. Harper retrieved the metadata and logs through bash -ic with gh, and rechecked them at FINAL. The Test job reports 7,788 passed, one slow, and ten skipped, in 144.897 s, at 2026-11-19 05:20:20 KST (Asia/Seoul) (job 135099827934). The Check job runs fmt, workspace Clippy with all features, and cargo deny check advisories sources (job 135099827773). A failure of plain cargo deny check on licenses is not a finding. Architecture Gates (job 135099827938), Coverage, Scripts, Secret Scan, Spec Vectors, Trace continuity, and GitGuardian also pass. CI uses rustc 1.100.0. This review does not rerun the full workspace suite or Clippy locally.

Every cargo command used /workspace/rvc/bin/locked-cargo, flock -x /tmp/harper-build.lock, rustc 1.99.1, CARGO_INCREMENTAL=0, PROTOC=/usr/local/bin/protoc, CARGO_BUILD_JOBS=1, and private /tmp/harper-pr1013-r1-target. The review did not use /workspace/rvc/target-shared. The private target is absent after the run. One cargo job ran at a time. Harper waited for the shared lock and left other processes alone. The worktree is clean at the pinned head and tree. No comment, review, push, merge, or source fix was made. Harper verified the pins, the assertion-only kills, the C4 survivor, the path and CLI probes, the STE items, and the CI counts, then wrote this FINAL.

Copies: lead/handoff/pr1013, pr1013-r1/out, /home/box/rvc-review-archive/pr1013. First line <!-- prepared by Harper for Ash -->. Not posted. Did not message Ash.

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 fix1 security recheck

Blake (RSA) completes this review at 2026-11-19 12:06 KST (Asia/Seoul). The scope is W4-D2a, issue #888, Class S.

Verdict for 0145a66a — BLOCK

B1 and B3 have gaps in the code. B7 through B11 are FIXED. The five required hand mutants fail named assertions. The clean-tree tests pass.

Pins and lineage

Ref Commit SHA Tree SHA
Assigned fix1 head 0145a66a342d11e06e6f0956c7412b6a6e6c5426 df83798d2cce99e4831d26b9bed792eab68398eb
Head parent 74f078c852fb6c178827e5408a109f634aed7e64 83382a1dd892fa3e7fde4778469c644bea279773
R1 head 58ed07d105e1affae6835e91972e1e6d442f021e 115fbda8135f18a0f23d892d82e3ee0845090d3a
Develop base 920a7afacce843426a54f719a198c5bc886a07e4 911477cb3bf504b2ac541df3cfe621c4fefa43c0

PR #1013 is open and in draft. The head branch is fix/w4-d2a-888. The base branch is develop.
The local and live pins match at the start, after the hand mutants, and before this report. Git reports a clean worktree at each of those checks.
Initial live pins, Initial status, Final live pins, Final local pins, Final status. Status after the hand mutants, Pins after the hand mutants.

Develop is an ancestor of the head. git rev-list --left-right --count returns 0 9 for 920a7afa...0145a66a and 0 7 for 58ed07d1...0145a66a.
The git range-diff shows 60ea92b6 = 60ea92b6 and 58ed07d1 = 58ed07d1. The stable patch IDs match in both ranges.
The fix1 range changes five files, with 1,303 insertions and 197 deletions. The develop range changes five files, with 1,714 insertions and 106 deletions.
All three pins have the same Cargo.lock blob. Lineage output.

R1 commit Stable patch ID in each range
60ea92b6 1789d6fc28fed204fd27a16d67d9b93db5a25343
58ed07d1 9e35a75198b733232bd62f9e92e95ab8295d627d

B1 — NOT FIXED. Use a stable database identity

The marker name still uses the configured database file name at crates/rvc/src/detection_marker.rs:181–205. Boot, write, and clear use this name at lines 229–234, 265–284, and 337–346. SQLite opens the configured path at crates/slashing/src/db/open.rs:48–65. Two names for one database can still select different markers. Boot can miss an existing detection. The PR body confirms this choice at line 175. Source, live PR body.

The name now keeps the native bytes. fallback_names_keep_the_native_bytes_of_the_db_file_name requires “Databases A and B must have different owned paths” at crates/rvc/src/detection_marker.rs:2375.

The writer resolves the owner at crates/rvc/src/detection_marker.rs:263,298–312. Clear compares the stored string directly at line 379. fallback_owner_of_a_relative_keystore_is_resolved_at_write_time requires the writer's absolute path at line 2433. fallback_owner_that_is_not_utf8_keeps_the_marker requires InvalidData and keeps the evidence at lines 2472–2481. These tests cover the byte and owner changes. Assertion text.

B2 — FIXED. Protect files in the database directory

Boot rejects configured file names in the marker namespace at crates/rvc/src/detection_marker.rs:145–157 and crates/rvc/src/bootstrap/run.rs:119–130. boot_refuses_a_db_name_that_cannot_get_a_fallback_marker checks the reason and that boot creates no database at lines 1508–1514.

The fallback writer permits an absent entry or a valid marker of the same owner at crates/rvc/src/detection_marker.rs:265–284,1947–1951. The reader requires a regular file at crates/doppelganger/src/marker_file.rs:71–93. fallback_write_replaces_only_a_marker_of_the_same_keystore requires AlreadyExists and checks that the entry stays intact at crates/rvc/src/detection_marker.rs:2588–2605.

The writer opens the temporary file with O_EXCL | O_NOFOLLOW at line 521. Cleanup runs only after the open succeeds. fallback_write_keeps_an_entry_that_it_did_not_create checks that an existing temporary file keeps its bytes at line 2653. marker_write_reports_a_failed_rename_and_removes_its_temp_file requires an error and checks that cleanup removes the writer's temporary file at lines 2540–2541. Assertion text.

B3 — NOT FIXED. Resolve the directory without symlinks

MarkerDir::open calls canonicalize on the configured directory at crates/rvc/src/detection_marker.rs:450–456. This call follows symlinks before the descriptor walk. It also maps NotFound to an absent directory. inspect_fallback then reports both markers absent at lines 225–228. The code cannot establish that the opened directory is the intended directory. A directory alias can cause boot to miss detection evidence. The PR body confirms that the configured directory can include a symlink at line 177. Source, live PR body.

After resolution, the walk uses O_NOFOLLOW and the marker operations use the directory descriptor. See crates/rvc/src/detection_marker.rs:463–479,512–553 and the clear sync at line 385. fallback_write_refuses_a_directory_that_changes_after_the_resolve requires an error for a change after resolution at line 2744. fallback_write_and_clear_hold_the_lock_and_one_directory_handle checks descriptor use at lines 2710–2721. fallback_reads_fail_closed_for_each_directory_state checks a symlink at the marker file at lines 2497–2512. These cases leave symlink resolution before the walk uncovered. Assertion text.

B4 — FIXED. Share the fallback lock

The writer and the clear hold the same directory lock from the owner read through the change. See crates/rvc/src/detection_marker.rs:265–284,337–385,486–505. The lock uses LOCK_EX | LOCK_NB. A different owner prevents replacement.

fallback_write_and_clear_hold_the_lock_and_one_directory_handle requires “The lock try in the write must fail” at line 2710. It checks the same result for the clear at line 2713. The other-owner row of fallback_write_replaces_only_a_marker_of_the_same_keystore requires AlreadyExists and unchanged bytes at lines 2565–2569 and 2588–2602. Assertion text.

B5 — FIXED. Keep ownerless and legacy evidence

This judgment uses the lead decision to preserve ownerless and legacy evidence. The clear returns InvalidData with the path and a recovery step at crates/rvc/src/detection_marker.rs:362–402. The CLI returns that error at bin/rvc/src/commands/doppelganger.rs:65, before the keystore marker unlink at line 81.

clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore checks the legacy path and the recovery step at crates/rvc/src/detection_marker.rs:2273–2285. It requires “The function must keep the marker with no owner” at line 2312. clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner checks both rows at bin/rvc/src/commands/doppelganger.rs:156–195. Each row checks the error, the fallback entry, and the keystore marker. Assertion text.

B6 — FIXED. Limit destination and temporary names

Boot checks the destination name and reads the fallback before signer setup at crates/rvc/src/bootstrap/run.rs:119–130,211–231. crates/rvc/src/detection_marker.rs:158–172 rejects destination names above 255 bytes when the configuration uses a fallback. The temporary name at lines 512–522 uses the process ID and the clock. The name fits within 66 bytes, even with 10 digits for the ID and 29 for the duration.

fallback_name_limits_hold_for_the_check_and_the_write requires a 255-byte destination for a 228-byte database name at line 2811. It requires the write to succeed at lines 2812–2816, which also checks that the temporary name fits. It rejects a 229-byte database name at lines 2771 and 2796–2802. boot_refuses_a_db_name_that_cannot_get_a_fallback_marker checks exit code 1 and the reason at crates/rvc/src/bootstrap/run.rs:1494–1514. Assertion text.

B7 — FIXED. Verify the CLI fallback error

clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner has an unreadable owned entry at bin/rvc/src/commands/doppelganger.rs:158. Its checks at lines 177–195 require the entry path in the error and keep both the entry and the keystore marker.

M03 discards the fallback clear error with unwrap_or_default() at bin/rvc/src/commands/doppelganger.rs:65. The test fails at line 177. The assertion is “The command must fail with an error that names” the owned path “and gives the recovery step.” The error value is empty. The nextest summary is 0.012 s. This failure is an assertion kill.
M03 log, M03 diff, M03 result.

B8 — FIXED. Verify the clear for an unreadable owned entry

clear_fallback_marker_keeps_an_unreadable_owned_entry checks the owned path and the read failure text at crates/rvc/src/detection_marker.rs:2344–2349. It checks that the owned entry and the legacy marker stay at lines 2350–2354.

The clear returns InvalidData at crates/rvc/src/detection_marker.rs:343–359. M11 maps an Unreadable owned entry to an absent entry at line 346. The test fails at line 2346. The assertion is “The clear must return InvalidData with the owned path.” The nextest summary is 0.015 s. This failure is an assertion kill.
M11 log, M11 diff, M11 result.

B9 — FIXED. Verify boot refusal for an unreadable owned entry

boot_refuses_an_unreadable_owned_fallback_marker requires exit code 12 at crates/rvc/src/bootstrap/run.rs:1473–1478. The refusal at lines 130 and 742–750 precedes signer setup at lines 211–231.

M05 skips the owned fallback refusal when the entry is Unreadable at crates/rvc/src/bootstrap/run.rs:746. The test fails at line 1473. The assertion is “Boot must return exit code 12 for the unreadable entry.” Boot then reports a missing database. The nextest summary is 0.013 s. This failure is an assertion kill.
M05 log, M05 diff, M05 result.

B10 — FIXED. Verify that the drain waits for a durable write

drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds uses a missing keystore and a marker failure hook at crates/rvc/src/liveness_loop.rs:3980–3991. The hook affects both marker writers and leaves the record writer alone at crates/rvc/src/detection_marker.rs:651–655,1603–1612. The missing directory causes the record write to fail.

The test checks that the drain stays active and the durability flag stays clear at crates/rvc/src/liveness_loop.rs:4007–4011. It drops the guard and requires exit code 12 and the fallback marker at lines 4012–4025.

M07 calls mark_detection_persisted() after a fallback write failure at crates/rvc/src/bootstrap/run.rs:713–718. The test fails at crates/rvc/src/liveness_loop.rs:4007. The assertion is “The drain must stay active while the three writes fail.” The nextest summary is 0.013 s. This failure is an assertion kill.
M07 log, M07 diff, M07 result.

B11 — FIXED. Verify exclusive creation of the temporary marker

fallback_write_keeps_an_entry_that_it_did_not_create supplies an existing regular file at crates/rvc/src/detection_marker.rs:2634–2639. It requires an error and checks both the temporary file and the existing destination at lines 2649–2663.

M10 removes O_EXCL from the temporary open flags at crates/rvc/src/detection_marker.rs:521. The test fails at line 2649. The assertion is “The write must fail when an entry is at the temp path. The result is Ok(()).” The nextest summary is 0.013 s. This failure is an assertion kill.
M10 log, M10 diff, M10 result.

N1 — FIXED. Use direct test prose

fallback_marker_name_appends_the_db_file_name states the directory and file name requirements in separate sentences at crates/rvc/src/detection_marker.rs:2174–2175. The clear test uses a vertical list for its cases at lines 2223–2237. The list uses the lead decision for ownerless and legacy evidence. The same test states the owner result at lines 2261–2262.

The fallback path setup uses “The test must find a fallback path beside the database” at these locations.

  • clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore at crates/rvc/src/detection_marker.rs:2248.
  • clear_detection_keeps_an_owned_fallback_marker_of_another_keystore at bin/rvc/src/commands/doppelganger.rs:213.
  • fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir at crates/rvc/src/liveness_loop.rs:3924.

boot_refuses_an_owned_fallback_marker names the JSON fixture at crates/rvc/src/bootstrap/run.rs:1393–1394. The two-instance test gives the requested exit code for instance B at crates/rvc/src/liveness_loop.rs:3955–3959. These edits use direct requirements or clear descriptions in the present tense. Assertion text.

N2 — NOT FIXED. Describe the drain fixtures

The PR body has “Test cases” at line 55 and “The selected keystore owns the fallback marker” at line 99. Line 111 gives the first two fixture descriptions. The third description says the test injects failures in all three writes. The missing directory causes the record write to fail in drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds at crates/rvc/src/liveness_loop.rs:3980–3991.

Use a separate list item for each fixture group. Use these sentences for the third item. “The test uses a missing keystore directory and a hook that fails both marker writes. The drain waits. It returns exit code 12 after the fallback write succeeds.” Live PR body.

N3 — NOT FIXED. State which tests fail and which tests pass

The PR body gives assertion failures and passes at lines 59–85. It uses short sentences for the repeat runs at line 113. Line 57 says that each red test fails on an assertion. Lines 59–60 also give counts of passes. Use “The red commits keep the previous behavior in temporary implementations. The next two rows give the results for each round.”

Line 131 says the mutant “is caught” and puts two results in one sentence. Use “The hand mutant causes 19 boot error checks to fail. It also causes two rvc-bin integration tests to fail their exit code checks.” Treat these counts as the author's reported results. Live PR body.

Hand-mutant results

The review applies each required hand mutant on the assigned head and then restores the file. Only an assertion kill counts. A timeout, a hang guard, a compile error, or an overflow panic does not count. The worktree is clean after each restore.

Mutant Result Failing assertion Evidence
M03 ASSERTION_KILL bin/rvc/src/commands/doppelganger.rs:177 “The command must fail with an error that names” the owned path “and gives the recovery step.” The nextest summary is 0.012 s. Log, Diff, Result
M11 ASSERTION_KILL crates/rvc/src/detection_marker.rs:2346 “The clear must return InvalidData with the owned path.” The nextest summary is 0.015 s. Log, Diff, Result
M05 ASSERTION_KILL crates/rvc/src/bootstrap/run.rs:1473 “Boot must return exit code 12 for the unreadable entry.” The nextest summary is 0.013 s. Log, Diff, Result
M07 ASSERTION_KILL crates/rvc/src/liveness_loop.rs:4007 “The drain must stay active while the three writes fail.” The nextest summary is 0.013 s. Log, Diff, Result
M10 ASSERTION_KILL crates/rvc/src/detection_marker.rs:2649 “The write must fail when an entry is at the temp path. The result is Ok(()).” The nextest summary is 0.013 s. Log, Diff, Result

Clean runs and tools

The compiler reports rustc 1.99.0 (b940084d7 2026-09-28). The workspace declares Rust 1.92 and edition 2021 at Cargo.toml:14–15. CI selects stable Rust at .github/workflows/ci.yml:41–43.

Every cargo call uses flock -x /tmp/harper-build.lock with the timeout inside the lock. Each call uses the private target, PROTOC=/usr/local/bin/protoc, and CARGO_INCREMENTAL=0. The logs give the full command arguments and the df -BG / and df -i / results. The prechecks meet the disk and inode limits.

Cargo command before the test filter Result Log
cargo nextest run -p rvc -p rvc-bin --all-features 22 tests run: 22 passed, 2054 skipped Targeted tests
cargo nextest run -p rvc --all-features --lib 6 tests run: 6 passed, 1612 skipped Drain and parser fixtures
M03 targeted rvc-bin test ASSERTION_KILL in 0.012 s M03 log
M11 targeted rvc test ASSERTION_KILL in 0.015 s M11 log
M05 targeted rvc test ASSERTION_KILL in 0.013 s M05 log
M07 targeted rvc test ASSERTION_KILL in 0.013 s M07 log
M10 targeted rvc test ASSERTION_KILL in 0.013 s M10 log

The two clean-tree commands exit 0. Each mutant run uses a targeted nextest filter only. This review has no full nextest, Clippy, or Docker runs.
Status after the first run, Status after the fixture run, Status after the hand mutants.

CI at the final pin check

GitHub reports 9 of 9 checks with a success result for commit 0145a66a342d11e06e6f0956c7412b6a6e6c5426.
A second check at 12:05 KST (Asia/Seoul) reports the same result for this head. The exact-head run list and the PR rollup give these results.
GitHub run list, PR check rollup.

Run ID Run title Status Conclusion Head SHA
44442644756 fix(rvc,rvc-bin): give each slashing database its own fallback marker (W4-D2a) completed success 0145a66a342d11e06e6f0956c7412b6a6e6c5426
Check Status Conclusion
Check COMPLETED SUCCESS
Test (all features) COMPLETED SUCCESS
Secret Scan COMPLETED SUCCESS
Coverage COMPLETED SUCCESS
Architecture Gates COMPLETED SUCCESS
Spec Vectors COMPLETED SUCCESS
Scripts COMPLETED SUCCESS
Trace continuity (end to end) COMPLETED SUCCESS
GitGuardian Security Checks COMPLETED SUCCESS

This report has no Note for Ash section.

Model: gpt-6.2-sol, effort high

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 (W4-D2a / #888) — Harper Formal fix1 recheck

Overall verdict: CLEARED. B1 through B6 PASS. N1 FAIL (NIT, follow-up). A NIT-only FAIL still clears the round.

  • Head: 0145a66a342d11e06e6f0956c7412b6a6e6c5426 (re-checked at FINAL against gh pr view 1013 and the review worktree; unchanged)
  • Tree: df83798d2cce99e4831d26b9bed792eab68398eb
  • Develop / parent: 920a7afacce843426a54f719a198c5bc886a07e4 — 9 ahead / 0 behind. Live develop stayed at this SHA during the recheck.
  • Reviewed R1 head (ancestor): 58ed07d105e1affae6835e91972e1e6d442f021e. The R1 commits 60ea92b6 and 58ed07d1 did not change. Seven additive commits sit on the reviewed head: 9829eeb4, 4b032d4c, 98428334, 4f4bccc0, df6cbae1, 74f078c8, 0145a66a. This reviewed-plus-additive fast-forward shape is expected. It is not a finding.
  • Branch fix/w4-d2a-888. DRAFT, MERGEABLE, OPEN. Author = committer = Joonkyo Kim <rootwarp@gmail.com> on the additive commits; each has a Good "git" signature.

Diff versus the reviewed head: 5 files, +1,303/−197.

This recheck used one Codex run: gpt-6.2-sol with model_reasoning_effort=high. The wrapper started at 2026-11-19 11:34:06 KST (Asia/Seoul) and exited 0 at 12:07:47 KST. The recheck covers only R1 findings B1–B6 and N1. It adds no new findings.

Recheck results

B1 — PASS

Boot checks a reserved database name before SQLite opens the file. The writer checks the owned path before a replacement. On the pinned head, boot_refuses_a_db_name_that_cannot_get_a_fallback_marker and fallback_write_replaces_only_a_marker_of_the_same_keystore passed (exit 0). A fixture with a second SQLite database also received AlreadyExists and kept the original bytes (exit 0).

Patch B1-replace-database.diff removes the replacement check. The same write test failed at detection_marker.rs:2573 in the patched file: “The write must return AlreadyExists with the owned path for a slashing database file. The result is Ok(()).” Exit 100. The worktree was restored.

B2 — PASS

fallback_marker_name now appends the native file-name bytes, not a lossy string. fallback_names_keep_the_native_bytes_of_the_db_file_name uses a\xff.sqlite and a\xfe.sqlite and passed (exit 0).

Patch B2-lossy-db-name.diff restores to_string_lossy(). The test failed at detection_marker.rs:2375: “Databases A and B must have different owned paths.” Both paths ended in .rvc.doppelganger_detected.a�.sqlite. Exit 100. The worktree was restored.

B3 — PASS

The writer records an absolute owner. A clear compares that recorded string. fallback_owner_of_a_relative_keystore_is_resolved_at_write_time and fallback_owner_that_is_not_utf8_keeps_the_marker passed (exit 0).

Patch B3-r1-owner.diff restores the R1 owner string and directory comparison. Both tests failed (exit 100). The relative-path case returned Removed instead of OtherOwner. The invalid UTF-8 case also returned Removed instead of InvalidData. The worktree was restored.

B4 — PASS

The temporary name is .rvc.doppelganger_tmp.<pid>.<nanos>.tmp and does not include the database basename. A fixture write and read for a 210-byte basename passed (exit 0). fallback_name_limits_hold_for_the_check_and_the_write accepts 228 bytes and refuses 229 bytes with “at most 255 bytes.” The boot test returns exit code 1 and does not create the database. Those tests passed (exit 0).

Patch B4-long-temp-name.diff restores the R1 temporary name. The 210-byte fixture and the 228-byte write both failed with “File name too long” (OS error 36). Exit 100. Patch B4-allow-long-name.diff lets a 229-byte name pass the check. The limit test and the boot test failed (boot returned exit code 12). Exit 100. The worktree was restored after each patch.

B5 — PASS

fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir and in_place_record_survives_marker_and_fallback_write_failure start paused Tokio time and await join_in_paused_time. The limit is 30 seconds of paused time. The completion signal is the drain join handle. The two-second std::thread::sleep in finish_within is not on these paths. Both tests passed (exit 0).

B6 — PASS

The command returns a fallback-clear error before it removes the keystore marker. clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner covers invalid JSON and an unreadable owned entry (a directory). It requires an error that names the path, keeps the owned bytes or directory, and keeps the keystore marker. The test passed (exit 0).

The original R1 C4 patch applied without changes (? to unwrap_or_default() at the fallback call). The same CLI test failed at doppelganger.rs:177: “The command must fail with an error that names {} and gives the recovery step.” The error text was empty. Exit 100. The worktree was restored.

N1 — FAIL (NIT, follow-up)

The two R1 commit messages still contain the wording identified in R1 (for example “red tests” and “before the signer”). The live PR body still uses “Shared invariant” and “red” for the previous behavior. The body now states that a clear keeps a valid marker with no owner. Some updated test prose is in place. The final assertion pair in clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore still uses “does not parse.” Concrete leftovers are in pr1013-fix1/scratch/harper-logs/N1-ste.md. This NIT does not change the verdict.

Tests and CI

The unmutated head passed the selected tests under rustc 1.99.1 (58cf8f9c3 2026-10-13, commit 58cf8f9c35b355199727627d07ae34a4e2846d67): 28 passed (26 rvc, 2 rvc-bin), 1,712 skipped, exit 0. A second run of open_slashing_db_rechecks_a_fallback_marker_beside_the_slashing_db passed (1 passed, exit 0). The 210-byte and second-database fixtures passed (exit 0). The shared 1.99.1 rustup path was missing, so Harper used a private official 1.99.1 toolchain and did not change /home/box/.rustup.

Hand mutants, each restored to a clean tree:

  • B1 replace-database: assertion kill, exit 100.
  • B2 lossy db name: assertion kill, exit 100.
  • B3 R1 owner: assertion kills, exit 100 (two tests).
  • B4 long temp name: assertion kills, exit 100 (210-byte and 228-byte writes).
  • B4 allow long name: assertion kills, exit 100 (229-byte check and boot).
  • B6 R1 C4: assertion kill, exit 100.

Six runs produced nine assertion failures. No timeout counted as a kill. This recheck did not run cargo mutants --in-diff. The optional red checkout at 4b032d4c was skipped.

Same-head CI run 44442644756 is success on 0145a66a342d11e06e6f0956c7412b6a6e6c5426. Harper retrieved the metadata and logs through bash -ic with gh, and rechecked them at FINAL. The Test job reports 7,802 passed, one slow, and ten skipped, in 171.048 s, at 2026-11-19 11:21:01 KST (Asia/Seoul) (job 135287028653). The Check job runs fmt, workspace Clippy with all features, and cargo deny check advisories sources (job 135287028272). Architecture Gates, Coverage, Scripts, Secret Scan, Spec Vectors, Trace continuity, and GitGuardian also pass. All nine required checks pass. CI uses rustc 1.100.0. This recheck does not rerun the full workspace suite or Clippy locally.

Every cargo command used /workspace/rvc/bin/locked-cargo, flock -x /tmp/harper-build.lock, rustc 1.99.1, CARGO_INCREMENTAL=0, PROTOC=/usr/local/bin/protoc, CARGO_BUILD_JOBS=1, and private /tmp/harper-pr1013-fix1-target. The review did not use /workspace/rvc/target-shared. The private target is absent after the run. One cargo job ran at a time. Harper waited for the shared lock and left other processes alone. The worktree is clean at the pinned head and tree. No comment, review, push, merge, or source fix was made. Harper verified the pins, the assertion-only kills, B5 paused-time waits, N1 leftovers, and the CI counts, then wrote this FINAL.

Copies: lead/handoff/pr1013, pr1013-fix1/out, /home/box/rvc-review-archive/pr1013. First line <!-- prepared by Harper for Ash -->. Not posted. Did not message Ash.

Add the tests for one owned fallback marker for each slashing database.
These tests fail on an assertion:

- The boot of instance B ignores the fallback marker of instance A.
- The boot refuses the owned marker of its database.
- clear-detection keeps the owned marker of another keystore.
- clear_fallback_marker reads the owner of the owned marker.
- The owned name is the legacy name, a dot and the database file name.

The four drain tests now give a FallbackMarkerPath and read the owned
marker. The two tests of the marker document pass on the base.

DetectionShutdown now holds a FallbackMarkerPath. The new functions are
stubs that keep the old behavior, so that the tests compile. The stubs
write, read and remove only the legacy name.
… (W4-D2a)

The drain wrote the fallback detection marker beside the slashing
database as .rvc.doppelganger_detected. Two instances whose databases
share a directory shared that file. The detection of one instance
refused the boot of the other, and clear-detection for one instance
removed the only record of the other.

The drain now writes .rvc.doppelganger_detected.<db file name>. The
document records the canonical keystore directory in the new optional
field keystore_dir. A version-1 document without the field still
parses, and an unknown field stays Corrupt. The keystore marker has no
owner field. Nothing writes the legacy name now.

Boot refuses on the owned name of its database and on the legacy name,
before the signer and again after the keystore lock.

clear-detection with --slashing-db-path removes the owned marker only
when it belongs to --datadir or records no owner. It keeps the marker
of another keystore and prints its path and owner. It stops on an owned
marker that does not parse, because its owner is unknown. It still
removes a legacy marker and prints "(no owner recorded)". The step
order does not change.
Harper B5. Two drain tests used finish_within. That helper starts a
thread that sleeps for two real seconds and then fails the test. A busy
runner can cause a failure that is not about the drain.

The two-instance test and the in-place record test now run with paused
Tokio time. The join handle of the drain task is the completion signal.
The new helper join_in_paused_time gives a limit of 30 seconds of paused
time. Paused time goes forward only while the runtime has no task that
can run, so the limit does not use the wall clock.

The other users of finish_within did not change in this pull request.
Add the tests for the round-1 review findings. The production code
of this commit keeps the round-1 behavior. Run the 17 tests on this
commit. Fourteen tests fail on assertions. Three tests pass.

The fourteen tests that fail check these rules:

- Two database file names that are different only in bytes that are
  not valid UTF-8 get two fallback markers (Harper B2, RSA B1).
- The marker records the keystore path that the writer resolved. A
  clear from a different working directory keeps the marker (Harper
  B3, RSA B1).
- A keystore path that is not valid UTF-8 keeps the marker (Harper
  B3, RSA B1).
- The fallback write does not replace an entry that is not a marker
  of the same keystore (Harper B1, RSA B2, RSA B4).
- The fallback write does not open or remove a temp entry that it did
  not create (RSA B2, RSA B11).
- The write and the clear hold one lock for the database directory.
  They use one directory handle (RSA B3, RSA B4).
- The write refuses a directory that changes after the resolve (RSA
  B3).
- A database file name of 228 bytes gets a fallback marker. The name
  check refuses a name of 229 bytes and a name in the reserved name
  space. Boot refuses these names before SQLite opens the database
  (Harper B1, Harper B4, RSA B2, RSA B6).
- The clear keeps a marker that records no owner and a legacy marker.
  It returns an error with the path and the recovery step (RSA B5).
  This is a spec change that Ash approved.

The three tests that pass check the fail-closed paths that round 1
already has (Harper B6, RSA B7 to B10):

- Boot returns exit code 12 for an unreadable owned marker.
- The clear keeps an unreadable owned marker and the legacy marker.
- The drain stays active while three writes fail. It returns exit
  code 12 after one write succeeds.

The CLI test for the legacy fallback marker has a new name,
clear_keeps_a_legacy_fallback_marker_beside_the_slashing_db, because
the command now keeps that marker.

This commit adds test-only hooks: arm_after_marker_dir_resolve,
arm_before_marker_temp_open and arm_before_fallback_unlink. It also adds
FALLBACK_LOCK_FILENAME and a check_slashing_db_name that accepts every
name, so that the tests compile.
… (W4-D2a fix1)

Round 1 gave each slashing database its own fallback marker. The
review found that the marker could replace a second database, that
two names could share one marker, and that an owner could name a
different keystore. This commit closes the round-1 findings of Harper
(B1 to B4) and RSA (B1 to B6, B11).

Identity (Harper B2, Harper B3, RSA B1):

- fallback_marker_name returns an OsString with the native bytes of
  the database file name. Two different names give two markers.
- The marker records the absolute keystore path that the writer
  resolves at write time. canonicalize resolves the part of the path
  that exists, and the missing names follow it. A path that is not
  valid UTF-8 is not recorded, so the marker records no owner.
- The clear compares the recorded owner with the path that a write
  for --datadir records now. It does not resolve the recorded owner
  again.

Name space and replacement (Harper B1, RSA B2, RSA B11):

- Boot refuses a database file name that starts with
  .rvc.doppelganger or ..rvc.doppelganger, before SQLite opens it
  (exit code 1).
- The fallback write does not replace an entry at the owned path
  unless it is a marker with the same owner. Else it returns
  AlreadyExists with the path, and the entry stays.
- The temp file is created with O_EXCL. A failed step removes only a
  temp file that the writer created and did not rename.

Directory handle (RSA B3):

- MarkerDir::open resolves the directory one time with canonicalize.
  A walk from / then opens each name with O_NOFOLLOW and O_DIRECTORY.
  A symlink that replaces a name after the resolve stops the walk.
- The reads, the temp create, the rename, the unlink, the lock and
  the directory fsync use that one handle. The keystore marker and
  the force-window file use the same writer.

Lock (RSA B4):

- The fallback write and the clear hold a flock on
  .rvc.doppelganger_lock in the database directory. The lock does not
  wait. The owner read, the replace check and the unlink occur under
  the lock.

Ownerless and legacy evidence (RSA B5, a spec change that Ash
approved):

- The clear keeps an owned marker that records no owner and a legacy
  .rvc.doppelganger_detected. It returns an error with the path and
  the recovery step, and the keystore marker stays. Issue #888 said
  that the clear removes these files.

File name limits (Harper B4, RSA B6):

- The temp name is .rvc.doppelganger_tmp.<pid>.<nanos>.tmp, at most
  57 bytes. A database file name of 228 bytes gets a marker.
- Boot refuses a database file name whose marker name has more than
  255 bytes, before the client can sign (exit code 1).

Other changes:

- doppelganger::marker_file::read_open_marker reads a marker through
  an open descriptor. read_marker_file and the directory handle use
  the same read rule.
- The drain logs the error of a failed fallback write.
- On a platform that is not Unix, the marker writes and the clear
  return Unsupported, and boot reads the fallback paths by path as in
  round 1. CI builds Linux only.

The fourteen tests that failed in the previous commit pass.
…-D2a fix1)

Harper N1 and RSA N1. Change the round-1 test prose that the two
reviews name, so that it follows ASD-STE100 Issue 9:

- detection_marker.rs: the two document tests, the name test, the
  clear test, and the doc of arm_marker_write_failures.
- bootstrap/run.rs: boot_refuses_an_owned_fallback_marker.
- liveness_loop.rs: the two-instance test and fallback_in.
- commands/doppelganger.rs: the fallback path message in
  clear_detection_keeps_an_owned_fallback_marker_of_another_keystore.

Merge three pairs of fix1 tests. Each assertion stays. The merge keeps
the size of fix1 near the limit of 1,500 lines.

- fallback_write_holds_the_lock_and_one_directory_handle and
  fallback_clear_holds_the_lock_and_one_directory_handle are now
  fallback_write_and_clear_hold_the_lock_and_one_directory_handle.
- fallback_write_accepts_a_db_file_name_of_228_bytes and
  slashing_db_name_check_refuses_reserved_and_too_long_names are now
  fallback_name_limits_hold_for_the_check_and_the_write.
- clear_keeps_a_legacy_fallback_marker_beside_the_slashing_db and
  clear_detection_stops_on_an_owned_fallback_marker_with_an_unknown_owner
  are now clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner.
  Its first row is the legacy marker.

Make three doc comments in detection_marker.rs shorter: the module
doc, check_slashing_db_name and resolved_owner. The code does not
change.
The test clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner
stops at the first row that fails. Put the two owned rows of Harper B6
and RSA B7 first: an owned file that does not parse, and an owned
directory. Then the mutants Harper C4 and RSA M03 fail on the assertion
of an owned row, not on the row of the legacy marker. The rows and the
assertions do not change.
…D2a fix1)

The workspace test run found two problems in detection_marker.rs:

- spawn_span_continuity reads each const item of a production file,
  and it refuses two items with one name. The two cfg forms of
  DIR_WALK had one name. The walk flag is now a cfg let statement in
  MarkerDir::open.
- no_rvc_prefix refuses a string literal that starts with "rvc." and a
  lowercase letter, in test code under src too. A row of the name-check
  test used such a literal. The row now makes its name from
  MARKER_FILENAME without the first dot.

The behavior does not change.
…er writer (W4-D2a fix1)

A list of the in-diff mutants showed four kinds of mutant that no test
could kill. Each one changes the behavior:

- The not-found guard of MarkerDir::open becomes true: a directory
  path that cannot resolve reads as Absent.
- The same guard becomes false: a missing database directory reads as
  Unreadable, and boot refuses.
- The not-found guard of MarkerDir::read becomes true: a symlink at the
  owned path reads as Absent.
- cvt returns Ok: a failed rename does not stop the marker write.

Add two tests:

- fallback_reads_fail_closed_for_each_directory_state reads three
  states: a missing directory, a directory path under a file, and a
  symlink at the owned path.
- marker_write_reports_a_failed_rename_and_removes_its_temp_file puts a
  directory at the keystore marker path. The write must fail, and its
  temp file must not stay.

The drain log of a failed fallback write is again the round-1 text,
and some doc comments are shorter, so that fix1 stays at the limit of
1,500 lines. The behavior does not change.
…fix2)

RSA B1: boot, the drain and clear-detection must use one stable identity
of the slashing database file. A second name of one database (a symlink,
a hard link, or a path with `..`) must select the same fallback marker,
or boot and the clear must refuse it.

RSA B3: the walk to the database directory must not follow a symlink at
any step. A missing directory must not read as absent.

These tests are new or changed:

- fallback_paths_with_a_symlink_or_a_parent_name_are_refused (new)
- second_names_of_a_database_are_refused_by_the_check_and_the_clear (new)
- boot_refuses_a_second_name_of_a_database_with_a_fallback_marker (new)
- fallback_reads_fail_closed_for_each_directory_state (the row for a
  missing directory now requires Unreadable and a failed clear)
- fallback_name_limits_hold_for_the_check_and_the_write (a new row for a
  file name with a NUL byte)

This commit does not change the production code. Each of the five tests
fails on a named assertion with the fix1 code.
…r (W4-D2a fix2)

RSA B1 (stable database identity). Boot, the drain and clear-detection
find the fallback marker by the name of the database file in its
directory. That name must be the only name of the file. When the
database directory gets a fallback marker, check_slashing_db_name now
refuses these paths before SQLite opens the database (exit code 1):

- a database file that is a symlink or is not a regular file
- a database file with more than one link
- a database directory that the walk below cannot open

clear_fallback_marker does the same check of the database file. Because
of this, two names of one database select one fallback marker, or boot
and the clear refuse the second name. The marker name does not change.

RSA B3 (no symlink resolution before the walk). MarkerDir::open does not
call canonicalize. The walk starts at the root for an absolute path and
at the working directory for a relative path. It opens each name with
O_NOFOLLOW | O_DIRECTORY. A symlink, a missing name, a file or a `..` in
the path stops the walk with an error. Then inspect_fallback gives
Unreadable for the two markers, the write fails and the drain tries
again, and the clear stops. A missing directory is not absent.

The keystore marker and the force-window file open the keystore
directory by its path, as boot does for its reads of that directory.

The test fallback_write_refuses_a_directory_that_changes_after_the_resolve
and its hook go away, because no resolve step occurs before the walk.
The row "a symlink at the last component" of
fallback_paths_with_a_symlink_or_a_parent_name_are_refused checks that a
symlink at the directory stops the read, the write and the clear.

Harper N1 (PR body wording). The PR body replaces "Shared invariant"
with these sentences. Boot checks four locations for evidence of
detection. These are the detection record, the keystore marker, the
owned fallback, and the legacy fallback. For a test result on a test
commit, the PR body uses "with the previous behavior" in place of "red".
The earlier commit messages do not change.

RSA N2 and N3 (PR body wording). The PR body uses the sentences that the
review gives.

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 fix2 security recheck

Blake (RSA) completes this review at 2026-11-19 17:56:50 KST (Asia/Seoul). The scope is W4-D2a, issue #888, Class S.

Verdict for 39cc4149 — BLOCK

Boot can miss an existing fallback marker when the same database has a different path spelling. Fallback selection also resolves symlinks before the walk.

Pins and lineage

PR #1013 is open and in draft. The head branch is fix/w4-d2a-888. The base branch is develop.

Ref Commit SHA Tree SHA
Assigned head 39cc4149ffc3e8de7f9a1032bc3fae5eb20df2b1 6a57ed511ccd1468fba8247a2f0020d9e96f3fcf
Head parent 1d86c6083ff596609d3eb535b0f20d6f4efdc867 400b3bdf478e3758580e3bf8ea0d8949758201fe
Rebased fix1 head 20da6d144dc07dd0ee2df9fbf5437d08d9577683 49c69ff1c56aea158366b08ed9cef7c610b850b3
Old fix1 head 0145a66a342d11e06e6f0956c7412b6a6e6c5426 df83798d2cce99e4831d26b9bed792eab68398eb
Assigned develop base 49d4cacdafe597490dd736da466eda660722c57a 38b362e7abf21b35c109f58da36097e625f76284
Fix2 before the rebase 23d23701df5b9d1d2b95c2274645327390b17281 4bbbc7ea6970cb2cbbeacb8853e82bddc311df87

The local and live pins match at the start and before this report. Git reports a clean worktree at both checks. Initial live pins, Final live pins, Final local pins, Final worktree status.

The git rev-list --left-right --count result is 0 11 for 49d4cacd...39cc4149. Develop is an ancestor. That range changes five files with 1,859 insertions and 106 deletions. The result for 20da6d14...39cc4149 is 0 2. That range changes these files with 262 insertions and 117 deletions.

  • bin/rvc/src/commands/doppelganger.rs
  • crates/rvc/src/bootstrap/run.rs
  • crates/rvc/src/detection_marker.rs

The git range-diff output shows = for all 11 commits. The output matches both copies of the range-diff. The trees of 39cc4149 and 23d23701 differ only in docs/releases/UNRELEASED.md. The Cargo.lock diff against develop is empty. Lineage output, Range-diff, Fix2 diff.

B1 — NOT FIXED. Use a stable database identity

At crates/rvc/src/detection_marker.rs:115–121, fallback_marker_dir returns None for a file name with no directory component. A path with an explicit . parent reaches the directory comparison. The two spellings can identify the same file in the working directory. When the keystore is elsewhere, the explicit parent permits a fallback and the empty parent skips it.

Boot skips both the database entry check and fallback lookup when this helper returns None. See crates/rvc/src/detection_marker.rs:164–180,209–211 and crates/rvc/src/bootstrap/run.rs:733–752. The drain and clear use the same selector at crates/rvc/src/bootstrap/run.rs:459–465 and bin/rvc/src/commands/doppelganger.rs:64–66. Config retains the supplied path at crates/rvc/src/config/types.rs:705–709,1788–1790. The database builder accepts an empty parent and passes that spelling to SQLite. See crates/rvc/src/config/builder.rs:740–750,777 and crates/slashing/src/db/open.rs:48–65. Thus boot can miss the only durable detection evidence. Source, Selector comparison.

The passing test fallback_dir_is_the_slashing_parent_only_when_it_differs asserts assert_eq!(fallback_marker_dir(&keys, Path::new("slashing.sqlite")), None); at crates/rvc/src/detection_marker.rs:2151. It confirms the branch that skips fallback selection. Selector test log.

For paths that select a fallback, boot and clear refuse symlinks, non-regular files, and files with multiple hard links. A missing database file can pass in an existing directory. The drain uses the configured file name inside the directory it opens. See crates/rvc/src/detection_marker.rs:273–291,345–350,489–511.

  • boot_refuses_a_second_name_of_a_database_with_a_fallback_marker passes. It requires exit 1 and the alias error text for the symlink and hard-link rows. Its . row requires exit 12. See crates/rvc/src/bootstrap/run.rs:1552–1567.
  • second_names_of_a_database_are_refused_by_the_check_and_the_clear passes. It requires the text “one link,” a clear error, and unchanged marker bytes. See crates/rvc/src/detection_marker.rs:2895–2911.
  • boot_refuses_an_owned_fallback_marker passes with a missing database file and an existing marker. See crates/rvc/src/bootstrap/run.rs:1396–1434.

Source, Targeted test log.

B3 — NOT FIXED. Resolve the directory without symlinks

MarkerDir::open walks from / or the working directory with O_NOFOLLOW | O_DIRECTORY. A missing component returns an error. The fallback read maps that error to two Unreadable results. Boot refuses the path at the name check or returns exit 12 at the marker check. See crates/rvc/src/detection_marker.rs:231–250,459–483 and crates/rvc/src/bootstrap/run.rs:119–130,742–750,824–831.

After the walk, the marker read and temporary file open use the descriptor. The rename, unlink, and sync also use it. See crates/rvc/src/detection_marker.rs:392–393,516–521,563–595.

However, fallback selection still calls fs::canonicalize on the database parent before the walk. See crates/rvc/src/detection_marker.rs:115–128,164–180,209–211. If the two resolved paths match, the selector returns None and the walk never runs. That decision follows symlinks and has no directory descriptor. A change to the directory path can invalidate the decision to skip the marker read. The live PR body describes this comparison at line 235. Source, Live PR body.

  • fallback_paths_with_a_symlink_or_a_parent_name_are_refused passes. It requires both reads to give Unreadable, the write and clear to fail, and the marker bytes to stay. It constructs the fallback directly at crates/rvc/src/detection_marker.rs:2574. Its assertions at lines 2581–2596 exercise the walk after selection.
  • fallback_reads_fail_closed_for_each_directory_state passes. Its assertion at crates/rvc/src/detection_marker.rs:2537–2541 requires an Unreadable result and a clear error for a missing directory.
  • The missing-directory row of boot_refuses_a_second_name_of_a_database_with_a_fallback_marker requires exit 1. See crates/rvc/src/bootstrap/run.rs:1558–1567.
  • open_slashing_db_rechecks_a_fallback_marker_beside_the_slashing_db passes. It requires DoppelgangerDetected after the database opens. See crates/rvc/src/bootstrap/slashing.rs:217–229.

Source, Targeted test log, Recheck test log.

N2 — FIXED. Describe the drain fixtures

The live body and Glen’s final copy match byte for byte. The live body uses three list items at lines 137–139. It names the missing keystore directory and the hook for both marker writes in the third item. It gives two short sentences for the drain result. “The drain waits. It returns exit code 12 after the fallback write succeeds.”

In drain_retries_while_three_writes_fail_and_exits_12_after_one_succeeds, the missing directory causes the record write to fail. The hook fails both marker writes. The test states “The drain must stay active while the three writes fail.” It also checks the durable flag, exit code 12, and the fallback marker. See crates/rvc/src/liveness_loop.rs:3979–4025. The separate list items and short sentences meet the wording request. Live PR body, Body comparison, Fixture source.

N3 — NOT FIXED. State which tests fail and which tests pass

The live body and Glen’s final copy match byte for byte. The introduction at line 73 says “The next two rows give the results for each round.” Lines 75–78 give three rounds as two list items and a paragraph. The table at line 103 gives the reported assertion failure for boot_refuses_a_second_name_of_a_database_with_a_fallback_marker. The test checks the exit code and error text at crates/rvc/src/bootstrap/run.rs:1564–1567.

Use “The next three entries give the test results with the previous behavior.” This sentence names the entries and states which behavior the tests cover.

The C1 text at line 173 now uses separate sentences for boot and CLI results. The counts are the author’s reported results. Live introduction, Live C1 wording, Body comparison, Assertion source.

Earlier findings

Fix2 does not reopen B2 or B4–B11 because the earlier commits stay = and the prior mutant-kill tests retain their required assertions. Range-diff, Prior test assertions.

Local tests and tools

I ran two targeted cargo nextest run --locked -p rvc -E commands on the clean head. Each log gives its exact filter and pre-run disk and inode counts.

Run Result Evidence
Boot, alias, directory, marker name, and directory handle tests 7 tests run: 7 passed, 1902 skipped. Exit 0. Test log, Worktree status
Fallback selection and boot recheck tests 2 tests run: 2 passed, 1907 skipped. Exit 0. Test log, Worktree status

The selector test and the source establish the remaining gaps. I applied no hand mutants. I did not run full Nextest, Clippy, or Docker.

rustc +1.99.0 --version reports rustc 1.99.0 (b940084d7 2026-09-28). Both runs use the shared flock, a private target directory, PROTOC=/usr/local/bin/protoc, and CARGO_INCREMENTAL=0. The first check reports 51G free and 396,768 free inodes. The second reports 43G free and 391,485 free inodes. Toolchain output, Cargo helper, Command record.

CI at the final pin check

The check at 2026-11-19 17:56:50 KST uses headSha 39cc4149ffc3e8de7f9a1032bc3fae5eb20df2b1. CI is green on this head.

Workflow Run ID Status Conclusion
CI 44488522661 completed success

The PR rollup reports 9 SUCCESS.

Run list, Check rollup.

Model: gpt-6.2-sol, effort high

@rootwarp rootwarp left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #1013 (W4-D2a / #888) — Harper Formal fix2 recheck

Overall verdict: CLEARED. B1 through B6 PASS. N1 FAIL (NIT, follow-up). A NIT-only FAIL still clears the round.

  • Head: 39cc4149ffc3e8de7f9a1032bc3fae5eb20df2b1 (re-checked at FINAL against gh pr view 1013 and the review worktree; unchanged)
  • Tree: 6a57ed511ccd1468fba8247a2f0020d9e96f3fcf
  • Develop / parent: 49d4cacdafe597490dd736da466eda660722c57a — 11 ahead / 0 behind. Live develop stayed at this SHA during the recheck. The branch is a fast-forward on develop.
  • Rebased fix1 tip (ancestor): 20da6d144dc07dd0ee2df9fbf5437d08d9577683. The branch was rebased onto develop after the merge of #1014. The assigned range-diff of all 11 commits is "=". This rebased fast-forward shape is expected. It is not a finding.
  • Branch fix/w4-d2a-888. DRAFT, MERGEABLE, OPEN.

Diff versus the rebased fix1 tip: 3 files, +262/−117 (crates/rvc/src/detection_marker.rs, crates/rvc/src/bootstrap/run.rs, bin/rvc/src/commands/doppelganger.rs). The change covers database identity and the directory walk.

This recheck used one Codex run: gpt-6.2-sol with model_reasoning_effort=high. The wrapper started at 2026-11-19 17:37:33 KST (Asia/Seoul) and exited 0 at 18:33:56 KST. The recheck covers only R1 findings B1–B6 and N1. It adds no new findings.

Recheck results

B1 — PASS

On the pinned head, fallback_write_replaces_only_a_marker_of_the_same_keystore requires AlreadyExists and keeps the file bytes. The reserved-name row in boot_refuses_a_db_name_that_cannot_get_a_fallback_marker requires exit code 1, the reason “name space,” and no file at the database path. Those tests passed (exit 0). A fixture with a second SQLite database also received AlreadyExists and kept the original bytes (exit 0).

The original B1-replace-database.diff did not apply, because fix2 changes the directory open next to the guard. The adapted patch removes the same replacement guard. The write test failed at detection_marker.rs:2649: “The write must return AlreadyExists with the owned path for a slashing database file. The result is Ok(()).” Exit 100. The worktree was restored.

B2 — PASS

fallback_names_keep_the_native_bytes_of_the_db_file_name uses a\xff.sqlite and a\xfe.sqlite and passed (exit 0). In detection_marker.rs:2406, the test requires different owned paths.

The original B2-lossy-db-name.diff applied. The same test failed at detection_marker.rs:2406: “Databases A and B must have different owned paths.” Both paths ended in .rvc.doppelganger_detected.a�.sqlite. Exit 100. The worktree was restored.

B3 — PASS

fallback_owner_of_a_relative_keystore_is_resolved_at_write_time and fallback_owner_that_is_not_utf8_keeps_the_marker passed (exit 0). The tests require OtherOwner or InvalidData and keep the marker.

The original B3-r1-owner.diff applied. Both tests failed (exit 100). The relative-path case at detection_marker.rs:2456 returned Removed instead of OtherOwner. The invalid UTF-8 case at :2495 returned Removed instead of InvalidData. The worktree was restored.

B4 — PASS

The temporary name stays short and does not include the database basename. fallback_name_limits_hold_for_the_check_and_the_write accepts 228 bytes and refuses 229 bytes with “at most 255 bytes.” The boot test returns exit code 1 and does not create the database. Those tests passed (exit 0). A fixture write and read for a 210-byte basename also passed (exit 0).

The original B4-long-temp-name.diff applied. The 210-byte write at detection_marker.rs:2875 and the 228-byte write at :2856 both failed with “File name too long (os error 36).” Exit 100. The original B4-allow-long-name.diff did not apply, because fix2 moves the fallback path check before the limit check. The adapted patch increases the same limit by one byte. The limit test failed at :2839 (“The result is Ok(())”). The boot test failed at bootstrap/run.rs:1503 with exit code 12 instead of 1. Exit 100. The worktree was restored after each patch.

B5 — PASS

fallback_marker_of_one_instance_does_not_refuse_another_sharing_the_db_dir and in_place_record_survives_marker_and_fallback_write_failure start paused Tokio time and await join_in_paused_time. The limit is 30 seconds of paused time. The completion signal is the drain join handle. The two-second std::thread::sleep in finish_within is not on these paths. Both tests passed (exit 0).

B6 — PASS

clear_detection_stops_on_a_fallback_marker_with_an_unknown_owner passed (exit 0). In doppelganger.rs:178, the error must name the entry and give the recovery step. The owned bytes or directory stay. The keystore marker stays.

The original R1 C4 patch applied without changes (? to unwrap_or_default() at the fallback call). The same CLI test failed at doppelganger.rs:178: “The command must fail with an error that names {} and gives the recovery step.” The error text was empty. Exit 100. The worktree was restored.

N1 — FAIL (NIT, follow-up)

The live PR body and the two fix2 commit messages (1d86c608 and 39cc4149) state the four boot checks in direct sentences. The body uses “With the previous behavior” for the earlier test result. The fix2 message says that the earlier commit messages stay. This recheck does not fail N1 only because those earlier messages stay.

The last two assertions of clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore still use “does not parse,” at detection_marker.rs:2350 and :2356. The fix1 recheck already recorded these messages. Use “The function must return InvalidData with the path of the marker with invalid JSON.” Use “The function must keep the marker with invalid JSON.” This NIT does not change the verdict.

Tests and CI

The unmutated head passed the selected tests under rustc 1.99.1 (58cf8f9c3 2026-10-13, commit 58cf8f9c35b355199727627d07ae34a4e2846d67): 14 passed (12 rvc, 2 rvc-bin), 1,728 skipped, exit 0. The first filter command had a syntax error and exited with code 94. The corrected command ran all 14 selected tests. Two private fixtures also passed (exit 0): a second slashing database keeps its bytes, and a 210-byte database name gets a marker. The shared 1.99.1 rustup path was missing, so Harper used a private official 1.99.1 toolchain and did not change /home/box/.rustup.

Diff-scoped cargo mutants --in-diff on 20da6d14..39cc4149, one package at a time:

  • rvc unit tests: 16 tested, 13 caught, 0 missed, 3 unviable, 0 timeout.
  • rvc boot integration test: 1 tested, 1 caught.
  • rvc-bin: 0 mutants (the package diff is a comment change).
  • Combined: 17 tested, 14 caught, 0 missed, 3 unviable, 0 timeout. Each of the 14 caught mutants failed on one or more assertions and exited 100. The three unviable builds failed with E0277 because FallbackInspect, FallbackClear, and MarkerDir have no Default implementation. No timeout counted as a kill.

Hand mutants, each restored to a clean tree. The first six attempts stopped at a stale signer-proto build path and exited 101. Those builds are not assertion kills. After cargo clean -p rvc-signer-proto --locked in the private target, the six patches ran again:

  • B1 replace-database (adapted): assertion kill, exit 100.
  • B2 lossy db name: assertion kill, exit 100.
  • B3 R1 owner: assertion kills, exit 100 (two tests).
  • B4 long temp name: assertion kills, exit 100 (210-byte and 228-byte writes).
  • B4 allow long name (adapted): assertion kills, exit 100 (229-byte check and boot).
  • B6 R1 C4: assertion kill, exit 100.

Six runs produced nine assertion failures. No timeout counted as a kill.

Same-head CI run 44488522661 is success on 39cc4149ffc3e8de7f9a1032bc3fae5eb20df2b1. Harper retrieved the metadata and logs through bash -ic with gh, and rechecked them at FINAL. The Test job reports 7,804 passed, one slow, and ten skipped, in 170.976 s, at 2026-11-19 17:41:16 KST (Asia/Seoul) (job 135437954337). The Check job runs fmt, workspace Clippy with all features, and cargo deny check advisories sources (job 135437954073). Architecture Gates, Coverage, Scripts, Secret Scan, Spec Vectors, Trace continuity, and GitGuardian also pass. All nine required checks pass. CI uses rustc 1.100.0. This recheck does not rerun the full workspace suite or Clippy locally.

Every cargo command used /workspace/rvc/bin/locked-cargo, flock -x /tmp/harper-build.lock, rustc 1.99.1, CARGO_INCREMENTAL=0, PROTOC=/usr/local/bin/protoc, CARGO_BUILD_JOBS=1, and private /tmp/harper-pr1013-fix2-target. The review did not use /workspace/rvc/target-shared. The private target is absent after the run. One cargo job ran at a time. Harper waited for the shared lock and left other processes alone. The worktree is clean at the pinned head and tree. No comment, review, push, merge, or source fix was made. Harper verified the pins, the assertion-only kills, B5 paused-time waits, N1 leftovers, and the CI counts, then wrote this FINAL.

Copies: lead/handoff/pr1013, pr1013-fix2/out, /home/box/rvc-review-archive/pr1013. First line <!-- prepared by Harper for Ash -->. Not posted. Did not message Ash.

…a fix3)

RSA B1 and B3 show that the fallback selection skips the fallback check
for two kinds of path:

- A bare file name such as slashing.sqlite. The selection gives no
  directory, but SQLite opens the file in the working directory.
- A database directory that is a symlink to the keystore directory. The
  selection resolves the two paths, finds one directory, and does not
  run the walk.

These tests fail on an assertion with the fix2 selector:

- fallback_dir_is_the_slashing_parent_only_when_it_differs (changed).
  The names slashing.sqlite and ./slashing.sqlite must select the
  working directory and the same marker. A relative path of the
  keystore directory must select no directory.
- fallback_selection_does_not_skip_a_directory_behind_a_symlink (new).
  For a database directory that is a symlink to the keystore directory,
  and for a keystore path that is a symlink to the database directory,
  the selection must give the database directory. The read must not
  give Absent, and the clear must fail.
- fallback_owner_keeps_a_symlink_in_the_keystore_path (new). The writer
  must record the keystore path with no symlink resolution.
- boot_refuses_an_owned_fallback_marker_of_a_bare_db_name (new). Boot
  must return exit code 12 for slashing.sqlite, as it does for
  ./slashing.sqlite.
- boot_refuses_a_second_name_of_a_database_with_a_fallback_marker
  (changed). A new row boots through a symlink to the keystore
  directory. Boot must return exit code 1 and the text "no symlink".

The test-only seam arm_walk_start sets the start directory of each walk
of a relative path on the test thread. Thus the tests do not change the
working directory of the process.

Harper N1 is also in this commit. The last two assertion messages of
clear_fallback_marker_removes_only_an_owned_marker_of_this_keystore now
use the words "invalid JSON". Its first assertion now expects the
absolute path of the other keystore. This test passes with the fix2
code.
…lk (W4-D2a fix3)

RSA B1 and B3 are not fixed in fix2. The fallback selection used a
different rule from the read, the write and the clear:

- A bare file name such as slashing.sqlite gave no fallback directory,
  so boot did not read the marker beside the database.
- The selection resolved the two paths with canonicalize. When the
  resolved paths were equal, the walk did not run, so a symlink could
  make boot skip the marker check.

Now one directory walk does the selection, the read, the write and
the clear:

- fallback_marker_dir uses the parent of the database, or "." for a
  bare file name. Thus slashing.sqlite and ./slashing.sqlite select the
  same directory and the same marker.
- same_dir opens the database directory and the keystore directory
  with the walk of MarkerDir::open (no path resolution, no symlink, no
  ".."). Then it compares (st_dev, st_ino) from fstat of the two
  handles. It does not compare paths.
- When a walk or an fstat fails, the selection cannot show that the
  two are one directory. It then gives the database directory, so boot
  reads the fallback and fails closed. It never skips the fallback
  check.
- resolved_owner does not call canonicalize. The owner is the keystore
  path made absolute with the working directory, with no "." component
  and no trailing slash. A second path of one keystore, such as a
  symlink, records a different owner, and a clear through that path
  keeps the marker.

No code in the selection, the read, the write or the clear calls
canonicalize now. The marker names and the marker document do not
change.

This change adds a limit. Boot refuses (exit code 1) a database
directory with a symlink in its path also when that directory is the
keystore directory, because the walk cannot show that the two are one
directory. The operator must use the real path.

The test-only seam arm_walk_start now has separate cfg(test) and
cfg(unix) attributes, so cargo-mutants does not mutate it. Its behavior
does not change.
@rootwarp rootwarp mentioned this pull request Nov 19, 2026
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant