Repository navigation
test(rvc): add builder-flag eviction tests through the store writers (W4-E6b3) - #1012
Conversation
…(W4-E6b3) Add the orchestrator_at_slot harness to epoch_boundary_restart.rs. It starts DutyOrchestrator::run and the epoch boundary runner with in-process doubles and paused time. A counting RegistrationSigner wraps the real SignerService. The keymanager DELETE test removes key A through ValidatorManagerAdapter::remove_validator. The epoch E+1 POST must not carry key A, and the signer must make no new signature. The proposer-config URL test sets the builder flag of key A to off through ProposerConfigApplier::apply. The epoch E+1 POST must not carry key A, and the signer must make no new signature. Then the test sets the flag to on. The signer must make one new signature, for key A, and the epoch E+2 POST must carry key A. The behavior shipped in W4-E5 and W4-E6a, thus these tests are a backfill. Hand mutants of production code gave an assertion failure for each test. The mutants are not in this commit.
Coverage ReportTotal: 94.70% (140581 / 148449 lines)
|
rootwarp
left a comment
There was a problem hiding this comment.
PR #1012 R1 security review
The review ends at 2026-11-19 02:27:26 KST (Asia/Seoul). Blake (RSA) reviews W4-E6b3, issue #877, Class T.
Verdict for 0100bbe6 — BLOCK
The review has 1 BLOCKING and 3 NIT findings.
Pins and lineage
| Ref | Commit SHA | Tree SHA |
|---|---|---|
| Assigned head | 0100bbe653283486fe8e9198121145de00000586 |
153fffc7a4ea81715a17586338ae9ddf74bf32e7 |
| Assigned parent and PR base | c82da1c5993d26169ee010418784dc3fcf7db1a1 |
5263875a091c63c9ac543b1fef87bdca1d501cd6 |
The head branch is test/w4-e6b3-877. The base branch is develop.
The PR is a draft and open at both checks.
The local and live head and base match the assigned pins at the start and before this report.
The detached worktree is clean at both checks and after each restore.
The head has one parent, the assigned develop commit.
git merge-base --is-ancestor exits 0. The symmetric range count is 0 1.
This history permits a fast-forward.
Git verifies the head's ED25519 SSH signature for rootwarp@gmail.com.
The stable patch ID is 79d354629baf07d2aec777bf6cb99b3947d024db.
The range has one file, 376 insertions, and five deletions. Cargo.lock has no changes.
The assignment has no earlier review head or rebase pair. Local start pins, live start pins, local final pins, live final pins, assigned diff.
B1 — BLOCKING. Check every POST in the epoch
At crates/rvc/tests/epoch_boundary_restart.rs:1083, the helper waits for at least the requested count.
At line 1091 it returns only one body by position.
T1 checks POST number two at lines 244–259. It leaves the rest of the mock's POST log unchecked.
M6 sends a second POST in epoch E+1 after the expected POST for B.
The mock accepts the extra POST with the cached registration for removed key A. T1 and T2 both pass.
A regression can send another registration for a removed key while the test passes.
Wait for the existing completion event for the epoch registration step before you check its effects.
Record every POST and every call to the registration signer for that epoch.
With this fixture, require one POST and check its complete body. Require the expected set of signer calls before the next store change.
The runner has a completion event at crates/rvc/src/orchestrator/epoch_boundary.rs:748. Source excerpts, M6 log.
N1 — NIT. Describe the calls that the tests make
| Location | Rewrite |
|---|---|
epoch_boundary_restart.rs:21–25 and :973–974 |
“orchestrator_at_slot starts DutyOrchestrator::run and its epoch boundary runner. The harness uses a mock beacon node in the test process. The tests update the live ValidatorStore and count calls to the registration signer.” |
epoch_boundary_restart.rs:223–227 and :242. PR body lines 11–14 and 104. |
“Use ValidatorManagerAdapter::remove_validator to remove key A from the store. The keymanager DELETE uses this adapter.” Use this assertion message at line 242. “The validator manager must remove key A from the store.” |
epoch_boundary_restart.rs:238 and :279 |
“The epoch E POST must contain keys A and B.” |
epoch_boundary_restart.rs:288 |
“While builder_enabled is false, the epoch E+1 POST must not contain key A.” |
epoch_boundary_restart.rs:308 |
“After the test sets builder_enabled to true, the epoch E+2 POST must contain key A.” |
epoch_boundary_restart.rs:253 and :293 |
“The service must make no calls to sign a registration in epoch E+1.” |
epoch_boundary_restart.rs:313 |
“In epoch E+2, the service must make one call to sign a registration for key A.” |
epoch_boundary_restart.rs:1116–1117 and :1128–1134. PR body line 25. |
“The wrapper records the public key before each call. SignerService handles the call.” For total, use “Return the number of calls to sign a registration.” For since, use “Skip the first from calls. Return the public key for each subsequent call.” |
N2 — NIT. Use lists for test conditions
Replace the prose in the two Assertions cells at PR body lines 37–38 with vertical lists.
For T1, use this text.
- The epoch E POST must contain keys A and B.
- After the adapter removes key A, the epoch E+1 POST must contain only the epoch E registration for key B.
- The service must make no calls to sign a registration in epoch E+1.
For T2, use this text.
- The epoch E POST must contain keys A and B.
- While
builder_enabledisfalse, the epoch E+1 POST must contain only the epoch E registration for key B. - The service must make no calls to sign a registration in epoch E+1.
- After the test sets
builder_enabledtotrue, the epoch E+2 POST must contain key A. - In epoch E+2, the service must make one call to sign a registration for key A.
N3 — NIT. Name the test results directly
| Location in the PR body | Rewrite |
|---|---|
| Lines 11 and 106 | “W4-E5 added cache eviction and new signatures for keys that return. W4-E6a added the registration call to the epoch boundary runner. This PR adds two integration tests for that behavior.” |
| Line 33 | “Advance virtual time by one slot per call. At the first slot of the epoch, wait at most half a slot for the registration POST. The reported assertion failures occur before this deadline.” |
| Lines 40–42 | Use “Assertion results for hand mutants” as the heading. Use “Each new test fails an assertion under at least one mutant.” |
| Lines 44–54 | Use “Assertion failure” in each affected Result cell. Use “A mutant causes an assertion failure for each condition below.” |
| Lines 62–64 | Use “Clean test results” as the heading. Use “After the last restore, all seven tests pass. Each new test also passes 50 repeat runs.” |
| Lines 87 and 107 | Use “Each new test fails a behavior assertion under at least one hand mutant.” |
The assertions give a clear meaning for the cited prose. The review applies Issue 9 to new or changed text. Test language notes, live PR body, issue #877.
Store writers, signatures, and secret material
T1 is test_builder_flag_off_via_keymanager_evicts_registration. T2 is test_builder_flag_off_via_proposer_config_url_evicts_registration.
T1 calls the validator manager at crates/rvc/tests/epoch_boundary_restart.rs:241.
The adapter calls remove_validator_durable at crates/rvc/src/keymanager_adapters/validator.rs:52.
The store removes the row in memory for this fixture. The keymanager router has no route that sets builder_proposals.
T2 calls ProposerConfigApplier::apply at crates/rvc/tests/epoch_boundary_restart.rs:282 and :302.
The mapper writes builder_enabled as builder_proposals at crates/rvc/src/background_tasks/config_url.rs:194.
The applier passes the update to the store at line 832.
The pre-Gloas runner calls submit_epoch_registrations at crates/rvc/src/orchestrator/epoch_boundary.rs:746.
At crates/builder/src/service.rs:316–321, the builder set excludes removed keys and keys with the builder flag off.
The service evicts other cache entries at line 238. It signs eligible entries before it sends the POST at line 282.
The test keeps both keys in a local key manager. The signer can therefore report attempts that the builder service must prevent.
The wrapper records the public key before it delegates to the real signer at crates/rvc/tests/epoch_boundary_restart.rs:1147–1149.
On the clean tree, neither test calls the registration signer for A while A is absent or the flag is false.
Both tests check for zero new calls in epoch E+1. T2 requires one new call for A after the flag changes to true.
The fixture supplies no proposer duties and uses a fork schedule without Gloas. Source excerpts, test map.
The diff changes only crates/rvc/tests/epoch_boundary_restart.rs. It adds no production change or test hook in production code.
The tests use the existing test-utils fixture at crates/rvc/src/orchestrator/coordinator/mod.rs:315–317.
The new keys are two synthetic arrays of repeated bytes at test lines 106–108. The new code adds no credentials or tokens.
Gitleaks 8.21.2 finds no leaks in the changed file or the one-commit range. Both scans use the strict CI config.
The first scan fetches public tooling directly. The repeat fetches both pinned inputs through read-only gh. Production check, first Gitleaks log, repeat Gitleaks log.
Hand mutants
Each run applies one source edit and restores the full file from HEAD. Each restore yields an empty git status --porcelain.
The table names the assertion locations in crates/rvc/tests/epoch_boundary_restart.rs.
Command time includes the wait for the lock. Test time comes from nextest.
| Mutant | Change | Result | Test evidence | Command time | Files |
|---|---|---|---|---|---|
| M1 | Skip cache eviction. Send all cached entries. | ASSERTION_KILL |
T1 fails at :246:5 (0.014 s). “The epoch E+1 POST must not carry key A after the keymanager DELETE.” T2 fails at :286:5 (0.014 s). “The epoch E+1 POST must not carry key A after builder_enabled false.” |
881.067 s | Log, diff, JSON, restore |
| M2 | Remove the builder flag filter. | ASSERTION_KILL |
T1 passes (0.010 s). T2 fails at :286:5 (0.011 s). “The epoch E+1 POST must not carry key A after builder_enabled false.” |
14.920 s | Log, diff, JSON, restore |
| M3 | Sign each eligible key each epoch. | ASSERTION_KILL |
T1 fails at :250:5 (0.012 s). “The signer must make no new registration signature in epoch E+1.” T2 fails at :290:5 (0.012 s). “The signer must make no new registration signature in epoch E+1.” |
37.340 s | Log, diff, JSON, restore |
| M4 | Skip cache eviction. Keep the POST filter. | ASSERTION_KILL |
T1 passes (0.011 s). T2 fails at :310:5 (0.011 s). “The signer must make one new registration signature in epoch E+2, for key A.” |
36.187 s | Log, diff, JSON, restore |
| M5 | Write false for both builder flag values. |
ASSERTION_KILL |
T1 passes (0.012 s). T2 fails at :306:5 (0.015 s). “The epoch E+2 POST must carry key A after builder_enabled true.” |
36.406 s | Log, diff, JSON, restore |
| M6 | Send a cached entry for a removed key in a separate POST. | SURVIVED |
T1 passes (0.011 s). T2 passes (0.012 s). | 36.617 s | Log, diff, JSON, restore |
M1 through M5 are assertion kills. No result depends on a build error, timeout, or shutdown guard.
T1 passes M2 because the store removes A from the enabled list.
T1 passes M4 because the POST still uses that list and the test does not add A again.
T1 passes M5 because the test does not use the config URL writer.
The five applied diffs match the supplied diffs byte for byte. Diff comparison.
M6 survives. The log shows one extra batch with one removed entry in epoch 4 (E+1) and zero failures.
This changes the mock's received POSTs and is a non-equivalent mutant.
T2 leaves A in the store, so M6 has no effect on its result.
M6 has no assertion failure. It demonstrates B1.
The log parser now keeps assertion details when nextest repeats a result in its summary.
Clean runs and commands
Both clean runs pass 7 of 7 tests in the assigned file. Each run includes T1 and T2. green before, green after.
T1 and T2 each pass all 50 repeat runs on the clean tree. Repeat test log.
The toolchain reports rustc 1.99.0 (b940084d7 2026-09-28).
The runner sets PROTOC=/usr/local/bin/protoc and CARGO_INCREMENTAL=0.
Every cargo call uses the shared lock and the private target. The lowest guard readings are 48 GB and 384,334 free inodes.
This review ran only targeted nextest. It ran no full-package or workspace-wide nextest suite, Clippy, Docker, or cargo deny. Rust version, lock runner, cargo environment, nextest config, test runner, result parser, progress.
PATH=/home/box/.cargo/bin:/usr/local/bin:$PATH rustc +1.99.0 --version
df -BG /
df -i /
review_root=/workspace/rvc/pr1012-rsa-r1
flock -x /tmp/harper-build.lock timeout 600 "$review_root/scratch/lc.sh" \
/tmp/pr1012-rsa-r1-target nextest run \
--config-file "$review_root/scratch/nextest.toml" \
-p rvc --all-features --test epoch_boundary_restart --no-fail-fastThe first clean run uses a 1,200 s time limit. The mutant commands append -E 'test(/^test_builder_flag_off_via_/)'.
M6 also appends --success-output final. The repeat run uses the filter and --stress-count 50.
CI on the assigned head
The final GitHub check ends at 2026-11-19 02:27:26 KST.
The PR rollup uses 0100bbe653283486fe8e9198121145de00000586 and shows 9 success.
| Actions run ID | Workflow | Status | Conclusion | headSha |
|---|---|---|---|---|
44353609562 |
CI | completed |
success |
0100bbe653283486fe8e9198121145de00000586 |
GitGuardian Security Checks (134975671749) reports success.
CI passes on this head.
PR check rollup, branch Actions runs, commit checks, CI commands.
Model: gpt-6.2-sol, effort high
rootwarp
left a comment
There was a problem hiding this comment.
PR #1012 (W4-E6b3 / #877) — Harper Formal R1
Overall verdict: REQUEST_CHANGES (1 BLOCKING, 3 NIT). Both new tests pass on the pinned head. The five supplied production mutants, and one extra payload mutant, fail by assertion. A further production change shows that both tests accept a registration POST from an earlier epoch.
- Head:
0100bbe653283486fe8e9198121145de00000586(re-checked at FINAL againstgh pr view 1012and the review worktree; unchanged) - Tree:
153fffc7a4ea81715a17586338ae9ddf74bf32e7 - Develop / parent:
c82da1c5993d26169ee010418784dc3fcf7db1a1— 1 ahead / 0 behind (rev-list --left-right --count=0 1). Live develop stayed at this SHA during the review. - Branch
test/w4-e6b3-877. DRAFT, MERGEABLE, OPEN. One signed commit sits directly on develop. This is the expected R1 shape. It is not a finding. Author = committer = Joonkyo Kim<rootwarp@gmail.com>; no trailers. The commit has a Good "git" signature for that address with ED25519 keySHA256:66/BtMChdDUV09Eu2LXyZuu5NU8naVYDPt2RRruMcWo. Authored and committed 2026-11-19 01:50:20 KST (Asia/Seoul).
Diff versus the assigned parent: 1 file, crates/rvc/tests/epoch_boundary_restart.rs, +376/−5. Cargo.lock is unchanged. Production code is not in the diff.
This review used one Codex run: gpt-6.2-sol with model_reasoning_effort=high. The wrapper started at 2026-11-19 02:05:13 KST (Asia/Seoul) and exited 0 at 02:37:04 KST.
BLOCKING
B1 — The tests accept a POST from an earlier epoch
registration_post waits until the POST count reaches the requested number, then reads that body. The watch can already hold that count when the wait starts. The tests do not check the arrival slot, so they still treat an early POST as the epoch E+1 (or E+2) POST.
M7 adds one to the processed slot before epoch selection. Both tests still pass (exit 0), and that run does not prove an early arrival. M7b adds SLOTS_PER_EPOCH - 1 (31). Diagnostic lines in that run show:
- The registration step for epoch 4 (E+1) ended in slot 99. Epoch 4 starts at slot 128.
- In the proposer-config test, the registration step for epoch 5 (E+2) ended in slot 131. Epoch 5 starts at slot 160.
Both tests passed (exit 0). The half-slot wait guard did not fire. This is a claim-bearing, non-equivalent survivor.
The tests must read the matching MockMethod::RegisterValidators timestamp from MockBeaconNodeClient::call_stamps, convert it to a slot with the paused clock origin, and compare it with the expected slot. Use: "Registration POST {count} must arrive in slot {expected_slot}. The recorded slot is {actual_slot}." Keep the half-slot time limit for the wait.
Evidence: pr1012-r1/scratch/harper-logs/mutants/M7b-register-early-confirmation/.
NIT
N1 — Rewrite new test prose
Some new assertion and helper text uses words outside the approved meanings, for example "carry" for POST contents. The meanings are clear. Concrete rewrites are in pr1012-r1/scratch/harper-logs/ste-review.md.
N2 — Rewrite the commit message
The commit message uses "wraps," "shipped," and "backfill." The meaning is clear. A full replacement is in the STE record.
N3 — Shorten the PR body and limit the mutant claim
The first summary sentence has 28 words. The proof heading uses informal terms. The matrix sentence states more than the five supplied runs show. M6 supplies separate evidence for the reused key-B payload. Concrete rewrites are in the STE record.
Tests, mutants, and CI
The unmutated head passes the targeted epoch_boundary_restart suite under rustc 1.99.1 (58cf8f9c3 2026-10-13): 7 passed, 0 skipped, exit 0. The selection includes both new tests. The shared 1.99.1 rustup path was missing, so Harper used a private official 1.99.1 toolchain (commit 58cf8f9c35b355199727627d07ae34a4e2846d67) and did not change /home/box/.rustup.
Hand mutants of production code (original patches applied with no adaptation for M1–M5):
- M1 keep-removed-key: assertion kill, exit 100. T1 at
:246:5, T2 at:286:5. - M2 keep-flag-off-key: assertion kill, exit 100. T1 passes (DELETE removes A from
list_enabled_pubkeys). T2 fails at:286:5. - M3 resign-every-epoch: assertion kill, exit 100. T1 at
:250:5, T2 at:290:5. - M4 no-eviction: assertion kill, exit 100. T1 passes (the send still selects current builder keys). T2 fails at
:310:5. - M5 flag-on-ignored: assertion kill, exit 100. T1 passes (DELETE path). T2 fails at
:306:5. - M6 change-cached-timestamp: assertion kill, exit 100. T1 at
:255:5, T2 at:295:5. This extra mutant covers the reused key-B payload claim. - M7 / M7b: both tests pass, exit 0. See B1. No half-slot guard fired in any run.
Generated cargo mutants --in-diff (test-only diff): exit 0, No mutants to filter. Zero production mutants, zero assertion kills, zero survivors, zero compile failures. The hand mutants are the strength evidence.
The new orchestrator_at_slot harness uses one Tokio thread with paused time, deterministic one-slot advances, a mock beacon node, a store without a config path, and an in-memory slashing database. It uses no network connections, file writes, or wall-clock sleeps. T1 writes through ValidatorManagerAdapter::remove_validator. T2 writes twice through production ProposerConfigApplier::apply on the same applier. Harper agrees with the plan choices: builder_keys supplies the fixture keys; the integration test uses a local counting signer because the coordinator helper is private; the old inherent register_validators path is gone, and the mutants use submit_epoch_registrations.
Same-head CI run 44353609562 is success on 0100bbe653283486fe8e9198121145de00000586. Harper retrieved the metadata and logs through bash -ic with gh, and rechecked them at FINAL. The Test job reports 7,781 passed, one slow, and ten skipped, in 146.945 s, at 2026-11-19 02:08:11 KST (Asia/Seoul) (job 134975670107). Doctests report 29 passed and 6 ignored. The Check job runs fmt and workspace Clippy with all features (job 134975670640). Architecture Gates (job 134975670769), Coverage, Scripts, Secret Scan, Spec Vectors, Trace continuity, and GitGuardian also pass. All nine required checks pass. CI uses rustc 1.100.0. This review does not rerun the full workspace suite or Clippy locally.
Every cargo command used /workspace/rvc/bin/locked-cargo, flock -x /tmp/harper-build.lock, rustc 1.99.1, CARGO_INCREMENTAL=0, PROTOC=/usr/local/bin/protoc, CARGO_BUILD_JOBS=1, and private /tmp/harper-pr1012-r1-target. The review did not use /workspace/rvc/target-shared. The private target is absent after the run. One cargo job ran at a time. Harper waited for the shared lock and left other processes alone. The worktree is clean at the pinned head and tree. No comment, review, push, merge, or source fix was made. Harper verified the pins, the assertion-only kills, the M7b survivor, the STE items, and the CI counts, then wrote this FINAL.
Copies: lead/handoff/pr1012, pr1012-r1/out, /home/box/rvc-review-archive/pr1012. First line <!-- prepared by Harper for Ash -->. Not posted. Did not message Ash.
This text replaces the description in commit 0100bbe: Add orchestrator_at_slot to epoch_boundary_restart.rs. The harness runs the orchestrator and its epoch boundary runner with paused time. It uses a mock beacon node and records calls to the local signer. The keymanager DELETE test removes key A with ValidatorManagerAdapter::remove_validator. The epoch E+1 POST must contain only the epoch E registration for key B. The signer must not sign a new registration in epoch E+1. The proposer configuration test sets builder_enabled to false for key A with ProposerConfigApplier::apply. The same epoch E+1 conditions apply. The test then sets builder_enabled to true for key A. In epoch E+2, the signer must sign one new registration for key A. The epoch E+2 POST must contain a registration for key A. W4-E5 and W4-E6a add this behavior. Temporary changes to the production code cause assertion failures in each new test. Commit 0100bbe contains the tests. This commit makes these changes to the two tests: A tracing layer records the "Epoch builder registration completed" event of the epoch boundary runner. The new helper registration_epoch waits for this event for one epoch. The wait has a time limit of half a slot. Then the helper gets each registration POST and each signer call after its last check. Each POST must arrive in the first slot of its epoch. The helper reads the arrival time from MockBeaconNodeClient::call_stamps. It uses the paused clock origin to change the time to a slot. Each epoch must have one POST. The signer wrapper records the registration and the signature of each call. The epoch E POST must contain the registrations that the signer signs in epoch E. The epoch E+1 POST must contain only the epoch E registration for key B. The epoch E+2 POST must contain only the new registration for key A and the epoch E registration for key B. A temporary production change sends a second POST in epoch E+1 with the cached registration for removed key A. The keymanager test then fails on the one-POST assertion. A second temporary change makes the runner use an epoch 31 slots early. Both tests then fail on the slot assertion. The test text now uses ASD-STE100 wording.
rootwarp
left a comment
There was a problem hiding this comment.
PR #1012 (W4-E6b3 / #877) — Harper fix1 recheck
Overall verdict: CLEARED (B1 PASS; N1 PASS, N2 PASS, N3 PASS). No new findings outside the assigned items.
- Head:
920a7afacce843426a54f719a198c5bc886a07e4(re-checked at FINAL againstgh pr view 1012and the review worktree; unchanged) - Tree:
911477cb3bf504b2ac541df3cfe621c4fefa43c0 - Develop / parent:
c82da1c5993d26169ee010418784dc3fcf7db1a1— 2 ahead / 0 behind (rev-list --left-right --count=0 2). Live develop stayed at this SHA during the recheck. - R1 head:
0100bbe653283486fe8e9198121145de00000586 - Branch
test/w4-e6b3-877. DRAFT, MERGEABLE, OPEN. The reviewed commit0100bbe6remains, plus one additive fix commit920a7afa. This shape is expected. It is not a finding. Author = committer = Joonkyo Kim<rootwarp@gmail.com>; no trailers. Both commits have a Good "git" signature for that address with ED25519 keySHA256:66/BtMChdDUV09Eu2LXyZuu5NU8naVYDPt2RRruMcWo. Recheck scope: Harper R1 B1 and N1–N3 only.
An independent git range-diff of c82da1c5..0100bbe6 against c82da1c5..920a7afa matches the supplied range-diff byte for byte. Diff of the fix commit: 1 file, crates/rvc/tests/epoch_boundary_restart.rs, +254/−121. Cargo.lock is unchanged.
This recheck used one Codex run: gpt-6.2-sol with model_reasoning_effort=high. The wrapper started at 2026-11-19 03:16:19 KST (Asia/Seoul) and exited 0 at 03:37:32 KST.
Item verdicts
B1 — PASS
Both new tests call registration_epoch, which waits for the registration step to end, reads new MockMethod::RegisterValidators stamps from MockBeaconNodeClient::call_stamps, converts each stamp to a slot with the paused clock origin, and compares that slot with the first slot of the epoch. The assertion message is “Registration POST {count} must arrive in slot {expected_slot}. The recorded slot is {actual_slot}.”
The original R1 M7b patch applies with both hunks and no adaptation. It adds SLOTS_PER_EPOCH - 1 to the processed slot before epoch selection. Both targeted tests fail by assertion at crates/rvc/tests/epoch_boundary_restart.rs:1176:13: “Registration POST 2 must arrive in slot 128. The recorded slot is 98.” The counts are left: 98 and right: 128. Nextest exits 100: 0 passed, 2 failed, 5 skipped. The half-slot wait guard does not fire. This is an assertion kill, not a timeout. The tree was restored clean after the run.
Evidence: pr1012-fix1/scratch/harper-logs/m7b/.
N1 — PASS
The named test-prose sites now use the R1 STE rewrites or equivalent wording. The messages use “contain” for POST contents and “sign” for signer actions. The slot assertion uses the R1 wording. The site map is in pr1012-fix1/scratch/harper-logs/ste/recheck.md.
N2 — PASS
The new fix commit 920a7afa uses the replacement description with STE wording. The reviewed 0100bbe6 message stays, as required.
N3 — PASS
The live PR body replaces the 28-word summary with three shorter sentences. The proof heading is “Assertion results for hand mutants.” The matrix identifies the five supplied runs, and it describes M6 as separate evidence.
Tests and CI
The unmutated head passes the targeted epoch_boundary_restart suite under rustc 1.99.1 (58cf8f9c3 2026-10-13): 7 passed, 0 skipped, exit 0. The selection includes both new tests. The shared 1.99.1 rustup path was missing, so Harper used a private official 1.99.1 toolchain (commit 58cf8f9c35b355199727627d07ae34a4e2846d67) and did not change /home/box/.rustup. This recheck did not run cargo mutants --in-diff.
Same-head CI run 44367055012 is success on 920a7afacce843426a54f719a198c5bc886a07e4. Harper retrieved the metadata and logs through bash -ic with gh, and rechecked them at FINAL. The Test job reports 7,781 passed, one slow, and ten skipped, in 145.556 s, at 2026-11-19 03:19:29 KST (Asia/Seoul) (job 135023517583). Both new tests pass in that job. Doctests report 29 passed and 6 ignored. The Check job runs fmt and workspace Clippy with all features (job 135023517352). Architecture Gates (job 135023516838), Coverage, Scripts, Secret Scan, Spec Vectors, Trace continuity, and GitGuardian also pass. All nine required checks pass. CI uses rustc 1.100.0. This review does not rerun the full workspace suite or Clippy locally.
Every cargo command used /workspace/rvc/bin/locked-cargo, flock -x /tmp/harper-build.lock, rustc 1.99.1, CARGO_INCREMENTAL=0, PROTOC=/usr/local/bin/protoc, CARGO_BUILD_JOBS=1, and private /tmp/harper-pr1012-fix1-target. The review did not use /workspace/rvc/target-shared. The private target is absent after the run. One cargo job ran at a time. Harper waited for the shared lock and left other processes alone. The worktree is clean at the pinned head and tree. No comment, review, push, merge, or source fix was made. Harper verified the pins, the M7b assertion kill, the STE items, and the CI counts, then wrote this FINAL.
Copies: lead/handoff/pr1012, pr1012-fix1/out, /home/box/rvc-review-archive/pr1012. First line <!-- prepared by Harper for Ash -->. Not posted. Did not message Ash.
rootwarp
left a comment
There was a problem hiding this comment.
PR #1012 fix1 security recheck
Blake (RSA) completes the review at 2026-11-19 03:38:54 KST (Asia/Seoul).
Verdict for 920a7afa — SECURITY-CLEAR
The review covers W4-E6b3, issue #877, Class T. B1 is FIXED. The two remaining nits do not block this verdict.
Pins and lineage
| Ref | Commit SHA | Tree SHA |
|---|---|---|
| Assigned head | 920a7afacce843426a54f719a198c5bc886a07e4 |
911477cb3bf504b2ac541df3cfe621c4fefa43c0 |
| Head parent and R1 head | 0100bbe653283486fe8e9198121145de00000586 |
153fffc7a4ea81715a17586338ae9ddf74bf32e7 |
| Develop base | c82da1c5993d26169ee010418784dc3fcf7db1a1 |
5263875a091c63c9ac543b1fef87bdca1d501cd6 |
PR #1012 is open and in draft. The head branch is test/w4-e6b3-877. The base branch is develop.
The local and live head and base match the pins at the start and before this report.
The detached worktree is clean at both checks.
The range 0100bbe6..920a7afa adds one commit. The range c82da1c5..920a7afa contains two commits.
git merge-base --is-ancestor exits 0. git rev-list --left-right --count returns 0 and 2, separated by a tab.
Develop is an ancestor of the head. The history permits a fast-forward.
Git verifies the ED25519 signature of 920a7afa for rootwarp@gmail.com. Initial local pins, initial live pins, final local pins, final live pins, signature verification.
Range diff and patch IDs
The regenerated range diff matches both Glen copies byte for byte.
It shows 0100bbe6 = 0100bbe6 and adds 920a7afa as entry 2. The first commit is identical to R1.
The fix changes only crates/rvc/tests/epoch_boundary_restart.rs. Git reports 254 insertions and 121 deletions.
The Cargo.lock diff is empty. The change is test-only. Range diff, range comparison, stable patch IDs, R1 patch ID comparison, file counts, source delta.
| Commit | Stable patch ID |
|---|---|
0100bbe6 |
79d354629baf07d2aec777bf6cb99b3947d024db |
920a7afa |
3335715c39720ee4b92c9421a74e35806ccfe270 |
B1 — FIXED. Check every POST in the epoch
T1 is test_builder_flag_off_via_keymanager_evicts_registration. T2 is test_builder_flag_off_via_proposer_config_url_evicts_registration.
The service awaits the signer calls and the POST batches at crates/builder/src/service.rs:260 and :282.
The batch loop awaits the mock at line 401. The mock records each arrival and each body at crates/bn-manager/src/mock.rs:1280–1281.
The runner emits the completion event at crates/rvc/src/orchestrator/epoch_boundary.rs:748 after the service returns success.
RegistrationSteps records the epoch at crates/rvc/tests/epoch_boundary_restart.rs:1210–1216.
registration_epoch waits for this event at lines 1163–1169. It reads every unchecked POST at line 1170.
It checks each arrival slot at lines 1171–1180 and requires one POST at line 1182.
It reads every unchecked signer call at lines 1188–1191.
CountingSigner records each registration before it calls the signer. It adds each successful signature at lines 1271–1281. Test source, registration call sites.
T1 and T2 check each epoch before the next store change. The body comparisons include every message field and the signature.
- In epoch E, both tests require one signer call per key and a POST with both signed entries at lines 318–333.
- In epoch E+1, both tests require zero signer calls and the complete epoch E entry for key B at lines 338–348.
- In T2 epoch E+2, the test requires one signer call for A at lines 304–308. It compares the full POST at lines 309–314. The POST must contain the new entry for A and the epoch E entry for B.
T1 changes the store at line 247. T2 changes it at lines 286 and 296.
The checks catch a second POST that contains a cached registration for a removed key. Epoch E checks, epoch E+1 checks, T2 epoch E+2 checks, source notes.
M6 — Assertion kill
M6 sends a cached registration for a removed key in a separate POST. The applied diff matches the R1 diff byte for byte.
The mock reports one extra batch in epoch 4 with one removed entry and zero send failures.
T1 fails in 0.026 s at crates/rvc/tests/epoch_boundary_restart.rs:1182:9. The assertion assert_eq!(posts.len(), 1) reports left: 2 and right: 1.
The message starts, “In epoch 4, the service must send one registration POST.” The log also includes both bodies.
This result is an assertion kill. The completion guard passes before the count check.
T2 passes in 0.031 s. It keeps A in the store.
Nextest exits 100 and reports 7 tests run: 6 passed, 1 failed, 0 skipped.
The command takes 982.877 s, which includes the wait for the lock.
Git restores the full source file from HEAD. The next git status --porcelain gives no output. M6 log, applied diff, JSON result, restore log, diff comparison.
N1 — NOT FIXED. Describe the calls that the tests make
The module and harness descriptions name the orchestrator and the mock at epoch_boundary_restart.rs:21–27 and :1050–1054.
The T1 description names the adapter at lines 227–229. The removal assertion names the validator manager at line 248.
The shared POST and signer assertions use the R1 wording at lines 322 and 342. T2 uses it at lines 292, 302, and 307.
CountingSigner::calls replaces total and since. This helper gives access to the signer records at lines 1257–1260.
The clean log reports PASS for T1 and T2. Test source, clean run.
Two changed passages still need more precise words.
- Line 254 names a keymanager DELETE. T1 calls the adapter directly at line 247. Use “After the adapter removes key A, the epoch E+1 POST must not contain key A.”
- The signer description at lines 1249–1251 and PR body line 27 says the wrapper adds the signature. At lines 1278–1281, this depends on a successful call. Use “If
SignerServicereturns a signature, the wrapper adds it to the record.”
These changes keep the context and the requirements clear. They concern the wording in the R1 nit. T1 assertion, signer source, live PR body.
N2 — FIXED. Use lists for test conditions
The live PR body gives T1 assertions as a vertical list at lines 55–59. It gives T2 assertions as a vertical list at lines 63–70.
The test descriptions also use vertical lists at epoch_boundary_restart.rs:232–235 and :264–273.
The comparison with Glen's PR body copy exits 0. The clean log reports PASS for both tests. T1 list, T2 list, body comparison, test descriptions, clean run.
N3 — NOT FIXED. Name the test results directly
The live PR body uses the R1 summary at lines 11 and 150. Line 41 describes the half-slot limit and the assertion failures.
Line 72 uses “Assertion results for hand mutants.” The Result cells at lines 80–89 say “Assertion failure.”
Line 105 uses “Clean test results.” Line 107 gives the clean test count.
Line 151 uses the direct hand-mutant sentence. The independent M6 and clean runs report the assertion failure and pass counts for T1 and T2. Live PR body, M6 log, clean run.
The self-check at line 129 still says the tests fail “with the fix removed.” The runs use temporary changes to the production code.
Use “Each new test fails a behavior assertion under at least one hand mutant.” This sentence names the test result and its condition. Self-check text.
Clean run and commands
After the restore, nextest exits 0 and reports 7 tests run: 7 passed, 0 skipped. Both T1 and T2 pass.
The worktree is clean after the run. Clean run.
The toolchain reports rustc 1.99.0 (b940084d7 2026-09-28).
The cargo wrapper sets PROTOC=/usr/local/bin/protoc and CARGO_INCREMENTAL=0.
Both calls use the shared flock and the private target. The lowest disk and inode guard readings are 49 GB and 409,775 free inodes.
The review runs this test file twice, once with M6 and once after the restore.
The review runs zero full nextest suites, Clippy commands, or Docker commands.
A shell quoting error omits paths from the first M6 progress entries. The journal now includes those paths. Rust version, cargo wrapper, lock runner, test runner, nextest config, progress.
The following command uses the M6 limit of 1,200 s. The clean call uses 600 s and the same test selection.
PATH=/home/box/.cargo/bin:/usr/local/bin:$PATH rustc +1.99.0 --version
df -BG /
df -i /
review_root=/workspace/rvc/pr1012-rsa-fix1
flock -x /tmp/harper-build.lock timeout 1200 "$review_root/scratch/lc.sh" \
/tmp/pr1012-rsa-fix1-target nextest run \
--config-file "$review_root/scratch/nextest.toml" \
-p rvc --all-features --test epoch_boundary_restart --no-fail-fastCI on the assigned head
The final CI and pin checks run at 2026-11-19 03:38:54 KST.
The PR rollup shows nine completed checks with conclusion SUCCESS. The commit check API shows the same nine successes on the pinned SHA.
| Actions run ID | Event | Status | Conclusion | headSha |
|---|---|---|---|---|
44367055012 |
pull_request |
completed |
success |
920a7afacce843426a54f719a198c5bc886a07e4 |
GitGuardian Security Checks also reports success on this head. Its check ID is 135023505769. PR rollup, Actions runs, commit checks, CI commands.
Model: gpt-6.2-sol, effort high
Closes #877
Closes #654
Closes #696
This PR completes #654 AC10 and AC11, and #696 item 5, as #877 states.
The branch is on develop
c82da1c5993d26169ee010418784dc3fcf7db1a1.Summary
W4-E5 added cache eviction and new signatures for keys that return. W4-E6a added the registration call to the epoch boundary runner. This PR adds two integration tests for that behavior.
The runner removes a cached registration when its key leaves the builder set. When a key becomes builder-enabled again, the runner signs a new registration. The tests use the two runtime store writers:
ValidatorManagerAdapter::remove_validatorto remove key A from the store. The keymanager DELETE uses this adapter.ProposerConfigApplier::applyto setbuilder_enabledfor key A.This PR changes only tests. It has no production change and no behavior change.
Changes
crates/rvc/tests/epoch_boundary_restart.rs:orchestrator_at_slot(slot, OrchestratorOpts):orchestrator_at_slotstartsDutyOrchestrator::runand the epoch boundary runner fromOrchestratorDeps::for_test.BuilderServicesends its requests toMockBeaconNodeClient. The mock records each registration POST and its arrival time.ValidatorStoreholds keys A and B in memory.CountingSignerrecords each call to sign a registration. The wrapper records the registration before each call.SignerServicehandles the call. Then the wrapper adds the signature to the record.PausedSlotClockgives the current slot. The duty phase offsets are zero.Epoch builder registration completedevent of the runner.Orchestrated::registration_epoch(epoch):epoch. The wait has a time limit of half a slot.MockBeaconNodeClient::call_stamps.test_builder_flag_off_via_keymanager_evicts_registration.test_builder_flag_off_via_proposer_config_url_evicts_registration.Tests
Each test uses a single-thread Tokio runtime with paused time. Each call to
tokio::time::advancemoves virtual time by one slot. The new harness uses no files or sockets. At the first slot of each checked epoch, the helper waits at most half a slot for the end of the registration step. The reported assertion failures occur before this time limit.test_builder_flag_off_via_keymanager_evicts_registrationbuilder_enabledisfalse, the next POST does not contain the key, and the service makes no new signature calls. After the test setsbuilder_enabledtotrue, the service signs a new registration and the next POST contains the keytest_builder_flag_off_via_proposer_config_url_evicts_registrationregistration_epochapplies these conditions to each checked epoch of both tests:test_builder_flag_off_via_keymanager_evicts_registration:test_builder_flag_off_via_proposer_config_url_evicts_registration:builder_enabledisfalse, the epoch E+1 POST must contain only the epoch E registration for key B.builder_enabledtotrue, the epoch E+2 POST must contain key A.Assertion results for hand mutants
W4-E5 and W4-E6a add the behavior, so the new tests pass on the base. I applied one change at a time to the production code. I ran the two new tests. I removed the change after each run with
git checkout -- <file>. After each removal,git status --shortshowed no production file. No change is in a commit. The table identifies each assertion that fails. No result comes from a timeout, the half-slot time limit, a compile error, or a panic outside an assertion. Each new test fails an assertion under at least one mutant.RSA M6 and Harper M7b are the review mutants of round 1. M1 to M5 are the five hand mutants of the first commit. This round ran M1 to M5 again on the new head.
crates/builder/src/service.rs,BuilderService::submit_epoch_registrationscrates/rvc/src/orchestrator/epoch_boundary.rs,EpochBoundaryTask::runlet slot = slot.saturating_add(SLOTS_PER_EPOCH - 1);before the epoch selectioncrates/builder/src/service.rs,BuilderService::submit_epoch_registrationscache.retain. The POST contains each cached entry.builder_enabledisfalse, the epoch E+1 POST must not contain key A.crates/builder/src/service.rs,BuilderService::builder_keysis_builder_enabledfilter.builder_enabledisfalse, the epoch E+1 POST must not contain key A.crates/builder/src/service.rs,BuilderService::submit_epoch_registrationslet unchanged = false;crates/builder/src/service.rs,BuilderService::submit_epoch_registrationscache.retain. Key A keeps its cache entry whilebuilder_enabledisfalse.crates/rvc/src/background_tasks/config_url.rs,ValidatorConfigUpdate::to_store_updatebuilder_proposals: self.builder_enabled.map(|_| false)builder_enabledtotrue, the epoch E+2 POST must contain key A.The config URL test passes under RSA M6. Key A stays in the store in that test, so the mutant finds no key to send. The keymanager test passes under M2, M4, and M5, because it does not use the builder flag or the URL writer.
A mutant causes an assertion failure for each condition below:
false.These runs do not reach the comparison with the unchanged epoch E registration of key B. Harper's round 1 review gives separate failure evidence for that comparison: a review mutant that changes the cached timestamp causes an assertion failure.
Clean test results
After the last restore, all seven tests in
epoch_boundary_restartpass. Each new test also passes 50 repeat runs.Gates
Final head:
920a7afacce843426a54f719a198c5bc886a07e4. Toolchain: 1.100.0. Each cargo gate used a private target directory.cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo nextest run -p rvc --all-features --no-fail-fastcargo nextest run -p rvc --all-features --test epoch_boundary_restart --stress-count 50 test_builder_flag_off_via_keymanager_evicts_registrationcargo nextest run -p rvc --all-features --test epoch_boundary_restart --stress-count 50 test_builder_flag_off_via_proposer_config_url_evicts_registrationcargo deny check advisories sourcescargo mutants--no-git)scripts/gitleaks_range_scan.sh <strict config> c82da1c5993d26169ee010418784dc3fcf7db1a1git diff --name-only c82da1c5993d26169ee010418784dc3fcf7db1a1..HEADcrates/rvc/tests/epoch_boundary_restart.rscargo nextest run -p rvc-architecture-tests --all-features --no-fail-fastPre-PR self-check
remove_validator,ProposerConfigApplier::apply,builder_enabled,submit_epoch_registrations, the keymanager routes,Epoch builder registration completed, andMockBeaconNodeClient::call_stamps. The source checks appear in the list below.hash_tree_roottests are KAT-anchored orkat_exemptwith a reason. This PR adds no test with such a name.git diff --name-only c82da1c5993d26169ee010418784dc3fcf7db1a1..HEADshows onlycrates/rvc/tests/epoch_boundary_restart.rs.Source checks on
c82da1c5:builder_proposals.ValidatorManagerAdapter::remove_validatorcallsValidatorStore::remove_validator_durable.ProposerConfigApplier::applywritesbuilder_enabledasbuilder_proposals.BuilderService::submit_epoch_registrationsevicts a key that left the builder set. The runner stepbuilder_registrationcalls it in each epoch.Epoch builder registration completedwith the epoch aftersubmit_epoch_registrationsreturnsOk(crates/rvc/src/orchestrator/epoch_boundary.rs:748).MockBeaconNodeClient::call_stampsrecords oneMockMethod::RegisterValidatorsstamp, in virtual time, before each registration call.BuilderService::register_validatorsmethod is absent from the base.Decisions (intentional)
builder_proposals. The keymanager test usesValidatorManagerAdapter::remove_validatorto remove key A from the store. The keymanager DELETE uses this adapter. The test keeps its assigned name.BuilderService::register_validatorsmethod is absent from the base, so the earlier failure at64c8ccb7cannot run here. Each new test fails a behavior assertion under at least one hand mutant.Epoch builder registration completedevent of the runner. A tracing layer in the test records it. This needs no production change.Fix round 1
registration_epochwaits for the end of the registration step. Then it gets every POST and every signer call after its last check. Each checked epoch must have one POST. The tests check the complete body of each POST and the signer calls before the next store change. RSA M6 now causes an assertion failure in the keymanager test.registration_epochreads eachMockMethod::RegisterValidatorsstamp fromMockBeaconNodeClient::call_stamps. It uses the paused clock origin to get the slot, and compares it with the first slot of the epoch. Harper M7b now causes the slot assertion failure in both tests.CountingSignertext use the rewrites of the review.0100bbe6.Release note
None. This PR has no behavior change. The W4-E6a note covers eviction.
Size
git diff --shortstat c82da1c5993d26169ee010418784dc3fcf7db1a1...HEAD: 1 file changed, 509 insertions(+), 5 deletions(-). Gross: 514 lines. Cargo.lock has no change.Fix round 1 (
git diff --shortstat 0100bbe653283486fe8e9198121145de00000586..HEAD): 1 file changed, 254 insertions(+), 121 deletions(-). Gross: 375 lines.