Repository navigation
rebase: guard automatic debug patches - #129
Conversation
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f59b2b3a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| policy = evaluate_debug_patch( | ||
| Path(repo), policy_before, local_verdict=local) |
There was a problem hiding this comment.
The new policy is invoked only by the remote-CI debug_fn; the other automatic debug path in rebase.v3_test_loop (lines 1332–1372) still accepts any agent edit once its post-debug rerun turns green. If that agent weakens the failing assertion, the modified test itself makes the rerun pass and the change can later be committed and pushed, bypassing the safety invariant this commit introduces. Capture and evaluate the policy snapshot around that debug agent as well.
Useful? React with 👍 / 👎.
| if Path(relative).suffix.lower() in _POLICY_SUFFIXES: | ||
| text = content.decode("utf-8", errors="replace") | ||
| lines = tuple(line.strip() for line in text.splitlines() | ||
| if _SENSITIVE_LINE.search(line)) |
There was a problem hiding this comment.
Recording only lines that themselves contain assert or a tolerance keyword misses common multiline and indirect oracles. For example, changing expected = 0.8 to 0.7 while leaving a later multiline assert ( line unchanged produces identical sensitive_lines; when the modified test passes locally, the test-hash check also permits it. The automatic agent can therefore weaken the oracle and send the resulting green patch to remote CI despite the advertised prohibition.
Useful? React with 👍 / 👎.
| test_paths = tuple(sorted( | ||
| path for path in set(before.test_hashes) | set(after.test_hashes) | ||
| if before.test_hashes.get(path) != after.test_hashes.get(path) | ||
| )) | ||
| if test_paths and local_verdict != "passed": |
There was a problem hiding this comment.
A single local_verdict="passed" authorizes every changed test file, but _verify_locally runs only the manifest job associated with the failed CI slug. If the agent edits a test outside that command's selection—such as another test directory or job—and avoids the sensitive-line heuristic, an unrelated green job permits the unexecuted test edit to be pushed. The decision needs evidence that each reported test_path was covered by the local verification, or it should reject cross-job test edits.
Useful? React with 👍 / 👎.
Summary
Tests
pytest -qsuiteruff checkfor the new policy module