Loosen float32 tolerance in edge-effects mutation test - #1850
Merged
jlarson4 merged 1 commit intoOct 5, 2026
Merged
Conversation
The expected score is baseline + shift, and the two partly cancel, so the float32 rounding error relative to the result can exceed pytest.approx's default rel=1e-6. With torch 2.14.1 on CPU the test fails at 1.3e-6 relative error (5.3e-7 with the locked torch 2.11). Use abs=1e-6, the tolerance the other edge-score comparisons in this file already use.
Collaborator
|
Looks good @YHC66, thanks for this slight tolerance adjustment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
tests/unit/tools/test_attribution_patching.py::test_edge_effects_mutation_only_changes_the_perturbed_writers_edgesfails with torch 2.14.1 (the latest release;pyproject.tomlallowstorch>=2.6, the lock pins 2.11.0):The expected value is
baseline_scores[edge] + shift, and for this edge the two partly cancel (≈ -5.3e-4 + 1.19e-3), so float32 rounding error relative to the result is amplified. The worst relative error over all edges is 1.3e-6 with torch 2.14.1 and already 5.3e-7 with torch 2.11.0, againstpytest.approx's defaultrel=1e-6. The library code is fine; only the tolerance is too tight for float32.This switches the assertion to
abs=1e-6, the tolerance the other edge-score comparisons in the same file already use. The shifts in this test are around 1e-3, so the check still catches a mutation that is missing or applied to the wrong edge.Observed on macOS arm64 (CPU). With torch 2.14.1 + transformers 5.17 this was the only failure in
tests/unitondev(6affbe99): 6933 passed, 1 failed.Type of change
Checklist: