Skip to content

Loosen float32 tolerance in edge-effects mutation test - #1850

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
YHC66:fix-edge-effects-mutation-tolerance
Oct 5, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
YHC66:fix-edge-effects-mutation-tolerance

Conversation

@YHC66

@YHC66 YHC66 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Description

tests/unit/tools/test_attribution_patching.py::test_edge_effects_mutation_only_changes_the_perturbed_writers_edges fails with torch 2.14.1 (the latest release; pyproject.toml allows torch>=2.6, the lock pins 2.11.0):

>   assert mutated_scores[edge] == pytest.approx(expected)
E   assert 0.000663460697978735 == 0.00066345982...8213 ± 6.6e-10

The expected value is baseline_scores[edge] + shift, and for this edge the two partly cancel (≈ -5.3e-4 + 1.19e-3), so float32 rounding error relative to the result is amplified. The worst relative error over all edges is 1.3e-6 with torch 2.14.1 and already 5.3e-7 with torch 2.11.0, against pytest.approx's default rel=1e-6. The library code is fine; only the tolerance is too tight for float32.

This switches the assertion to abs=1e-6, the tolerance the other edge-score comparisons in the same file already use. The shifts in this test are around 1e-3, so the check still catches a mutation that is missing or applied to the wrong edge.

Observed on macOS arm64 (CPU). With torch 2.14.1 + transformers 5.17 this was the only failure in tests/unit on dev (6affbe99): 6933 passed, 1 failed.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

The expected score is baseline + shift, and the two partly cancel, so the
float32 rounding error relative to the result can exceed pytest.approx's
default rel=1e-6. With torch 2.14.1 on CPU the test fails at 1.3e-6
relative error (5.3e-7 with the locked torch 2.11). Use abs=1e-6, the
tolerance the other edge-score comparisons in this file already use.
@jlarson4

jlarson4 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Looks good @YHC66, thanks for this slight tolerance adjustment

@jlarson4
jlarson4 merged commit 1012730 into TransformerLensOrg:dev Oct 5, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants