Repository navigation
Add the arm64 NEON implementation of the ScaleInvariantMatrixMatch kernels - #1602
Merged
Merged
Conversation
Add a TestInfra for the four kernel functions (compute_scale and compute_error, plain and weighted). Each test generates a random width x height matrix pair (and weight matrix for the weighted variants), runs the Default path and the ISA path on identical input, and compares the results. The Default path is the source of truth, so each ISA registers the same (width, height, use_weights) combinations against Default. The sizes cover 1x1 up to 1000x1000, both divisible and undivisible by the vector lengths (4, 8, 16), including the short-row cases that exercise unaligned row prefixes and partial vector tails. The SSE4.1 path back-reads up to 3 floats before a row start, so the matrices are allocated with 16 floats of head padding to keep those reads in-bounds. Scales are not computed exactly by all paths (different addition orders), so results are compared with a tolerance: pass if within 4096 ULPs, or within 1e-4 of the magnitude of the accumulated sum (the absolute part covers near-zero results, where ULPs are far smaller than the accumulated rounding of a multi-million-term sum). The tests are registered per-ISA in each x86 Core file (SSE, AVX2, AVX512), guarded by the same PA_AutoDispatch and CPU_CAPABILITY_CURRENT checks the dispatchers use, so each architecture only runs the tests for the ISAs it can execute.
Mysticial
force-pushed
the
kernels-simm-neon
branch
from
October 17, 2026 10:08
79b75c9 to
897f0aa
Compare
Each ISA kernel requires a minimum row width (4 for SSE4.1, 8 for AVX2, 16 for AVX512): below it, the partial-row loads span past the row end and can read past the end of the allocation. The test registration now skips sizes below the ISA's minimum width. The test matrices also get 15 floats of tail padding after the last row: the SSE4.1 partial-tail load (PSHUFB-based) reads a full 16-byte word that extends up to VL - 1 floats past the row end, which would otherwise run past the allocation on the last row.
…rnels Context_arm64_NEON implements the same SumATA2/SumError routines with float32x4_t vectors (VL = 4). The partial-row loads are lane-insert loads into zeroed vectors, so they never read out of bounds in either direction (no back-reads at the row start, no reads past the row end), unlike the SSE4.1 PSHUFB-based loads. Dispatched for width >= 4 when OK_M1 is enabled; the unit tests (compare against the Default path) are registered for all test sizes with width >= 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1601 (the unit tests for these kernels) — review/merge #1601 first, this one only contains the NEON port.
Adds
Context_arm64_NEON(float32x4_t, VL = 4) implementing the sharedSumATA2/SumErrorroutines from Routines.h, with all four kernel entry points (compute_scale/compute_error, plain and weighted), dispatched forwidth >= 4underOK_M1.The partial-row loads differ from the x86 ISAs by design: instead of the PSHUFB/mask-load tricks, they are lane-insert loads (
vld1q_lane_f32) into zeroed vectors:alignlanes. No back-read, so nothing is read before the row start.lengthremaining floats into a zeroed vector. Nothing is read past the row end.This means the kernel has no out-of-bounds reads in either direction, unlike the SSE4.1 path (whose PSHUFB tail load reads up to VL - 1 floats past the row end and relies on the caller's buffer having tail padding — see the
TAIL_PADcomment in the test infrastructure from #1601).Unit tests: the same TestInfra as #1601 (41 width/height pairs, plain + weighted, Default as source of truth, ULP/absolute tolerance) is registered for all sizes with width >= 4 → 64 tests on arm64. Verified locally: full suite passes (5,495 passed / 0 failed), including all 64 NEON tests.