gh-156443: Keep PyLong loop carries as twodigits in shifts and division

[XiaohongGong]

gh-156443: Keep PyLong loop carries as twodigits in shifts and division

Several functions in longobject.c (v_lshift, v_rshift, and x_divrem) narrowed a loop carry value to digit or sdigit, then widened it again on the next iteration.

On AArch64, that 64-to-32-to-64 conversion inserts an extra mov on the loop-carried critical path. Keeping the carry at twodigits until the function returns drops that mov and shortens the carry chain. Results are unchanged for valid limbs.

Use v_lshift as an example, the loop on AArch64 previously contains a redundant mov w3, w3 on the carry chain:

  ldr  w0, [x4, x2, lsl #2]    ; a[i]
  mov  w3, w3                  ; carry chain
  lsl  x0, x0, x24             ; a[i] << d
  orr  x0, x0, x3              ; carry chain
  and  w1, w0, #0x3fffffff
  ubfx x3, x0, #30, #32        ; carry chain
  str  w1, [x26, x2, lsl #2]
  add  x2, x2, #1
  cmp  x25, x2
  b.ne

Keeping the carry as twodigits removes that narrowing conversion. The loop is optimized to:

  ldr  w0, [x4, x2, lsl #2]    ; a[i]
  lsl  x0, x0, x24             ; a[i] << d
  orr  x0, x0, x3              ; carry chain
  and  w1, w0, #0x3fffffff
  str  w1, [x26, x2, lsl #2]
  add  x2, x2, #1
  lsr  x3, x0, #30             ; carry chain
  cmp  x25, x2
  b.ne

The instruction count on the loop carried chain is reduced from 3 to 2. We can observe ~20% performance improvement of the
pyperformance pidigits benchmark on an NVIDIA Grace CPU, while no material regressions observed on other platforms and benchmarks.

Tests cover divmod of saturated limbs with quotients near BASE (x_divrem's inner loop), and intra-digit shifts via float() and true division: full-limb values and powers of ten.

Fixes gh-156443.

Co-authored-by: Kyrylo Tkachov ktkachov@nvidia.com

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[XiaohongGong]

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

[picnixz]

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

Be careful with which email you sign the CLA (see https://devguide.python.org/getting-started/pull-request-lifecycle/#why-do-i-need-to-sign-the-cla-again)

[StanFromIreland]

Please remove the implementation details, and focus on user-facing changes. E.g., "speed up X on X".

[XiaohongGong]

I'v updated the NEWS.d. Could you please take another look about that? Thanks for your comment!

[XiaohongGong]

Hi, I'm an engineer from NVIDIA that has signed the CLA and PSF. What should I do to pass the CLA check?

Be careful with which email you sign the CLA (see https://devguide.python.org/getting-started/pull-request-lifecycle/#why-do-i-need-to-sign-the-cla-again)

Thanks for the comment! I confirmed that my company (NVIDIA) has signed the CLA and my email and github are all correct. Any other checks/steps that should I do to pass the CLA?

[StanFromIreland]

You'll have to sign (again) by clicking the button above, I'm afraid otherwise we can't do anything here.

[XiaohongGong]

You'll have to sign (again) by clicking the button above, I'm afraid otherwise we can't do anything here.

If I sign again by clicking the button above, it means that I will sign on behalf on the individual contributor, which may not be recommended? The contribution is made on behalf of my company. My understanding is that NVIDIA has an existing PSF Contributor Agreement. Could you please let me know whether any additional action is needed to associate this PR/GitHub account with the corporate agreement?

Note that I'v clicked the sign button above and the CLA check has passed. But I'm aware that should be a mistake and I should not do that.