Skip to content

fix(array): scale decimal-to-float scalar casts across the whole scale domain - #9883

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/decimal-scalar-scale-factor
Open

jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/decimal-scalar-scale-factor

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

DecimalScalar::cast computed the scale factor for the float target as
10_i128.pow(self.decimal_type.scale() as u32)
(vortex-array/src/scalar/typed_view/decimal/scalar.rs:98). DecimalDType::try_new bounds scale
above at MAX_SCALE (76) and not at all below, so that expression is wrong at both ends. Measured
in release:

input produced correct
decimal(76,39) storing 1f64 -4.796830079047167e-38 1e-39
decimal(5,-5) storing 1f64 inf 100000.0

Above scale 38 the factor overflows i128 and wraps. A negative scale reaches pow as a wrapped
exponent, since -5i8 as u32 is 4294967291 — and semantically a negative scale should scale the
value up, not divide by anything.

Three places in the same crate already get this right: DecimalValue::rescale_i256
(dvalue.rs:127) widens the scale to i16 before branching on its sign and takes the factor
through the fallible decimal_scale_factor, and the array kernel
arrays::decimal::compute::cast::cast_to_f64 (cast.rs:196) takes it as 10f64.powi(-scale).
Only the scalar float branch did not.

The extreme scales are already pinned for the integer target, on both sides:
cast_decimal_to_integer_policy (cast.rs:508) has cases at scale -2, 76 and -128 and
asserts the array and scalar casts agree, and cast_decimal_to_integer_wide_storage (:691) does
the same for decimal(76,40). That target goes through DecimalToIntegerCast, which is fine. The
float target had no equivalent.

Fix

Take the factor in f64 — the widest the -128..=76 domain needs is 10^128, well inside range —
and branch on the sign.

Division is kept for non-negative scale rather than multiplying by the inverse, so every scale up
to 22, where 10^scale is exactly representable, returns the same bits as before. Multiplying by
the inverse instead would shift 12345 / 100 by an ulp.

The i128 funnel above (to_i128, which errors for an i256 value wider than i128 where the
array kernel yields a float) is a third scalar/array divergence, left alone here: turning that
Err into an Ok is a semantics change, not a bug fix. The neighbouring TODO(connor) still
applies to the remaining as casts.

Tests

cargo test --release -p vortex-array --no-fail-fast: 3535 + 73 + 1 passed, 3526 + 73 + 1 before.
Two arrays::listview - should panic tests fail identically before and after — they rely on
debug_assert, which is off in release.

Restoring the old expression fails all five scale-domain cases and the f32 case while leaving the
three exactness cases green, which is what shows the common path is untouched.

Scale 23..=38 can shift by up to ~2 ulp, since powi accumulates rounding where the old exact
i128 factor did not. No DecimalDType in the tree uses a scale in that band: the literal scales
across the workspace are {-128, -67, -19, -11, -10, -8, -5, -4, -2, 0..6, 10, 20, 40}.

AI assistance

Written with agentic AI assistance; I reproduced both wrong values before changing anything, and
the first version of the fix (multiply by the inverse, matching the array kernel exactly) was
rejected by its own exactness test.

…e domain

`DecimalScalar::cast` computed the scale factor as
`10_i128.pow(self.decimal_type.scale() as u32)`. `DecimalDType` admits any
`i8` scale up to `MAX_SCALE` (76), so that expression is wrong at both ends:

- a scale above 38 overflows `i128`, which in release wraps to a garbage
  factor — `decimal(76,39)` storing 1 cast to `-4.8e-38` instead of `1e-39`;
- a negative scale reaches `pow` as a wrapped `u32` exponent, because `-5i8
  as u32` is 4294967291 — `decimal(5,-5)` storing 1 cast to `inf` instead of
  `100000.0`. A negative scale should scale the value up, not divide it.

Take the factor in `f64` and branch on the sign, the way the array kernel
`arrays::decimal::compute::cast::cast_to_f64` already does. The widest
factor the domain needs is `10^128`, well inside `f64` range.

Division is kept for non-negative scale rather than multiplying by the
inverse, so every scale up to 22 — where `10^scale` is exactly
representable — returns the same bits as before.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant