Conversation
Read small-precision Decimal128 values from native long words, retaining the full-width Arrow conversion when the integer cannot fit in a long. Handle both native word orders and use the source scale when constructing BigDecimal, leaving requested rescaling and overflow checks to Spark. Preserve Spark's expanded Decimal representation for checked integer casts. Add tests for decimal values, rescaling, malformed buffers, both word orders, slices, object independence, negative scales and checked casts. Validated on Spark 3.5.9/Scala 2.12 and Spark 4.1.2/Scala 2.13, with 26 targeted tests per version plus Javadoc and test formatting checks. AI-assisted implementation and tests. Signed-off-by: Peifeng Li <lipeifeng@xiaohongshu.com>
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | dict_canonicalize_gt_u8_avx2[16000000] |
8 ms | 10.1 ms | -21.32% |
| ❌ | WallTime | words_gather_scalar_avx2[65536] |
8.3 µs | 9.4 µs | -12.06% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 13.4 µs | +52.66% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing xiaoh1024:exp/decimal-accessor-pr (f3a1eb6) with develop (d1fe2dc)
Footnotes
-
218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
2 participants
Summary
Closes #9837. See the issue for the motivation, design, benchmark results, and decimal-specific validation.
Changes
SmallDecimalAccessorfor source precision 1–18, preserving full-width fallback and Spark's decimal conversion semantics.AI assistance: AI assistance was used for implementation, tests, validation tooling, and this description, including translation from Chinese.