Skip to content

[AURON #2576] Support arrays_overlap natively - #2582

Open
Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:Auron-2576-support-arrays-overlap-natively
Open

Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:Auron-2576-support-arrays-overlap-natively

Conversation

@Sigma-Ma

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2576

Rationale for this change

Spark's arrays_overlap expression currently relies on expression fallback in Auron.

What changes are included in this PR?

  • Convert ArraysOverlap through the existing extension-function path.
  • Implement native overlap checks for supported atomic element types.
  • Preserve Spark's null and empty-array behavior, NaN equality, and distinct signed zeros.
  • Add Rust coverage for scalars, broadcasting, slices, empty batches, and invalid arguments.
  • Add Spark compatibility tests that verify native execution and complex-element fallback.

Are there any user-facing changes?

Yes. arrays_overlap runs natively for supported atomic element types. Unsupported element types continue using Spark fallback.

How was this patch tested?

  • ./dev/reformat --check.
  • cargo test --profile pre --locked -p datafusion-ext-functions.
  • ./build/mvn -Ppre -Pspark-3.5 -Pscala-2.12 \-pl spark-extension-shims-spark test -DskipBuildNative -Dsuites=org.apache.auron.AuronFunctionSuite.
  • ./build/mvn -Ppre -Pspark-4.0 -Pscala-2.13 -pl spark-extension-shims-spark test -DskipBuildNative -Dsuites=org.apache.auron.AuronFunctionSuite.

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: OpenAI Codex (GPT-6)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

Signed-off-by: Ma Zhengxuan <1319614897@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support arrays_overlap natively

1 participant