[AMORO-4386][Optimizer] Support Spark 4.1 for Iceberg optimizer

[sivakumar-mahalingam]

Why are the changes needed?

The external Spark optimizer currently follows Amoro's global Spark and Scala versions. This prevents the optimizer from running with Spark 4.1 images because Spark 4.1 uses Scala 2.13 and requires a newer Java and dependency baseline.

This pull request adds Spark 4.1 support specifically for the external Spark optimizer used with native Iceberg tables. It does not enable general Spark 4.1 support for AMS, mixed-format Spark modules, or Paimon.

Closes #4386.

Brief change log

• Add optimizer-specific Spark version, Spark major version, Scala binary version, and Jackson properties.
• Add the optimizer-spark-4.1 Maven profile targeting Spark 4.1.3, Scala 2.13, and JDK 17.
• Preserve the existing Spark 3.5 optimizer build and global Spark configuration.
• Produce a distinct amoro-optimizer-spark-4.1_2.13 artifact.
• Keep Spark, Scala, and unshaded Jackson classes out of the shaded optimizer JAR.
• Update optimizer distribution assembly and Docker build logic to select the optimizer-specific artifact.
• Add a Spark 4.1 optimizer Docker matrix entry while retaining Spark 3.5 as the default image.
• Add a JDK 17 and Spark 4.1 CI job with a shaded-JAR isolation check.
• Expand the local-mode optimizer test suite to two positive and three negative cases covering:
  • Successful local Spark execution and optimizer output.
  • Factory property and native-Iceberg metadata propagation.
  • Worker-side optimizer failure propagation.
  • Missing executor-factory metadata.
  • Malformed serialized task input.
• Document the optimizer-only Spark 4.1 build command in the README.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Test Execution

Spark 4.1.3 optimizer tests on JDK 17:

.\mvnw.cmd test -pl amoro-optimizer/amoro-optimizer-spark -am '-Pjava17,optimizer-spark-4.1' '-Dtest=TestSparkOptimizerLocalMode' '-Djacoco.skip=true' -B -ntp

Result:

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Spark 3.5.8 regression tests on JDK 11:

.\mvnw.cmd clean test -pl amoro-optimizer/amoro-optimizer-spark -am '-Pjava11,spark-3.5' '-Dtest=TestSparkOptimizerLocalMode' '-Djacoco.skip=true' -B -ntp

Result:

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Native-Iceberg physical rewrite regression suite on JDK 17 with the Spark 4.1 optimizer profile:

.\mvnw.cmd test -pl amoro-format-iceberg -am '-Pjava17,optimizer-spark-4.1' '-Dtest=IcebergRewriteExecutorTest' '-Djacoco.skip=true' -B -ntp

Result:

Tests run: 54, Failures: 0, Errors: 0, Skipped: 20
BUILD SUCCESS

Additional validation:

• Inspected the shaded Spark 4.1 optimizer artifact and confirmed that it contains no Spark, Scala, or unshaded Jackson classes.
• Built the optimizer image using the official apache/spark:4.1.3 base image.
• Ran the packaged SparkOptimizer entry point through Spark 4.1.3 local mode. It initialized and reached AMS registration; registration then failed as expected because no AMS instance was running.
• Ran a native Iceberg v2 rewrite inside the Spark 4.1 image using the production SparkOptimizerExecutor and IcebergRewriteExecutorFactory. Two Parquet input files were rewritten into one output file containing both records.
• Submitted the optimizer using Spark 4.1.3 Kubernetes cluster mode on a temporary kind cluster. The driver and executor started and reached AMS registration without Spark, Scala, or dependency linkage errors.
• Ran repository formatting checks and git diff --check.

A full clean verify was attempted but stopped during Apache RAT validation because of the existing tracked test fixture:

amoro-format-mixed/amoro-mixed-hive/src/test/resources/string_is_bytes.parquet

Targeted Spark optimizer and native-Iceberg test suites completed successfully.

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? README

[sivakumar-mahalingam]

Hi @j1wonpark @zhoujinsong, this PR adds Spark 4.1 support for the Iceberg optimizer (#4386) and is ready for review. Could you please take a look when you get a chance? Thanks!

[j1wonpark]

Thanks for fixing the AMS test failures. These fixes also address the CI failures in #4378, and I’d like to review them independently. Could you please extract the AMS test fixes into a separate PR?

[sivakumar-mahalingam]

Sure, I extract it to a separate pr and update to you in this thread.

[sivakumar-mahalingam]

Hi @j1wonpark @zhoujinsong, Could you please review the pr when you get a chance? Thanks!

[xxubai]

Could we standardize on AWS SDK 2.29.52. And verify compatibility with Spark 3.5 and the other affected components.

[xxubai]

Are these fixes releated to optimizer spark 4.1?