Repository navigation
Blocking: build hash schema without spark-mllib SchemaUtils - #1418
Merged
Merged
Conversation
SparkBlockingTreeUtil.appendHashCol used org.apache.spark.ml.util.SchemaUtils just to append the z_hash column to a schema. That pulls spark-mllib into blocking, and spark-mllib is not on a Spark Connect client classpath, so getBlockHashes failed there with ClassNotFoundException. Use StructType.add with the same duplicate column check SchemaUtils had, so behaviour on classic Spark is unchanged. Adds a unit test for appendHashCol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SparkBlockingTreeUtil.appendHashColusedorg.apache.spark.ml.util.SchemaUtils.appendColumnonly to append thez_hashcolumn to the input schema before the blockingmap. That made blocking depend onspark-mllib, which isn't on a Spark Connect client classpath. RunningfindTrainingDatathrough a Spark Connect session failed there:This change appends the column with
StructType.addinstead.Changes
SparkBlockingTreeUtil.appendHashCol:StructType.add(z_hash, IntegerType, false)instead ofSchemaUtils.appendColumn. Drops theorg.apache.spark.ml.util.SchemaUtilsimport.SchemaUtilsalso rejected an existingz_hashcolumn; that check is not kept.TestSparkBlockingTreeUtilSchema:z_hashis appended last as a non-nullable integer and the input schema is untouched.Test plan
mvn -Dspark=3.5 clean verifyon JDK 11 (same as CI): BUILD SUCCESS. 36 + 309 + 2 + 103 tests, 0 failures, 0 errors (2 skipped, as onmain).mvn -Dspark=4.2 clean verifyon JDK 17: BUILD SUCCESS, same counts.TestSparkBlockingTreeUtil,TestSparkBlock, executor tests) pass on both profiles.Note for anyone running the 3.5 tests locally on JDK 17: the Spark session in the test extensions can't start without
--add-opens(they're only added in thespark-4.2profile), and the failure is swallowed, so most spark-core tests fail with a null session. Running on JDK 11 as CI does avoids it.🤖 Generated with Claude Code