feat(core)!: bump substrait to v0.105.0

[nielspardon]

Bumps the substrait-packaging artifacts to v0.105.0 and implements its main feature, detached expression subtrees (substrait-io/substrait#1218), in both directions. Protobuf parsers reject messages nested more than 100 levels below the root (protobuf-java, C++ and upb alike), and every nested function call costs three levels, so a plan with roughly 30 levels of nested expressions could be serialized by substrait-java but not parsed back, even by substrait-java.

• Reading: ProtoExpressionConverter resolves detached_expression_ordinal by converting the entry in place, so the entry binds to the input schema and correlation scope of the referencing site. DetachedSubtrees enforces the spec's rules on the reachable portion of the table (in range, referenced at most once, acyclic, entry not itself an ordinal) and rejects violations with IllegalArgumentException. Unreferenced entries are neither converted nor validated. ProtoPlanConverter / ProtoExtendedExpressionConverter supply each container's table, and the new ProtoRelConverter.from(PlanRel) does the same for a single PlanRel (and refuses a bare-relation PlanRel with an accurate message). An ordinal converted outside a container is rejected.
• Writing: NestingDetacher runs on each PlanRel and ExtendedExpression after POJO → proto. A container that fits within the default limit of 100 is returned unchanged, so only plans that default parsers would reject change. Otherwise an expression that doesn't fit is detached once it lies deeper than half the limit, and its subtree is processed again from its new position directly below the container. Ordinals are assigned in pre-order. The detacher is on by default; new PlanProtoConverter / ExtendedExpressionProtoConverter constructors take NestingDetacher.withMaxDepth(n) or disabled(). The fit check scans the serialized bytes rather than walking the tree reflectively; that walk had made toProto about 8x slower on TPC-DS plans, while the scan costs about 1.4x of toProto + toByteArray.
• avg over a decimal now returns DECIMAL?<38,S> (substrait-io/substrait#1253). Isthmus already emitted a nullable average, so only a stale test comment changes.
• File-level description in extension YAMLs (substrait-io/substrait#1257) is ignored, as the parser already did for unknown top-level keys.

The API is named and shaped for both detached tables, so detached relations (substrait-io/substrait#1236) can land without another breaking change. The read side inlines resolved entries and keeps no table. An ordinal inside a google.protobuf.Any payload that a custom converter keeps without interpreting would lose its target. The default converters keep no such payloads.

The design was aligned with the substrait-python implementation (substrait-io/substrait-python#317): same depth counting, cut rule and ordinal order. That one is opt-in, because Python has no POJO → proto step to hook.

Closes #1352

BREAKING CHANGE: avg over a decimal resolves to a nullable DECIMAL?<38,S>, and plans or extended expressions whose expressions nest deeper than 100 protobuf levels are now serialized with detached expression subtrees, which consumers older than spec v0.105.0 cannot read.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: substrait-io/substrait-java/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 4974cf16-39d1-4b09-8b12-508eb1ef2ffb

📥 Commits

Reviewing files that changed from the base of the PR and between 52cd094 and 1388df6.

📒 Files selected for processing (9)

• core/src/main/java/io/substrait/expression/proto/DetachedSubtrees.java
• core/src/main/java/io/substrait/expression/proto/NestingDetacher.java
• core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java
• core/src/main/java/io/substrait/extendedexpression/ExtendedExpressionProtoConverter.java
• core/src/main/java/io/substrait/extendedexpression/ProtoExtendedExpressionConverter.java
• core/src/main/java/io/substrait/plan/PlanProtoConverter.java
• core/src/main/java/io/substrait/plan/ProtoPlanConverter.java
• core/src/main/java/io/substrait/relation/ProtoRelConverter.java
• core/src/test/java/io/substrait/type/proto/DetachedExpressionRoundtripTest.java

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

📝 Summary

Summary by CodeRabbit

• New Features
  • Deep expressions in plans and extended expressions are now automatically split into detached subtrees, helping them stay within protobuf nesting limits.
  • Added support for resolving detached expressions when reading plans and extended expressions, including validation of invalid references.
  • Added options to customize or disable automatic detachment. Existing conversion behavior uses a default nesting limit.

The PR adds protobuf expression detachment for deeply nested plan and extended-expression containers, resolves detached ordinals during input conversion, and updates the packaging version to Substrait 0.105.0.

Changes: Detached expression support

Layer / File(s)	Summary
Rewrite deeply nested expressions core/src/main/java/io/substrait/expression/proto/NestingDetacher.java	NestingDetacher checks protobuf message depth and rewrites eligible expression subtrees as ordinal references in PlanRel and ExtendedExpression containers.
Detach expressions on output core/src/main/java/io/substrait/plan/PlanProtoConverter.java, core/src/main/java/io/substrait/extendedexpression/ExtendedExpressionProtoConverter.java, core/src/test/java/io/substrait/type/proto/DetachedExpressionRoundtripTest.java	Plan and extended-expression output converters invoke NestingDetacher. Tests cover shallow containers, deep round trips, and unreferenced detached entries.
Resolve detached ordinals on input core/src/main/java/io/substrait/expression/proto/DetachedSubtrees.java, core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java, core/src/main/java/io/substrait/relation/ProtoRelConverter.java, core/src/main/java/io/substrait/extendedexpression/ProtoExtendedExpressionConverter.java, core/src/main/java/io/substrait/plan/ProtoPlanConverter.java, core/src/test/java/io/substrait/type/proto/DetachedExpressionRoundtripTest.java	Input conversion resolves ordinals against the detached-expression table in the active container. Tests cover invalid references, root validation, and expression round trips.

Substrait 0.105.0 alignment

Layer / File(s)	Summary
Update release version and test comment gradle/libs.versions.toml, isthmus/src/test/java/io/substrait/isthmus/SubstraitTypeSystemTest.java	The packaging version changes from 0.104.0 to 0.105.0. A test comment now describes the decimal average as nullable.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant PlanProtoConverter
  participant NestingDetacher
  participant ProtoPlanConverter
  participant ProtoRelConverter
  participant ProtoExpressionConverter
  participant DetachedSubtrees
  PlanProtoConverter->>NestingDetacher: detach each root PlanRel
  NestingDetacher-->>PlanProtoConverter: return rewritten PlanRel
  ProtoPlanConverter->>ProtoRelConverter: convert PlanRel
  ProtoRelConverter->>ProtoExpressionConverter: convert root with active detached-subtree resolver
  ProtoExpressionConverter->>DetachedSubtrees: resolve detached-expression ordinal
  DetachedSubtrees-->>ProtoExpressionConverter: convert referenced expression
  ProtoRelConverter-->>ProtoPlanConverter: return Plan.Root

Loading

Merge Risk: ⚪ Minimal · up to 1388d

This change adds support for detached expression subtrees and updates the Substrait version. No concrete merge-blocking issue was identified. Serialized expressions deeper than 100 protobuf levels will use detached subtrees, which consumers older than v0.105.0 cannot read. The PR documents this, and detachment can be disabled.

Pre-merge checks · 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 92 functions across 12 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title is concise, uses valid Conventional Commit syntax, and accurately identifies the v0.105.0 core update. It does not mention detached expression support, but it remains clearly related to the …
Description check		The description explains the rationale, reading and writing behavior, compatibility impact, related changes, performance considerations, and breaking changes. It provides the required release context …
Linked Issues check		The PR satisfies the coding requirements in directly linked issue #1352. gradle/libs.versions.toml updates substrait-packaging to 0.105.0. DetachedSubtrees resolves and validates reachable det…
Out of Scope Changes check		The changed files remain connected to issue #1352. Converter changes implement detached-expression reading and writing. Tests verify the new behavior and validation rules. The version bump and decimal…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}