Skip to content

[5/7] Implement native SEA parameter batching - #1632

Merged
sreekanth-db merged 1 commit into
mainfrom
sreekanth-db/stack/native-batch-sea
Nov 16, 2026
Merged

sreekanth-db merged 1 commit into
mainfrom
sreekanth-db/stack/native-batch-sea

Conversation

@sreekanth-db

@sreekanth-db sreekanth-db commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


Description

Add opt-in native PreparedStatement batch execution through the Statement Execution API.

  • Add the driver-owned StatementParameterSet request model.
  • Serialize ordered parameter sets as parameter_sets with zero-based ordinals.
  • Ensure singular parameters and batch parameter_sets are mutually exclusive.
  • Omit row_limit so update-count results are not truncated.
  • Reuse the existing SEA submission, polling, timeout, cancellation, and result lifecycle.
  • Reuse shared native routing, 42P02 fallback, repeat-count expansion, and result-error handling.

Native batching remains disabled by default through EnableNativeBatching=0.

Testing

  • Full DatabricksSdkClientTest: 49 passed.
  • Targeted SEA request, capability, and fallback tests passed.
  • Live serverless SEA native insertion passed with update-count and table-row verification.
  • Live DBR 18.1 SEA compatibility fallback passed; verified one native attempt followed by three individual legacy executions.
  • Temporary test tables were removed.
  • Isaac review completed; the sole ordinal concern was invalidated by the zero-based backend contract and successful live native execution.

Additional Notes to the Reviewer

The existing singular SEA path retains its current behavior. Zero-based ordinal conversion is applied only while constructing native parameter_sets.

NO_CHANGELOG=true

@github-actions

Copy link
Copy Markdown

JDBC integration tests triggered (replay, thrift + sea). View workflow runs.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@sreekanth-db sreekanth-db changed the title Implement native SEA parameter batching [5/7] Implement native SEA parameter batching Aug 11, 2026
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-sea branch from 8b0bfbb to 07f3135 Compare August 24, 2026 23:48
@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 07f3135

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@sreekanth-db sreekanth-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Sep 15, 2026
@vikrantpuppala
vikrantpuppala removed their request for review October 23, 2026 09:06
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-sea branch from 07f3135 to 89eddf8 Compare November 5, 2026 10:03
@sreekanth-db
sreekanth-db changed the base branch from main to sreekanth-db/stack/native-batch-thrift November 5, 2026 10:03

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly solid. The batch path reuses the existing SEA lifecycle and keeps parameters and parameter_sets mutually exclusive. One medium concern: the full batch payload is re-sent in every polling GET body for non-realtime sessions, and the new request-model setters have only light test coverage.

return executeStatement(
sql,
computeResource,
Collections.emptyMap(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The batch request goes into the shared executeStatement loop as suppliedRequest. In that loop, every PENDING/RUNNING poll builds new Request(GET, ..., serializeStatementRequest(typedStatementId, request)). For non-realtime sessions, StatementId.forCurrentSession returns an id with sessionId() == null, so serializeStatementRequest serializes the whole ExecuteStatementRequest again as the body of each GET poll.

For singular statements that body is small. For native batching it now holds the full parameter_sets payload, which can be thousands of rows × N parameters, and it gets re-serialized and re-sent on every poll interval. A long-running batch INSERT pays this cost on every poll, both in CPU and in bandwidth.

Suggested fix: send a GetStatementRequest (only the statement id) for polling, as getStatementResult already does, or at least don't pass the original execute request into the poll body when parameterSets != null.


public ExecuteStatementRequest setParameters(Collection<StatementParameterListItem> parameters) {
this.parameters = parameters;
if (parameters != null) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — There is no direct unit test for the new mutual-exclusion setters, or for equals/hashCode/toString now that parameterSets is included. The setters depend on call order: setParameters(emptyList) silently drops previously set parameterSets, and setParameterSets(null) leaves parameters as it was. The only coverage is the single happy-path order inside testExecuteStatementBatchBuildsSeaParameterSets.

A small ExecuteStatementRequestTest should check both orderings. It should also confirm that toString logs only parameterSetCount and never parameter values, since keeping batch values out of logs is the intent here.

(Anchored to the nearest changed line — see the description for the exact location.)

@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-sea branch from 89eddf8 to f2b870e Compare November 5, 2026 12:38

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Nit

Mostly sound, with one medium efficiency concern: in the default non-realtime mode, the full batch payload is re-sent on every status poll. Request modelling, making parameters and parameter_sets mutually exclusive, and the zero-based ordinal conversion all look correct. Nit: the debug log in executeStatement reports parameters = {} for batch executions, which could mislead anyone reading logs; consider logging the parameter-set count instead.

Other findings

  • ⚪ Nit — testExecuteStatementBatchBuildsSeaParameterSets only checks INT/STRING values on the second parameter of the first set. The new mapToParameterListItem(parameter, ordinal) overload also handles DECIMAL type-string derivation and null values (setValue(null)), and neither is checked in the batch path. It also doesn't check that the second set's ordinals restart at 0, or that wait_timeout/on_wait_timeout/session fields carry over from getRequest. Consider adding a set with a null value and a BigDecimal, and asserting on capturedSets.get(1), so the cardinal→zero-based conversion is verified per set rather than only for the first one.

mapToParameterListItem(parameter, parameter.cardinal() - 1))
.collect(Collectors.toList())))
.collect(Collectors.toList());
return request.setParameterSets(seaParameterSets).unsetRowLimit();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The batch request now holds every parameter_sets entry, and it is sent again on every status poll, not only on the initial POST. For non-realtime sessions (the default), StatementId.forCurrentSession returns an id with sessionId() == null. In that case serializeStatementRequest(typedStatementId, request) serializes the whole ExecuteStatementRequest as the body of each GET /statements/{id} poll.

For singular statements this payload is small, so the pre-existing behaviour didn't matter. A native batch is meant to carry thousands of rows, and a long-running batch (PENDING/RUNNING across many polls) will upload the full batch payload on every poll interval. That costs bandwidth and serialization CPU, and can hit request-size limits on a GET.

Suggested fix: in the polling loop, serialize a lightweight GetStatementRequest().setStatementId(statementId) as getStatementResult already does, instead of reusing request. At minimum, avoid sending the batch request for batch executions. A test that captures the poll-request bodies, checks that parameter_sets is absent, and uses a mocked RUNNING→SUCCEEDED sequence would lock this down.

gibkim pushed a commit to gibkim/databricks-jdbc that referenced this pull request Nov 5, 2026
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-jdbc/pull/1623/files) to
review incremental changes.
-
[**stack/native-batch-routing**](databricks#1623)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1623/files)]
← _this PR_
-
[stack/native-batch-thrift](databricks#1625)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1625/files/597237e946403890862ba875b54121b489cfec12..b88cd288f12efdd3126506dc37a443dfea7c1842)]
-
[stack/native-batch-sea](databricks#1632)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1632/files/b88cd288f12efdd3126506dc37a443dfea7c1842..07f31353e7ff8c94a644c74a4ace3d970dd63bac)]
-
[stack/native-batch-integration-tests](databricks#1633)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1633/files/07f31353e7ff8c94a644c74a4ace3d970dd63bac..2d0f4da302b183f15de9804af1c12346b5531bf4)]
-
[stack/native-batch-activation](databricks#1634)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1634/files/2d0f4da302b183f15de9804af1c12346b5531bf4..f4e3ff24e5d54451eec9603e429fce7549f291b8)]

---------
## Description

Add the left-side capture and routing boundary for native
prepared-statement batching.

- Snapshot ordered parameter sets when `addBatch()` is called.
- Preserve one-based JDBC indexes in the shared model; transport
adapters will convert wire ordinals.
- Use the same parameter-set model for existing legacy execution.
- Add an injectable `NativeBatchExecutor` capability boundary.
- Route native-enabled parameterized INSERTs to that boundary when
supported.

Production still supplies an unsupported native executor, so customer
execution remains on the existing legacy path.

## Testing

- Focused batch capture, routing, and PreparedStatement tests: 69
passed.
- Live serverless warehouse validation passed for individual,
parameterized rewrite, and interpolated rewrite flows.
- Isaac review completed with zero final findings.

## Additional Notes to the Reviewer

This PR does not add a Thrift/SEA native transport implementation,
result extraction, fallback, telemetry, or customer-visible behavior
changes.

NO_CHANGELOG=true

---------

Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-thrift branch from 0a108fd to e988f1a Compare November 16, 2026 10:10
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-sea branch from f2b870e to 9690269 Compare November 16, 2026 10:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall. One low concern: each status poll re-serializes the full batch request. Ordinal conversion, parameters/parameter_sets mutual exclusion, removing row_limit, and passing through the 42P02 fallback signal are all implemented correctly and covered by tests.

Collections.emptyMap(),
statementType,
session,
parentStatement,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — executeStatementBatch passes the full batch ExecuteStatementRequest into the shared executeStatement loop. While the statement is PENDING/RUNNING, that loop calls serializeStatementRequest(typedStatementId, request) on every GET status poll. For non-realtime sessions (statementId.sessionId() == null), that serializes the whole ExecuteStatementRequest again, now including every parameter_sets entry. With a single statement the payload is a handful of parameters. With a native batch it holds every row's values, so a long-running batch of thousands of rows re-serializes the full payload (possibly megabytes) on each poll. It may also send that payload as a GET body; I haven't checked whether the SDK transport drops GET bodies.

The old singular path had the same quirk, but this change makes its cost scale with batch size. A contained fix is to have the poll serialize a GetStatementRequest (as getStatementResult already does), or pass a lightweight request object, so polling cost doesn't grow with batch size.

Base automatically changed from sreekanth-db/stack/native-batch-thrift to main November 16, 2026 11:24
Submit ordered parameter_sets through the existing SEA execution lifecycle so opt-in prepared batches share the same routing, fallback, and JDBC result handling as Thrift.

Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-sea branch from 9690269 to a1c2023 Compare November 16, 2026 11:25

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@sreekanth-db
sreekanth-db added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 1b58125 Nov 16, 2026
25 checks passed
@sreekanth-db
sreekanth-db deleted the sreekanth-db/stack/native-batch-sea branch November 16, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants