Skip to content

[4/7] Implement native Thrift parameter batching - #1625

Merged
sreekanth-db merged 4 commits into
mainfrom
sreekanth-db/stack/native-batch-thrift
Nov 16, 2026
Merged

sreekanth-db merged 4 commits into
mainfrom
sreekanth-db/stack/native-batch-thrift

Conversation

@sreekanth-db

@sreekanth-db sreekanth-db commented Aug 10, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


Description

Add opt-in native PreparedStatement batch execution over Thrift.

  • Send ordered parameter sets through TExecuteStatementReq.batchParameters in one request.
  • Use negotiated protocol V10 for all-purpose clusters.
  • Attempt native batching for warehouses/serverless regardless of reported protocol.
  • For warehouses/serverless, fall back to the existing legacy path on exact 42P02 [UNBOUND_SQL_PARAMETER] errors.
  • Return ordered JDBC update counts, including expansion of repeat-compressed backend rows.
  • Preserve atomic native failures as BatchUpdateException with backend SQLState, vendor code, and cause.

Native batching remains disabled by default through EnableNativeBatching=0.

Testing

  • 443 focused native Thrift, PreparedStatement, result-set, protocol, routing, and telemetry tests passed.
  • Live serverless Thrift warehouse native insertion passed.
  • Live DBR 18.1 warehouse compatibility fallback passed.
  • Live DBR 19 all-purpose V10 native insertion passed.
  • Each live test verified JDBC update counts, inserted row count, and temporary-table cleanup.
  • Isaac review completed.

Additional Notes to the Reviewer

Real backend validation showed successful batch results may compress identical counts using the repeat column; the driver expands these rows back to one JDBC count per submitted parameter set.

NO_CHANGELOG=true

@github-actions

Copy link
Copy Markdown

JDBC integration tests triggered (replay, thrift + sea). View workflow runs.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-factored opt-in native Thrift batch path with solid unit coverage (routing, fallback, repeat expansion, ordinal mapping). Ordinal 0-based remap, unsetParameters/unsetResultRowLimit, the exact-42P02 legacy fallback, and 1-based column-index checks in getBatchUpdateCounts all check out. One low-severity note about EXECUTE_FAILED being reported when a successful batch returns an unexpected result shape.

Scope: reviewed as a stacked increment (7d19fed..head); the parent's changes are reviewed on the parent PR.

@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 29c3268

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Solid, well-tested increment — native Thrift batch execution is opt-in, gated correctly, and the result-count extraction (with repeat-column expansion, maxRows bypass, and cardinality validation) is defensive and covered by unit tests. No correctness or resource-lifecycle bugs found. One low-severity robustness note on the fallback detection heuristic, plus a nit on the exception message.

Scope: reviewed as a stacked increment (7d19fed..head); the parent's changes are reviewed on the parent PR.

@sreekanth-db sreekanth-db changed the title Implement native Thrift parameter batching [4/7] Implement native Thrift parameter batching Aug 11, 2026
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-thrift branch from 29c3268 to b88cd28 Compare August 24, 2026 23:48

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@sreekanth-db sreekanth-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Sep 15, 2026
@vikrantpuppala
vikrantpuppala removed their request for review October 23, 2026 09:06
}
}

long[] executeIndividually(List<BatchParameterSet> batchParameterSets)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: What's the reason for adding this method? Can we make executeIndividualStatements public and re-use it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. The executeIndividually wrapper has been removed, and executeIndividualStatements is now called directly.

} catch (NativeBatchResultException e) {
throw e;
} catch (SQLException e) {
if (isUnsupportedNativeBatching(e)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should do the 42P02 fallback for SQL warehouse only, not APC.

This is because (1) we know exactly whether an APC supports native batching based on thrift protocol version, so there's no need to check 42P02; (2) the fallback to individual execution causes latency overhead

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. The 42P02 [UNBOUND_SQL_PARAMETER] compatibility fallback is now limited to warehouses. Tests verify that warehouses fall back to legacy execution while all-purpose clusters propagate the error.

false);

assertArrayEquals(new long[] {1, 1, 1}, resultSet.getBatchUpdateCounts(3));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we also have a test case to verify that partial success is supported, like
[[num_affected_rows=1, repeat=3], [num_affected_rows=0, repeat=2], [num_affected_rows=1, repeat=4]]

If we support that in the future, it'll be nice to make this driver version compatible with it

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added coverage for multiple repeat-compressed rows with mixed affected-row counts, verifying expansion to the correctly ordered JDBC update-count array.

List<BatchParameterSet> batch = createBatch(1);
when(nativeBatchExecutor.execute(INSERT_SQL, batch))
.thenThrow(
new SQLException("[UNBOUND_SQL_PARAMETER] Native batching is unsupported", "42P02", 0));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional nit: The error message thrown in production will be

Found the unbound parameter: <name>. Please, fix `args` and provide a mapping of the parameter to either a SQL literal or collection constructor functions such as `map()`, `array()`, `struct()`."

Consider using that if you'd like the higher test fidelity.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s keep the shorter message to summarize the issue clearly without the backend-specific args details.

try {
return result.getBatchUpdateCounts(parameterSets.size());
} catch (SQLException e) {
throw new NativeBatchResultException(e);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to return EXECUTE_FAILED instead? When the code reaches this part, we know that the insert execution is successful, because waitForExecutionResult would throw otherwise

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since waitForExecutionResult has already succeeded, reporting EXECUTE_FAILED would incorrectly indicate that the inserts failed and could encourage an unsafe retry. NativeBatchResultException distinguishes this as a result-extraction failure, warns that rows may already be committed, and clears the stored batch to prevent accidental re-execution.

sql,
stackTraceMessage,
getFutureBatchResult(sql, parameterSets, StatementType.UPDATE),
true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this false to match the legacy batch behavior? 🤔

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to false to match legacy batching. On timeout, this keeps the statement open and cancels the local future, although the backend operation may still be running. We are retaining the existing behavior for consistency; native and legacy timeout handling can be revisited together.

}

@Override
public DatabricksResultSet executeStatementBatch(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add executeStatementBatch to mapMethodToOperationType to make the latency tracking (recordOperationLatency) work?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. executeStatementBatch now maps to OperationType.EXECUTE_STATEMENT, with test coverage for the latency telemetry mapping.

@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-thrift branch from b88cd28 to 106f277 Compare November 5, 2026 10:03

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 4 Low · 1 Nit

Looks good overall: routing, ordinal mapping, repeat expansion and the atomic-failure → BatchUpdateException handling are correct. The findings are all low-severity: missing tests for the repeat validation and the new error code, a silent native→legacy fallback, the native path ignoring interpolateParameters, and widened throws clauses on the public class.

Other findings

  • ⚪ Nit — DatabricksPreparedStatement is a public class, and this widens the declared exception of its executeBatch()/executeLargeBatch() from DatabricksBatchUpdateException to SQLException. Callers that hold the concrete type and only catch BatchUpdateException will stop compiling. The method can now also throw NativeBatchResultException, which is deliberately not a BatchUpdateException. That is fine under the java.sql.Statement contract, but it is a source-compatibility change and should be called out in release notes when the feature is activated.

try {
while (next()) {
long repeatCount = hasRepeatCount ? getLong(REPEAT_COUNT) : 1;
if (repeatCount < 1 || repeatCount > expectedCount - index) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — There is no test for the repeat validation branch. None of the new tests reach it: no case with repeat = 0, a negative value, or a value larger than the parameter sets still unfilled (for example, repeat=3 with expectedCount=2), and no case where an extra row arrives after index == expectedCount. This guard is what keeps a bad backend response from causing an ArrayIndexOutOfBoundsException inside the counts[index++] loop, so it deserves a direct test. One example: next() returns two rows with repeat 2 and 1 while expectedCount=2; that should throw "invalid repeat count: 1".

throw e;
} catch (SQLException e) {
if (isUnsupportedNativeBatching(e)) {
return legacyExecutor.executeBatch(batchParameterSets);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — When the backend rejects native batching with 42P02 [UNBOUND_SQL_PARAMETER], the code switches to the legacy path without logging anything. For example, a warehouse on DBR 18.1 with EnableNativeBatching=1 sends a native request that is rejected every time, then runs the legacy batch. That costs an extra round-trip on every executeBatch(), and users cannot tell from the logs that native batching never actually takes effect. Add a LOGGER.debug/info line (no secrets in it). Also consider caching the 'unsupported' result per connection so later batches skip the doomed native attempt.

if (!InsertStatementParser.isParametrizedInsert(sql)) {
return legacyExecutor.executeIndividually(batchParameterSets);
}
if (!connection.getConnectionContext().isNativeBatchingEnabled()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The native path ignores interpolateParameters (supportManyParameters=1). In the legacy executor that flag inlines values into the SQL text. Here, native batching sends every value as a bound TSparkParameter even when the user explicitly chose interpolation, and native batches have their own parameter limits (the test uses [PARAMETER_BATCH_ERROR] Too many parameters). This may be intentional, but it is a behavior difference worth either documenting or gating, e.g. skip native when interpolateParameters is true, or fall back on that error.

+ cause.getMessage(),
cause,
DatabricksDriverErrorCode.RESULT_SET_ERROR);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — This adds a new driver-emitted error (reusing RESULT_SET_ERROR), but no test checks the emitted error name and numeric vendor code. The existing assertions only compare getSQLState() to the string "RESULT_SET_ERROR". CLAUDE.md also asks for the telemetry classification to be recorded and the PR template's telemetry-error section to be filled in, and the PR description has neither.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 2 Nit

Mostly solid: the native Thrift batching is gated correctly, ordered parameter sets reach the request, and repeat-compressed counts are expanded. There is one medium issue: the warehouse compatibility fallback isn't cached, so a warehouse without native support pays a failing round trip on every batch. The two low findings are a gap in wrapping count-read errors and missing tests for the repeat-count checks.

Other findings

  • ⚪ Nit — Only SQLException from getBatchUpdateCounts is turned into NativeBatchResultException. If a RuntimeException is thrown while reading counts (e.g. a driver exception from a value conversion or the result iterator), it skips this catch and also skips the catch (NativeBatchResultException) in DatabricksPreparedStatement.executeLargeBatch. The batch is then never cleared, even though the backend has already committed the rows, so a caller retry would insert them again. That is exactly what the new exception is meant to prevent. Catching RuntimeException here as well (and wrapping it the same way) would close the gap.
  • ⚪ Nit — The repeat-count validation (repeatCount < 1 || repeatCount > expectedCount - index) has no tests. The new tests cover ordering, repeat expansion, the cardinality mismatch, and the missing column, but none sends a repeat of 0, a negative value, a null repeat value (which getLong returns as 0), or a repeat that goes past the expected count. These checks are what stop the array from overflowing or silently coming back short, so add a test for each case.

} catch (NativeBatchResultException e) {
throw e;
} catch (SQLException e) {
if (isUnsupportedNativeBatching(e)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The 42P02 [UNBOUND_SQL_PARAMETER] fallback is decided again on every executeBatch() call, and nothing remembers the result. supportsNativeParameterBatching returns true for every Warehouse no matter what protocol version it reports. So on a warehouse that doesn't support native batching (e.g. the DBR 18.1 case in the PR description), every batch sends a native request that is sure to fail before it runs the legacy path. That adds a wasted round trip to every batch for the life of the connection.

Suggested fix: once the compatibility error is seen, cache it on the client or connection (for example a nativeBatchingUnsupported flag that isSupported() checks), so later batches go straight to the legacy executor.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can take up this performance optimization in a follow-up PR.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine to handle this in a follow-up PR. The current code still sends a native request on every executeBatch() and only drops to legacy after it fails, so nothing changes yet. I'm keeping this thread open as the reminder for that work. A small fix would be to set a nativeBatchingUnsupported flag on the connection or client the first time the error shows up, and have isSupported() check it. If it helps, link the tracking issue here when you file it.

gibkim pushed a commit to gibkim/databricks-jdbc that referenced this pull request Nov 5, 2026
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-jdbc/pull/1623/files) to
review incremental changes.
-
[**stack/native-batch-routing**](databricks#1623)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1623/files)]
← _this PR_
-
[stack/native-batch-thrift](databricks#1625)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1625/files/597237e946403890862ba875b54121b489cfec12..b88cd288f12efdd3126506dc37a443dfea7c1842)]
-
[stack/native-batch-sea](databricks#1632)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1632/files/b88cd288f12efdd3126506dc37a443dfea7c1842..07f31353e7ff8c94a644c74a4ace3d970dd63bac)]
-
[stack/native-batch-integration-tests](databricks#1633)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1633/files/07f31353e7ff8c94a644c74a4ace3d970dd63bac..2d0f4da302b183f15de9804af1c12346b5531bf4)]
-
[stack/native-batch-activation](databricks#1634)
[[Files
changed](https://github.com/databricks/databricks-jdbc/pull/1634/files/2d0f4da302b183f15de9804af1c12346b5531bf4..f4e3ff24e5d54451eec9603e429fce7549f291b8)]

---------
## Description

Add the left-side capture and routing boundary for native
prepared-statement batching.

- Snapshot ordered parameter sets when `addBatch()` is called.
- Preserve one-based JDBC indexes in the shared model; transport
adapters will convert wire ordinals.
- Use the same parameter-set model for existing legacy execution.
- Add an injectable `NativeBatchExecutor` capability boundary.
- Route native-enabled parameterized INSERTs to that boundary when
supported.

Production still supplies an unsupported native executor, so customer
execution remains on the existing legacy path.

## Testing

- Focused batch capture, routing, and PreparedStatement tests: 69
passed.
- Live serverless warehouse validation passed for individual,
parameterized rewrite, and interpolated rewrite flows.
- Isaac review completed with zero final findings.

## Additional Notes to the Reviewer

This PR does not add a Thrift/SEA native transport implementation,
result extraction, fallback, telemetry, or customer-visible behavior
changes.

NO_CHANGELOG=true

---------

Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
@sreekanth-db
sreekanth-db added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
Route eligible prepared batches through compute-aware Thrift batchParameters, preserve legacy compatibility fallback, and map repeat-compressed backend results to ordered JDBC update counts.

Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
Surface post-success count-read errors as RESULT_SET_ERROR and clear committed batches before propagating so retries cannot duplicate inserted rows.

Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
@sreekanth-db
sreekanth-db force-pushed the sreekanth-db/stack/native-batch-thrift branch from 0a108fd to e988f1a Compare November 16, 2026 10:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall, with two low-severity concerns. The routing, the zero-based batch ordinals, the expansion of repeat rows into per-set counts and the error mapping all look right.

} catch (NativeBatchResultException e) {
throw e;
} catch (SQLException e) {
if (isUnsupportedNativeBatching(e)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — If a warehouse rejects native batching with 42P02 [UNBOUND_SQL_PARAMETER], the driver falls back, but it doesn't remember that. Every later executeBatch() on the same connection sends the native request again and only then runs the legacy path. On older warehouses (e.g. the DBR 18.1 case in the description) that means every batch pays an extra failed server round-trip. Each attempt also builds a server-error SQLException, which goes through the driver's ERROR-level failure telemetry. Since this unsupported-feature result won't change for the session's compute, consider caching it (for example, a flag on the connection or client that isSupported() checks) so the probe runs once per connection.

class NativeBatchResultException extends DatabricksSQLException {

NativeBatchResultException(SQLException cause) {
super(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — This PR adds a new driver-emitted error, but the tests can't tell its error code apart from the cause's. Both tests (DatabricksPreparedStatementTest.testExecuteBatchThrowsResultErrorWhenNativeCountsCannotBeRead and PreparedStatementBatchExecutorTest.resultExtractionFailureThrowsDedicatedException) build the cause with SQLState "RESULT_SET_ERROR". So assertEquals("RESULT_SET_ERROR", exception.getSQLState()) would still pass if the wrapper just passed the cause's SQLState through. Also, the (reason, cause, DatabricksDriverErrorCode) constructor sets the vendor code from the cause rather than from RESULT_SET_ERROR.getCode(), and nothing tests the numeric code. In the real path the error is reported twice to telemetry: once by the inner DatabricksSQLException in getBatchUpdateCounts and again by this wrapper. Please give the cause a different SQLState in the test and assert the error name and code. Please also record the telemetry classification as CLAUDE.md requires.

@sreekanth-db
sreekanth-db added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 4a784fa Nov 16, 2026
39 of 40 checks passed
@sreekanth-db
sreekanth-db deleted the sreekanth-db/stack/native-batch-thrift branch November 16, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants