Repository navigation
fix: retry ExecuteStatement when the connection fails before the request is sent - #598
Conversation
|
@bot |
|
Pushed Started from b798b84. Done: ExecuteStatement requests in |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
|
@bot Please revert the redirect-handling change: restore the existing redirect behavior for The cross-driver check found that the kernel and JDBC SEA also follow redirects and retry connection failures without a redirect-specific guard. We have no evidence that Databricks Thrift executes a statement and then redirects, so let's treat this as optional shared hardening and handle it separately. This comment was generated with GitHub MCP. |
|
Pushed Started from 6a3e4a7. I reverted the redirect change and kept the connection-failure retry. |
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
…est is sent ExecuteStatement was never retried on a transport failure. Retrying it after the request may have reached the server can run a statement twice, and Node cannot tell a connection closed after delivery apart from a stale keep-alive socket. So retry it only on connect-phase errors (ECONNREFUSED, ENOTFOUND, EAI_AGAIN), where no request bytes were written, and surface the original error when retries run out. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
99e3c57 to
35626ad
Compare
There was a problem hiding this comment.
Verdict: 1 Nit
Looks good, no blocking issues. I checked UnsentRequestRetryPolicy against HttpRetryPolicy. Status-code responses are never retried. Only ECONNREFUSED/ENOTFOUND/EAI_AGAIN use the shared attempt and timeout budget. A RetryError from exhausted retries is unwrapped only when its payload is one of those connection errors, so ExecuteStatement callers get the same error type they get today and no RetryError with a non-Request/Response payload leaks out. Unit tests cover retry, exhaustion, timeout, no retry after the request was sent, and no regression for OpenSession or idempotent methods.
Other findings
- ⚪ Nit —
getClosedPort()inThriftHttpConnectionRetry.test.tsfrees an ephemeral port and later connects to it to getECONNREFUSED. Another process could take that port in between, which would make the refusal tests flaky. The window is small. A port that is reliably closed (for example, keeping a listener bound and callingdestroy()on each accepted socket isn't equivalent) or acreateConnectionthat emits a syntheticECONNREFUSEDerror would remove the race.
…yPolicy The policy now retries a non-idempotent request whenever the server didn't run it: a failure to connect, a 429, or a 503 with Retry-After. That's more than an unsent request, so name it for the requests it serves. It isn't exported and hasn't been released (#598), so no alias is needed. No behavior change. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Thrift
ExecuteStatementis excluded fromretryableThriftMethods, so any transport failure fails the query on the first attempt. Retrying it unconditionally (as the conformance spec and the reference ADBC PR do) can run an INSERT/MERGE/DDL twice. I reproduced it locally on Node 24 with the driver's own node-fetch and keep-alive agent. A connection that is closed after the peer read the full request (what the conformance proxy'sclose_connectiondoes) and a stale keep-alive socket whose request was never read look exactly the same to the client:ECONNRESET"socket hang up", on a connected, reused socket, with the request fully written. So Node can't prove those requests were never delivered.This PR retries
ExecuteStatementonly when the connection fails before it is established, which is the case where we can prove nothing was sent:ECONNREFUSED,ENOTFOUND,EAI_AGAIN. It uses the existingretryMaxAttempts/retriesTimeout/ backoff settings. HTTP status codes and failures on a connected socket are still not retried. When retries run out, the original connection error is thrown instead of aRetryError, so callers keep the error they get today. Other methods are unchanged.Verification:
OpenSessionis still not retried. Idempotent methods still retry. Full unit suite: 1548 passing. Lint, prettier, type-check and build are clean.origin/mainfails in 1 ms withECONNREFUSED. This branch retries after the 2 s backoff and returns the row.ECONNRESETafter one attempt.useKernel: trueSELECT 1 works.NullRetryPolicy, about +0.2 µs per ExecuteStatement (median 88 ns vs 292 ns, 10 runs each). Loopback ExecuteStatement round-trip median: 190.5 µs (main) vs 197.7 µs (branch), within that benchmark's noise. Live ExecuteStatement median: 106.5 ms (main, 5 runs) vs 107.7 ms (branch, 6 runs).Not addressed here:
[sea]: the kernel not retrying a connection close on ExecuteStatement is indatabricks/databricks-sql-kernel(databricks/databricks-sql-kernel#185), not in this repo.Addresses #458
This pull request and its description were written by Isaac.