Repository navigation
fix: retry Thrift requests on the same statuses and network errors as the kernel - #605
vuanhphung wants to merge 15 commits into
Conversation
There was a problem hiding this comment.
Verdict: 1 Low
Looks good: one low concern about changelog wording. The retry classification is right: 429 and 503+Retry-After retry on ExecuteStatement, the ambiguous statuses don't, and OpenSession now uses the idempotent policy. Tests cover each case at the policy level and in the end-to-end connection tests. Follow-up work outside this PR: the existing Retry-After seconds-vs-milliseconds mismatch in getRetryAfterHeader, which the description already mentions.
2cbb8c0 to
79ae23e
Compare
There was a problem hiding this comment.
Verdict: 1 Nit
Looks good, nothing above nit. I traced the routing in ThriftHttpConnection.getRetryPolicy, the shared planRetry budget and how BaseCommand turns RetryError into HiveDriverError. They match the PR description and the changelog's note on the behavior change, and the unit and connection-level tests cover the new status and Retry-After paths.
Other findings
- ⚪ Nit —
NonIdempotentRetryPolicy.isRetryabledecides onresponse.headers.has('Retry-After'), so a 503 with an empty or unparsableRetry-Afterstill counts as "not run" and is retried with exponential backoff. That is probably what you want (the header being present is what signals intent), but a test case pinning it down would document the choice.
|
Re the nit on |
…ment Thrift OpenSession was never retried, so a single 503, 429, 502, 504 or 408 failed the connection, and ExecuteStatement failed on a 503 with Retry-After, which the server returns without running the statement. The kernel backend retries both. OpenSession joins the idempotent methods, HttpRetryPolicy retries 408, and ExecuteStatement is also retried on a 503 with Retry-After. So that a wrong host or an untrusted certificate on OpenSession isn't retried for the whole budget and doesn't lose its cause, the shared classifier no longer retries certificate verification failures or ENOTFOUND, as in the kernel. After other network errors the retry-exhaustion message names the error code, and a CloudFetch error thrown without retrying no longer carries the presigned URL. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
979bb18 to
a9a6052
Compare
There was a problem hiding this comment.
Verdict: 2 Medium · 1 Low · 1 Nit
Needs changes. The diff and the PR description disagree: there is no OpenSessionRetryPolicy, no 429 retry on ExecuteStatement, no rename and no Retry-After unit fix. OpenSession retries network errors, which the description says it must not do. The description, CHANGELOG and code should be brought into line before merge.
Other findings
- ⚪ Nit — The PR description says
ExecuteStatement"also retries 429 and 503 withRetry-After, the kernel's non-idempotent status rule" and thatUnsentRequestRetryPolicyis renamedNonIdempotentRetryPolicy. HereisRetryableaccepts only503+Retry-After, the class keeps its old name, and the newThriftHttpConnectionRetryandUnsentRequestRetryPolicytests assert that 429 is not retried. The CHANGELOG entry also leaves out 429. Decide which behavior is intended and align the code, tests, CHANGELOG and description. As it stands, the ERRORRECOV-009/010 (429) results in the description can't come from this diff.
There was a problem hiding this comment.
Verdict: 1 Nit
Looks good. I found no correctness issues. The shared status and connect-error rules, the Retry-After change from seconds to milliseconds (and its check against retriesTimeout), the request-timeout unwrapping, and the feature-flag deadline (AbortSignal.timeout is fine on the supported Node ≥22) all hold up when traced through invokeWithRetry, UnsentRequestRetryPolicy and describeDownloadError. One follow-up that this PR didn't introduce but now matters more since more statuses are retried: ThriftHttpConnection doesn't drain the body of a non-200 response before retrying, so keep-alive sockets can be held a little longer.
Other findings
- ⚪ Nit — In
HttpRetryPolicy.test.ts, the second loop under 'should check if HTTP transaction is safe to retry' (statuses 402, 406, …, 501) tests nothing the full 1–999 sweep above it doesn't already cover. You could drop it, or keep it as documentation and label it that way.
|
On the review of 26abeb4:
|
…yPolicy The policy now retries a non-idempotent request whenever the server didn't run it: a failure to connect, a 429, or a 503 with Retry-After. That's more than an unsent request, so name it for the requests it serves. It isn't exported and hasn't been released (#598), so no alias is needed. No behavior change. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
OpenSession, ExecuteStatement on a 429 or a 503 with Retry-After, and idempotent RPCs on statuses main didn't retry (408, 501, other 4xx) failed with THTTPException, which carries statusCode, the response and the server's detail, or, for OpenSession, with the network error. Once their new retries ran out they failed with a HiveDriverError that has none of these. Throw the last attempt's error instead, as the ExecuteStatement connect-failure retry already does. Retries that idempotent RPCs made before (429, 5xx except 501, network errors) keep failing with the HiveDriverError. BaseCommand no longer needs to name the network error code, which only mattered for OpenSession, so it is back as on main. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Node names 28 OpenSSL certificate verification errors and reports every other one with the code UNSPECIFIED (ncrypto's X509Pointer::ErrorCode), for example a CA without keyCertSign, an unhandled critical extension or a SHA-1 signature. The certificate list missed it, so OpenSession retried such a failure for the whole default budget (about 30 s), and CloudFetch downloads retried it too. Treat it as a certificate failure, which the kernel never retries either. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
…t one Rewrite the #605 entry as short sentences on what is now retried, what no longer is, Retry-After, the feature-flag cap and the error once retries run out, which is now the original error for calls that weren't retried before. Merge #458's ExecuteStatement connect-failure entry, which #605 broadens to any failure to connect and narrows to drop ENOTFOUND. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern: the CHANGELOG entry doesn't mention that a public error type changes. I traced the status and connect-error classification, the Retry-After handling and its budget check, which error is thrown once retries run out (THTTPException, RetryError or the network error), and the feature-flag fetch deadline. All match the tests and the description; AbortSignal.timeout is available on every Node version the package supports (≥22).
Idempotent RPCs used to retry these and then fail with HiveDriverError; now they fail at once with the node-fetch FetchError. Say so, and limit the HiveDriverError sentence to the network errors still retried. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
main retried a status below 100 on idempotent RPCs, as it did a 429 and a 5xx other than 501, and ended in HiveDriverError. Include it in the statuses whose retry exhaustion keeps the RetryError, so that row doesn't change either. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Six short sentences: what is now retried, ExecuteStatement, what is no longer retried, Retry-After and the feature-flag cap, and the error once retries run out. Drop the previously-was asides, the CloudFetch message format (#471 documents it) and the retriesTimeout guard on Retry-After. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
The last sentence said newly retried statuses keep their old error, which isn't true for CloudFetch: a 501 there used to be a plain Error and now ends as RetryError. Name CloudFetch first so the claim covers only Thrift calls. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
There was a problem hiding this comment.
Verdict: 2 Nit
Looks good, no blocking issues; I traced the retry classification, the unwrapping of exhausted retries and the feature-flag deadline, and found only two nits. Possible follow-up: isConnectError treats any FetchError whose message ends in reason: as a failure to connect; that rests on how Node reports an AggregateError, so it's worth checking against future Node versions.
Other findings
- ⚪ Nit —
CloudFetchResultHandler.describeDownloadErrornow also rewrites aFetchErrorthrown without retrying, such as a certificate failure, toCloudFetch result download failed: network error <code>. That drops node-fetch's readable reason (e.g.unable to verify the first certificate). Dropping it keeps the presigned URL out of the message, so the trade-off is reasonable. The CHANGELOG entry still says these failures "fail at once with node-fetch'sFetchError" without mentioning the new message, and a sentence there would help callers who match on the message. - ⚪ Nit — In
FeatureFlagCache.fetchWithRetry, the 10 sFETCH_DEADLINE_MSsignal now limits the whole fetch, retries included, so the per-attempttimeout: 10000infetchFeatureFlagscan never fire first. It is redundant, and a comment or removing it would make the deadline easier to follow.
|
On the review of 271d2d5 (2 nits), both left as they are:
|
Problem
On the Thrift backend a single transient HTTP error fails the connection or the statement, while the kernel (SEA) backend retries it:
OpenSessionis never retried,ExecuteStatementfails on a 429 or a 503 withRetry-After(the server didn't run it), 408 isn't retried, andRetry-After's value has no effect. More generally, Thrift and the kernel retry different sets of statuses and network errors.Found by the cross-driver specs ERRORRECOV-001, 007–012 and SESSION-013: live runs on
mainthrough the test proxy show 1 Thrift attempt in every case, while the SEA leg (and the C#, Go and Python Thrift drivers) retry and pass.Change
Which requests are retried now matches the kernel per request class (
src/client/retry.rsRetryStrategy, plus its CloudFetch chunk retry), except for the differences listed at the end, in one shared classification inHttpRetryPolicy.ts(isRetryableStatus,isConnectError,PERMANENT_NETWORK_ERROR_CODES). Node's own backoff, jitter and budget stay. Rule, per the user: add the retries Node is missing, undo the differences this PR introduced, and keep pre-existing Node retries the kernel doesn't do, except certificate failures andENOTFOUND, which the user chose to stop.CreateSession,CloseSession,GetStatementStatus,CancelStatement,CloseStatement,ExecuteMetadataQuery(retry.rs:93-98)OpenSession(added) and the otherretryableThriftMethods(ThriftHttpConnection.ts:79)ExecuteStatementExecuteStatement(NonIdempotentRequestRetryPolicy, renamed from #598'sUnsentRequestRetryPolicy, which isn't exported or released)CloudFetchDownload+ chunk retry (streaming_provider.rs:1116-1176)CloudFetchResultHandler(getRetryPolicy('cloudFetch'))FeatureFlagCache.fetchWithRetry)GetResultChunks(by chunk index)FetchResultsadvances the cursor, so it stays unretried; delegation-token RPCs unretriedRetried? (
main→ this PR; ✓ = same as the kernel)Retry-AfterEAI_AGAINEHOSTUNREACH,ENETUNREACH(one address, or every address of a host with several), socket closed during the TLS handshakeEPIPE, socket hang up after sendingretriesTimeoutleaves time, so not with the defaults; the error staysRequest timed out)OpenSession, whichmaindidn't retry at all)UNSPECIFIED, Node's code for every X509 error it doesn't name)ENOTFOUNDconnect()isn't stretchedRetry-Afterchanges whether a retry happens only for a 503 onExecuteStatement, on both sides. When present, the wait is the header's value in seconds, clamped to[retryDelayMin, retryDelayMax], and a wait that would end afterretriesTimeoutisn't started, as the kernel does.Once the retries this PR adds run out, a Thrift call fails with the error
maingave it:THTTPException(statusCode,responseand the #467 server detail), or the network error forOpenSession, as #598 already does forExecuteStatementconnect failures. Idempotent RPCs keepmain'sHiveDriverErroron the statuses below 100, 429, 5xx-except-501 and network errors they already retried (getRetryExhaustedErrorinThriftHttpConnection.ts); a CloudFetch download that runs out of retries fails withRetryError. A CloudFetch error thrown without retrying is described by its code, so its message doesn't carry the presigned URL.Size: 235 changed non-test lines in
lib/(174 added, 61 removed); 23 of them are the list of certificate error codes.Verification
npm test(1840 passing),npm run lint,npm run type-check, prettier. ThroughThriftHttpConnectionagainst a local server, the errors once retries run out (OpenSession 503/429/501 and a refused or reset connection, GetOperationStatus 408/501, ExecuteStatement 429 and 503 withRetry-After: 9 tests fail on 26abeb4) and the guards that GetOperationStatus on a 503, a status below 100 (rawHTTP/1.1 099status line) or a refused connection keeps theRetryError. An OpenSession over TLS with a cert that Node rejects asUNSPECIFIED(generated per run with openssl, likeHttpConnection.test.ts, so no key is committed) fails at once with that error, and the rpc and CloudFetch policies don't retry it (4 tests fail on 26abeb4). Every row that changed has a unit test that fails on c6b3606 (before the parity change), including the status table, ExecuteStatement 429 and connect failures (also a real multi-address connect failure throughThriftHttpConnection), the RPC request timeout end to end throughThriftHttpConnection(andRequest timed outwhen the timeouts use up the attempts orretriesTimeout, which the e2eTimeoutstests pin), and CloudFetchENOTFOUND. With the default retry settings, a flags endpoint that keeps failing (503 withRetry-After: 60) or never responds falls back to defaults at the deadline; both tests hang without it.retryMaxAttempts: 3), onmain→ this head: OpenSession 503/429/502/408/501, GetOperationStatus 408/501/402, ExecuteStatement 429 and 503 withRetry-Afterfail with the sameTHTTPException(status and server detail; 1 → 3 requests), OpenSession refused with the sameFetchErrorECONNREFUSED, GetOperationStatus 503 and refused with the sameHiveDriverError. With the default retry settings, an OpenSession whose cert fails asUNSPECIFIED(a CA without keyCertSign, an unhandled critical extension, a SHA-1 signature) fails at once with the reason, as onmain; 26abeb4 took 5 connections and 30 s and hid it behind aHiveDriverError.Node.js Integration Tests(replay, thrift and sea) on dd04e5f: run 44020351876 passed. Since thenlib/changed only by the status-below-100 clause, which no recording can reach. The e2e jobs on Node 22, 24 and 26 passed too.UNSPECIFIED, not the attempts below):THTTPException503retriesTimeout1 s vs 5 s)Earlier heads of this PR ran the same rows 3 times each with the same results.
Driver-test follow-ups after merge
RetryStatusCodesTestsThrift "503 on ExecuteStatement — driver does NOT retry": its scenario's 503 carriesRetry-After, so it now sees 5 attempts instead of 1. Its Thrift 429/503-with-Retry-After cases (skipped as PECO-3071) can be enabled.XPASS. #2074's ERRORRECOV-001 note thatRetry-Afteris ignored, and the ERRORRECOV-004 Node skip reason (a dropped OpenSession isn't retried), are now stale.Still different from the kernel
EPROTOwrong version number) isn't retried onExecuteStatement: missing, not detected (accepted gap, see follow-ups). node-fetch keeps only the code and message, and the same codes can come after the request is sent. These are mostly configuration errors (wrong port or TLS version) that a retry can't fix. A socket closed during the handshake is retried.OpenSessionjoining the idempotent RPCs now also apply to it.FetchResults(no 1:1 equivalent).Retry-Afteras an HTTP date; the kernel'sretriesTimeoutguard also covers exponential waits.Follow-ups
ExecuteStatement, which needs tracking each socket's connect phase.This pull request and its description were written by Isaac.