Repository navigation
fix: retry one-shot Thrift CloudFetch 401/403/404 downloads and time out stalled attempts - #610
vuanhphung wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Mostly solid, with 2 concerns about timeout semantics. The per-attempt socketTimeout measures total elapsed time, while on Thrift requests it is an inactivity timeout, so lowering it can now fail large or slow downloads. Separately, link refreshes start a fresh retriesTimeout deadline each time, which loosens the end-to-end bound added in #520.
…stalls A one-shot 401, 403 or 404 from cloud storage failed the query: CloudFetchResultHandler threw on any non-OK response and HttpRetryPolicy retries only 429 and 5xx. Other 4xx responses failed the same way, and a stalled attempt was never timed out, so it waited until retriesTimeout instead of being retried. On 401/403/404 the presigned URL itself was rejected, so fetch a fresh link for the same rows with FetchResults(startRowOffset) and download it, counting each refresh against retryMaxAttempts. RowSetProvider serializes its FetchResults calls and pages by explicit offset after a refresh, since the refresh may move the server cursor. Retry other 4xx with the same URL, and apply socketTimeout to each download attempt so a stalled one is retried. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
… refreshes Review follow-up. The per-attempt download timer now restarts on every received chunk, so socketTimeout aborts only an attempt that stops receiving data, not a large download still in progress. Link refreshes reuse the download's retriesTimeout deadline instead of starting a new one each. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
… the trickle test margin The CHANGELOG now says the idle retry applies when socketTimeout is shorter than retriesTimeout (both default to 15 minutes). CONNECTION_PARAMETERS.md notes that socketTimeout also bounds each Thrift CloudFetch download attempt and that link refreshes count against retryMaxAttempts. The trickling-body unit test had only 100 ms of slack per gap; it now uses a 300 ms idle timeout over a ~1 s body. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
0b03672 to
1caae64
Compare
…ix refresh-budget and socketTimeout docs Every URL in the shared-deadline test now answers 403 after 120 ms, so a fresh retriesTimeout per refreshed URL would refresh 9 times and fail with the 403, while the shared deadline times out within 3 GETs. The docs now say a download tries at most retryMaxAttempts URLs, each retried under its own budget, and the socketTimeout JSDoc gives the Thrift and kernel semantics and defaults. fetch() takes deadlineAt explicitly instead of defaulting it. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low-severity test gap. I traced the refresh loop, the shared deadline, the per-attempt idle abort (including which abort sets timeoutError and which sets ETIMEDOUT), and the offset tracking for prefetched and direct results in RowSetProvider, and found no correctness problems. If refreshLinks itself fails, the caller sees that Thrift error instead of the original CloudFetch 403; this is acceptable, since the query failed before this change too.
The CloudFetch download idle timeout reads socketTimeout from ClientConfig, so the existing connect() retry-option tests now also cover socketTimeout: a set value, 0, and the default when omitted. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Nit
Looks good. The refresh, same-URL retry and idle-timeout logic matches the description, fetchNext and refreshLinks are serialized correctly, and the new tests cover the main paths. One low edge case: the loop does not check the shared deadline before refreshing a link. Follow-ups, not caused by this PR: links already known to be expired still fail upfront with CloudFetch link has expired instead of being refreshed, and non-OK response bodies are still left unconsumed across retries.
Other findings
- ⚪ Nit — The refresh loop does not check whether the shared
retriesTimeoutdeadline has already passed before it callsrefreshLink. Suppose the deadline runs out while the 403 GET is in flight, or during theFetchResultsrefresh, which the deadline does not bound. The loop still issues the refresh RPC and then callsfetchwithdeadlineMs = 0. That fetch starts a GET, aborts it on the next tick, and throwsRetryError(TimeoutExceeded, undefined). That call'slastFailureis still unset, so the message is a bareRetry timeout exceeded. The 403 that actually caused the failure is lost, and you pay for one extraFetchResultsround trip and one doomed request.
Suggested fix: in the loop condition, stop when deadlineAt !== undefined && Date.now() >= deadlineAt. The loop then falls through to the existing CloudFetch HTTP error 403 … path.
…sTimeout The refresh FetchResults RPC is not bounded by the download deadline. If it ran past it, the next fetch started a GET that aborted at once and failed with a bare 'Retry timeout exceeded'. Now the download fails with the same TIMEOUT_EXCEEDED RetryError, with the rejected 401/403/404 response as its cause, and without the doomed GET. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Re the review-body Nit on 0411453 (refresh loop and the shared deadline): fixed in 723056c. The meaningful window is the refresh |
The retriesTimeout deadline was a timer inside each GET, cleared before a link refresh, so a refresh that was queued behind another FetchResults call, or whose RPC was slow, kept the download pending past the deadline (forever, if it never returned). One deadline now spans the whole download, refreshes included; a refresh that returns late can't resume the download, and a refresh still queued when its download gives up is skipped. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good: one low test-coverage gap. I traced the refresh loop and deadline races, the idle-timer cleanup, RowSetProvider serialization, and paging by offset after a refresh, and found no correctness issues. As a nit, the new CHANGELOG entry has no PR reference (#610), although the neighbouring entries have theirs.
A refresh response without a link at the rejected link's row offset (only links for other rows, or none) must fail the download with the original HTTP error after one refresh, and must never download another chunk's link in its place. Also adds the PR number to the CHANGELOG entry. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
…need Drop the hardening that no CLOUDFETCH cell or regression guard needs: skipping a queued link refresh after its download gave up (its late result is already ignored), two deadline checks that the per-attempt check already covers, and the socketTimeout/retryMaxAttempts notes in CONNECTION_PARAMETERS.md and the ConnectionOptions JSDoc. DownloadDeadline.race is now a plain Promise.race, fetch() takes init as optional again so PlainHttpTransport.test.ts is unchanged, and the CHANGELOG entry is shorter. Tests that only covered the removed parts are dropped. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with 1 low concern: a refreshed link is matched only by its start row offset. I checked the deadline, abort, serialization and offset-paging logic against the callers (ThriftOperationBackend is the only one that builds CloudFetchResultHandler/RowSetProvider) and found no correctness problems. The follow-ups the PR lists (draining non-OK bodies, refreshing an expired link) are pre-existing gaps, not regressions.
…d of refreshing the link Per the relaxed CloudFetch spec (a one-shot 401/403/404 must not fail the query; a refresh or a same-URL retry both count, and other 4xx may fail with a clear error), drop the link refresh, the RowSetProvider offset paging and serialization, the cross-refresh download deadline and the other-4xx retry opt-in. A CloudFetch download now marks a 401/403/404 response retryable, so HttpRetryPolicy retries the same URL within its existing attempt budget, backoff and the #520 per-download deadline. 400/405/412 keep failing with 'CloudFetch HTTP error <status>'. The per-attempt socketTimeout idle timeout stays. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
On the Thrift backend, a one-shot 401, 403 or 404 from cloud storage on a CloudFetch download failed the query with
CloudFetch HTTP error 403 Forbidden, becauseHttpRetryPolicyretries only 429 and 5xx. A stalled download attempt was never timed out, so it waited untilretriesTimeout(15 minutes by default) instead of being retried. Found by databricks-driver-test CLOUDFETCH-001/002/003/007, live against a warehouse with proxy-injected failures.The CloudFetch spec was relaxed for this. For 001/002/007, a one-shot 401/403/404 must not fail the query, and a link refresh or a same-URL retry both count; JDBC and the kernel already recover with a same-URL retry. For 006/008/009, a 400/405/412 may end in a clear terminal error naming the status, which Node already does; Go doesn't retry them, and Python's Thrift path doesn't retry a 400. So this PR no longer refreshes links or retries other 4xx.
The fix:
retryable, andHttpRetryPolicyretries the same URL within its existing attempt budget, backoff and the per-downloadretriesTimeoutdeadline ([coverage] Conformance findings: CLOUDFETCH-018 #520). When attempts run out it fails withRetryError(… (last error: HTTP 403: Forbidden)).socketTimeoutis now also the inactivity timeout of each download attempt, restarted when headers and each body chunk arrive, and a timed-out attempt is retried asETIMEDOUT. It takes effect whensocketTimeoutis shorter thanretriesTimeout.connect({ socketTimeout })now reaches the download path.CloudFetch HTTP error <status>.socketTimeout, same-URL retryCloudFetch HTTP error <status>SEA/kernel paths are unchanged. Size: +46/−7 non-test lines (it was +277/−75 before the spec was relaxed).
Testing:
HttpRetryPolicy: a one-shot 401/403/404 is retried with the same URL; a persistent 403 ends inRetryErrornaming HTTP 403; 400/405/412 fail after one GET; a stalled attempt is timed out and retried, with no URL in the error; a body that keeps arriving is not cut off. Also the policy'sretryableflag andconnect({ socketTimeout }). Full unit suite: 1784 passing; lint, type-check and prettier pass.XPASS. For 001/002/007 the query succeeds (143,672 rows) after a same-URL retry; driver-test main still asserts the old refresh contract there, and its tripwires are being rewritten to the relaxed one. 006/008/009 fail withCloudFetch HTTP error 400/405/412as the relaxed contract allows, and 004/005/010..017 pass.Follow-ups (not in this PR):
expiryTimeinstead of failing withCloudFetch link has expired.socketTimeoutinCONNECTION_PARAMETERS.md.This pull request and its description were written by Isaac.