Repository navigation
fix: report a close or cancel during a wait as OperationStateError - #609
vuanhphung wants to merge 6 commits into
Conversation
Closing or cancelling an operation while fetchAll(), fetchChunk() or
finished() waits on it could reject the wait with the server's answer
for the released statement. On Thrift, closing the session while a query
runs makes the in-flight GetOperationStatus return HTTP 409 ("A concurrent
GetOperationStatus request updated the command. Please retry."), which
surfaced as a THTTPException. DBSQLOperation now records a close or cancel
from the moment it is requested and reports such failures as
OperationStateError CLOSED or CANCELED.
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall. The new close/cancel flags fix the 409 case as described, and the tests cover the main paths. One low concern: if the in-flight request fails before a close or cancel that then fails, the waiter is told CLOSED/CANCELED while the operation stays open, and the original error is dropped without a log. That case isn't tested.
A status poll or result fetch could fail while the close or cancel that caused it was still pending. The waiter was then told CLOSED or CANCELED even if that request went on to fail, leaving the operation open and dropping the original error. Keep the latest close/cancel request and await it before choosing the error: report the client's action only if it succeeded, otherwise rethrow the original error. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
A cancel and a close (e.g. from session.close()) can overlap. With only the latest request kept, a close that failed on the unfinished statement made the waiter rethrow the poll error even when the concurrent cancel succeeded. Keep the close and cancel requests separately and await both. Add tests for a failed cancel and for the overlap, and the CHANGELOG reference. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
The existing close test resolved CloseOperation within a few microtasks, so it passed even without awaiting the close request. Add the CONCURRENT-004 ordering where the poll fails first and the close then succeeds, and rename the existing test to the ordering it covers. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
Besides the 409, the server can answer the in-flight GetOperationStatus with 404 RESOURCE_DOES_NOT_EXIST once the close released the statement. Pin that shape with a real THTTPException. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good, with one low concern. The fix is narrow, the microtask ordering that clientStopError depends on is sound (closeInternal/cancelInternal set their flags before the derived .catch() continuation runs), and the tests cover both the remap path and the cases that keep the original error. Follow-up, not caused by this PR: getSchema()/getMetadata() result-metadata RPCs that fail during a close still pass the raw transport error through.
|
Reopening: this was auto-closed by GitHub because the merged driver-test PR databricks/databricks-driver-test#2082 referenced it after a closing keyword ("fix: …#609"). It is still awaiting review. |
Two close() calls can overlap too, e.g. a caller's own close() alongside session.close(). Keeping only the latest request let a failed second close make the waiter rethrow the poll error while the first close was still about to succeed. Track every close and cancel request until it settles and await them all. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
On the review-body follow-up ( |
|
Closing: not needed. On main, a caller that retries after the 409 "…Please retry." gets OperationStateError(CLOSED) immediately, without a server call: the operation's closed flag is set before the session closes and failIfClosed() short-circuits. Verified live on Thrift in 30 of 30 runs, and SEA surfaces CLOSED directly. A retry fired synchronously before the close completes can see one 404 RESOURCE_DOES_NOT_EXIST first. The driver-test CONCURRENT-004 spec is being relaxed to accept this retryable-conflict shape. |
Closing a session while one of its queries is running should make the code waiting on that query (
fetchAll(),fetchChunk(),finished()) fail with a closed error. On Thrift it often fails with the transport error instead:session.close()sendsCloseOperationwhile the wait'sGetOperationStatusis still in flight. The server answers that poll with HTTP 409, or sometimes with HTTP 404RESOURCE_DOES_NOT_EXISTfor the released statement, andDBSQLOperationpasses the error through, even though the client itself closed the operation. The "Please retry" text points callers the wrong way. The databricks-driver-test CONCURRENT-004 test (databricks/databricks-driver-test#2082) reproduces it live againstmain(8ccddd0): the proxy recorded the polls as200,409and the waiter got the 409. In #2082's runs the failed poll was a 409 in 10 of 13 and a 404 in the other 3. A cancel at the same point already getsCANCELED_STATEback, and SEA already reports a closed error.DBSQLOperationnow tracks its close and cancel requests until they settle. When a wait or fetch fails with anything other than anOperationStateError, it waits for every request still in flight. If a close or cancel succeeded, it reportsOperationStateErrorCLOSEDorCANCELED. Otherwise it rethrows the original error. All of them are awaited because closes and cancels can overlap, for example a cancel followed bysession.close(), or a caller's ownclose()alongsidesession.close().Verification, with the kernel at 0b5d0ed:
tests/unit/DBSQLOperation.test.ts. 7 fail with the baseDBSQLOperation(93a9112): close → closed, with the close settled before or after the poll fails, and with the poll failing on a 409 or on a 404THTTPException; cancel → canceled; a cancel overlapped by a failed close; a second close overlapping a successful one and failing; and a result fetch failed by close. The two overlap cases also fail when only the latest close or cancel is awaited (f12a671 fails both, 1ffdb2b the duplicate-close one), and the pending-close cases fail when nothing is awaited. The other 4 cover the cases that must keep the original error.npm test(1762 passing), lint, type-check and prettier are clean.RaceConditionTests(now on itsmain, from #2082) with the Thrift xfail turned into the hard assertion: CONCURRENT-004 and CONCURRENT-005 pass 3/3 on Thrift and 3/3 on SEA, with the driver atmain91a64ff plus this PR (and earlier at 8ccddd0 plus this PR). Every Thrift CONCURRENT-004 run hit the 409 race and reportedThe operation was closed by a client. The unmodified tripwire XPASSes against this build, so databricks-driver-test needs a PR that flips it to a hard assertion once this merges.This pull request and its description were written by Isaac.