From a9a6052eedee7a4366d5265e981c6ecbe5286f9a Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 02:06:42 +0000 Subject: [PATCH 01/15] fix: retry Thrift OpenSession and a 503-with-Retry-After ExecuteStatement Thrift OpenSession was never retried, so a single 503, 429, 502, 504 or 408 failed the connection, and ExecuteStatement failed on a 503 with Retry-After, which the server returns without running the statement. The kernel backend retries both. OpenSession joins the idempotent methods, HttpRetryPolicy retries 408, and ExecuteStatement is also retried on a 503 with Retry-After. So that a wrong host or an untrusted certificate on OpenSession isn't retried for the whole budget and doesn't lose its cause, the shared classifier no longer retries certificate verification failures or ENOTFOUND, as in the kernel. After other network errors the retry-exhaustion message names the error code, and a CloudFetch error thrown without retrying no longer carries the presigned URL. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 1 + lib/connection/connections/HttpRetryPolicy.ts | 37 ++++++++++- .../connections/ThriftHttpConnection.ts | 5 +- .../connections/UnsentRequestRetryPolicy.ts | 7 +- lib/hive/Commands/BaseCommand.ts | 11 ++-- lib/result/CloudFetchResultHandler.ts | 15 +++-- .../connections/HttpRetryPolicy.test.ts | 65 ++++++++++++++++++- .../ThriftHttpConnectionRetry.test.ts | 59 ++++++++++++++--- .../UnsentRequestRetryPolicy.test.ts | 38 ++++++++++- tests/unit/hive/commands/BaseCommand.test.ts | 27 +++++++- .../result/CloudFetchResultHandler.test.ts | 21 +++++- 11 files changed, 253 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd6726b5..88c092bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Thrift `OpenSession` is now retried like other idempotent requests: on transient HTTP errors (408, 429, 5xx except 501) and transient network errors such as a refused or reset connection. HTTP 408 is now retried on every idempotent request. Thrift `ExecuteStatement` is now also retried on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried. Server certificate verification failures and a host name that doesn't resolve (`ENOTFOUND`) are no longer retried by idempotent Thrift requests, CloudFetch downloads or the feature-flag fetch: the original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget. **Behavior change:** when it runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/connection/connections/HttpRetryPolicy.ts b/lib/connection/connections/HttpRetryPolicy.ts index 3e6a2e53..ff6d4211 100644 --- a/lib/connection/connections/HttpRetryPolicy.ts +++ b/lib/connection/connections/HttpRetryPolicy.ts @@ -36,10 +36,38 @@ const RETRYABLE_NETWORK_ERROR_CODES = new Set([ 'EHOSTUNREACH', 'ENETUNREACH', 'EPIPE', - 'ENOTFOUND', 'EAI_AGAIN', ]); +// Retrying can't fix these, so they are thrown at once, as in the kernel: the client's certificate +// verification (OpenSSL's X509 codes as Node names them, and its hostname check) and a host name that +// doesn't resolve (`ENOTFOUND`; a temporary DNS failure is `EAI_AGAIN`). +const PERMANENT_NETWORK_ERROR_CODES = new Set([ + 'CERT_CHAIN_TOO_LONG', + 'CERT_HAS_EXPIRED', + 'CERT_NOT_YET_VALID', + 'CERT_REJECTED', + 'CERT_REVOKED', + 'CERT_SIGNATURE_FAILURE', + 'CERT_UNTRUSTED', + 'DEPTH_ZERO_SELF_SIGNED_CERT', + 'ERROR_IN_CERT_NOT_AFTER_FIELD', + 'ERROR_IN_CERT_NOT_BEFORE_FIELD', + 'ERR_TLS_CERT_ALTNAME_FORMAT', + 'ERR_TLS_CERT_ALTNAME_INVALID', + 'HOSTNAME_MISMATCH', + 'INVALID_CA', + 'INVALID_PURPOSE', + 'PATH_LENGTH_EXCEEDED', + 'SELF_SIGNED_CERT_IN_CHAIN', + 'UNABLE_TO_DECODE_ISSUER_PUBLIC_KEY', + 'UNABLE_TO_DECRYPT_CERT_SIGNATURE', + 'UNABLE_TO_GET_ISSUER_CERT', + 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY', + 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', + 'ENOTFOUND', +]); + // Fallback message patterns for errors that don't carry an errno. node-fetch surfaces // "socket hang up" as a generic FetchError, and "Premature close" when the response // body stream closes before all data is received — both occur regularly when a @@ -172,6 +200,10 @@ export default class HttpRetryPolicy implements IRetryPolicy= 500 && statusCode !== 501); diff --git a/lib/connection/connections/ThriftHttpConnection.ts b/lib/connection/connections/ThriftHttpConnection.ts index fa686596..4cd64ff4 100644 --- a/lib/connection/connections/ThriftHttpConnection.ts +++ b/lib/connection/connections/ThriftHttpConnection.ts @@ -76,6 +76,8 @@ type ThriftClient = { }; const retryableThriftMethods = new Set([ + // A repeated OpenSession leaves at most an unused session, which the server expires + 'OpenSession', 'GetOperationStatus', 'CancelOperation', 'CloseOperation', @@ -93,7 +95,8 @@ const retryableThriftMethods = new Set([ 'GetCrossReference', ]); -// Not safe to repeat once the server may have the request, so retried only if it was never sent +// Not safe to repeat once the server may have run the request, so retried only if it was never sent or +// the server rejected it unrun (see UnsentRequestRetryPolicy) const unsentRequestRetryableThriftMethods = new Set(['ExecuteStatement']); export default class ThriftHttpConnection extends EventEmitter { diff --git a/lib/connection/connections/UnsentRequestRetryPolicy.ts b/lib/connection/connections/UnsentRequestRetryPolicy.ts index 14357c52..1f70521a 100644 --- a/lib/connection/connections/UnsentRequestRetryPolicy.ts +++ b/lib/connection/connections/UnsentRequestRetryPolicy.ts @@ -7,7 +7,8 @@ import RetryError from '../../errors/RetryError'; // socket (ECONNRESET, EPIPE, timeouts, ...) are excluded: the server may already have the request. const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']); -// For operations that are not safe to repeat: retries only requests that never left the client. +// For operations that are not safe to repeat: retries only requests that never left the client, and a +// 503 with `Retry-After`, where the server asks for a retry without running the request (as in the kernel). export default class UnsentRequestRetryPolicy extends HttpRetryPolicy { public async invokeWithRetry(operation: RetryableOperation): Promise { try { @@ -21,8 +22,8 @@ export default class UnsentRequestRetryPolicy extends HttpRetryPolicy { } } - protected isRetryable(): boolean { - return false; + protected isRetryable({ response }: HttpTransactionDetails): boolean { + return response.status === 503 && response.headers.has('Retry-After'); } protected isRetryableNetworkError(error: unknown): boolean { diff --git a/lib/hive/Commands/BaseCommand.ts b/lib/hive/Commands/BaseCommand.ts index 22ac66fb..0db2b029 100644 --- a/lib/hive/Commands/BaseCommand.ts +++ b/lib/hive/Commands/BaseCommand.ts @@ -27,16 +27,15 @@ export default abstract class BaseCommand { ) { statusCode = error.payload.response.status; } + // After network errors, name the last one's code (e.g. ECONNREFUSED) instead of a status + const code = (error.payload as { code?: unknown } | null | undefined)?.code; + const cause = statusCode ?? (typeof code === 'string' && code ? code : 'Error'); switch (error.errorCode) { case RetryErrorCode.AttemptsExceeded: - throw new HiveDriverError( - `Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Max retry count exceeded.`, - ); + throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Max retry count exceeded.`); case RetryErrorCode.TimeoutExceeded: - throw new HiveDriverError( - `Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Retry timeout exceeded.`, - ); + throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Retry timeout exceeded.`); // no default } } diff --git a/lib/result/CloudFetchResultHandler.ts b/lib/result/CloudFetchResultHandler.ts index fbfe5c7c..1e30363b 100644 --- a/lib/result/CloudFetchResultHandler.ts +++ b/lib/result/CloudFetchResultHandler.ts @@ -1,4 +1,4 @@ -import fetch, { RequestInfo, RequestInit, Request, Response } from 'node-fetch'; +import fetch, { FetchError, RequestInfo, RequestInit, Request, Response } from 'node-fetch'; import { TGetResultSetMetadataResp, TRowSet, TSparkArrowResultLink } from '../../thrift/TCLIService_types'; import HiveDriverError from '../errors/HiveDriverError'; import RetryError, { RetryErrorCode } from '../errors/RetryError'; @@ -244,10 +244,12 @@ export default class CloudFetchResultHandler implements IResultsProvider { } // Status codes between 100 (including) and 500 (excluding) should not be retried - // The only exception is 429 (Too many requests) + // The only exceptions are 408 (Request Timeout) and 429 (Too many requests) for (let status = 100; status < 500; status += 1) { - const expectedResult = status === 429 ? true : false; + const expectedResult = status === 408 || status === 429; expect(policy['isRetryable'](createStub(status))).to.equal(expectedResult); } @@ -401,6 +401,67 @@ describe('HttpRetryPolicy', () => { } }); + describe('network error classification', () => { + function systemError(code: string): FetchError { + return new FetchError(`request to https://example.com failed, reason: ${code}`, 'system', { + code, + } as unknown as NodeJS.ErrnoException); + } + + function createPolicy() { + return new HttpRetryPolicy(new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 5 })); + } + + // Certificate verification failures and a host name that doesn't resolve + for (const code of [ + 'DEPTH_ZERO_SELF_SIGNED_CERT', + 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', + 'SELF_SIGNED_CERT_IN_CHAIN', + 'CERT_HAS_EXPIRED', + 'ERR_TLS_CERT_ALTNAME_INVALID', + 'ENOTFOUND', + ]) { + it(`throws ${code} at once, without retrying`, async () => { + const error = systemError(code); + const operation = sinon.stub().rejects(error); + + try { + await createPolicy().invokeWithRetry(operation); + expect.fail('It should throw the error'); + } catch (e) { + if (e instanceof AssertionError) { + throw e; + } + expect(e).to.equal(error); + } + expect(operation.callCount).to.equal(1); + }); + } + + // Transient connect/DNS failures, a TLS alert sent by the server, and unavailable revocation data + for (const code of [ + 'ECONNREFUSED', + 'ECONNRESET', + 'EAI_AGAIN', + 'ERR_SSL_TLSV13_ALERT_CERTIFICATE_REQUIRED', + 'UNABLE_TO_GET_CRL', + ]) { + it(`retries ${code}`, async () => { + const operation = sinon.stub(); + operation.onCall(0).rejects(systemError(code)); + operation.onCall(1).resolves({ + request: new Request('http://localhost'), + response: new Response(undefined, { status: 200 }), + }); + + const result = await createPolicy().invokeWithRetry(operation); + + expect(result.response.status).to.equal(200); + expect(operation.callCount).to.equal(2); + }); + } + }); + it('should cancel a pending backoff sleep when the signal is aborted', async () => { const context = new ClientContextStub({ retryDelayMin: 10000, diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 4492f69b..c5849ec1 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -20,7 +20,13 @@ function thriftMessage(name: string, type: number, seqId: number): Buffer { const CALL = 1; const REPLY = 2; -type ServerAction = 'reply' | 'close-after-request' | 'status-503'; +type ServerAction = + | 'reply' + | 'close-after-request' + | 'status-408' + | 'status-429' + | 'status-503' + | 'status-503-retry-after'; interface TestServer { port: number; @@ -39,8 +45,11 @@ async function startServer(actions: ServerAction[]): Promise { if (action === 'close-after-request') { // The full request has been received; drop the connection without a response req.socket.destroy(); - } else if (action === 'status-503') { - res.writeHead(503); + } else if (action === 'status-503-retry-after') { + res.writeHead(503, { 'Retry-After': '1' }); + res.end(); + } else if (action.startsWith('status-')) { + res.writeHead(Number(action.slice('status-'.length))); res.end(); } else { const body = Buffer.concat(chunks); @@ -179,20 +188,52 @@ describe('ThriftHttpConnection retries', () => { expect(server.requests).to.equal(2); }); - it('does not retry ExecuteStatement on an HTTP error status', async () => { - const { server, connection } = await setup(['status-503'], 0); + for (const status of ['status-503', 'status-429', 'status-408'] as const) { + it(`does not retry ExecuteStatement on an HTTP error status (${status})`, async () => { + const { server, connection } = await setup([status], 0); - const error = await expectRejection(send(connection, 'ExecuteStatement', 1)); + const error = await expectRejection(send(connection, 'ExecuteStatement', 1)); + + expect(error).to.be.instanceOf(THTTPException); + expect(error.statusCode).to.equal(Number(status.slice('status-'.length))); + expect(server.requests).to.equal(1); + }); + } + + it('retries ExecuteStatement on a 503 with Retry-After, which the server did not run', async () => { + const { server, connection } = await setup(['status-503-retry-after', 'reply'], 0); - expect(error).to.be.instanceOf(THTTPException); - expect(error.statusCode).to.equal(503); + const result = await send(connection, 'ExecuteStatement', 1); + + expect(result).to.equal('ExecuteStatement result'); + expect(server.requests).to.equal(2); + }); + + for (const status of ['status-503', 'status-429', 'status-408'] as const) { + it(`retries OpenSession on a transient HTTP error (${status})`, async () => { + const { server, connection } = await setup([status, 'reply'], 0); + + const result = await send(connection, 'OpenSession', 1); + + expect(result).to.equal('OpenSession result'); + expect(server.requests).to.equal(2); + }); + } + + it('retries OpenSession when the connection is refused', async () => { + const { server, agent, connection } = await setup(['reply'], 1); + + const result = await send(connection, 'OpenSession', 1); + + expect(result).to.equal('OpenSession result'); + expect(agent.connectAttempts).to.equal(2); expect(server.requests).to.equal(1); }); it('does not retry other non-idempotent operations when the connection is refused', async () => { const { server, agent, connection } = await setup(['reply'], 1); - const error = await expectRejection(send(connection, 'OpenSession', 1)); + const error = await expectRejection(send(connection, 'FetchResults', 1)); expect(error).to.be.instanceOf(FetchError); expect(error.code).to.equal('ECONNREFUSED'); diff --git a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts index ca01a9a7..653d7e14 100644 --- a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts +++ b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts @@ -64,9 +64,20 @@ describe('UnsentRequestRetryPolicy', () => { }); } - for (const status of [429, 500, 503]) { - it(`does not retry an HTTP ${status} response`, async () => { - const details = { request: new Request('http://localhost'), response: new Response(undefined, { status }) }; + function statusResponse(status: number, headers: Record = {}) { + return { request: new Request('http://localhost'), response: new Response(undefined, { status, headers }) }; + } + + for (const [status, headers] of [ + [429, {}], + [500, {}], + [503, {}], + [500, { 'Retry-After': '1' }], + ] as const) { + it(`does not retry an HTTP ${status} response ${ + Object.keys(headers).length ? 'with' : 'without' + } Retry-After`, async () => { + const details = statusResponse(status, headers); const policy = createPolicy(); const operation = sinon.stub().resolves(details); @@ -76,6 +87,27 @@ describe('UnsentRequestRetryPolicy', () => { }); } + it('retries an HTTP 503 with Retry-After, which the server did not run', async () => { + const operation = sinon.stub(); + operation.onCall(0).resolves(statusResponse(503, { 'Retry-After': '1' })); + operation.onCall(1).resolves(okResponse()); + + const result = await createPolicy().invokeWithRetry(operation); + + expect(result.response.status).to.equal(200); + expect(operation.callCount).to.equal(2); + }); + + it('throws a RetryError once a persistent 503 with Retry-After runs out of attempts', async () => { + const operation = sinon.stub().resolves(statusResponse(503, { 'Retry-After': '1' })); + + const error = await expectRejection(createPolicy(3).invokeWithRetry(operation)); + + expect(error).to.be.instanceOf(RetryError); + expect((error as RetryError).errorCode).to.equal(RetryErrorCode.AttemptsExceeded); + expect(operation.callCount).to.equal(3); + }); + it('rethrows the original error once max attempts is reached', async () => { const error = systemError('ECONNREFUSED'); const operation = sinon.stub().rejects(error); diff --git a/tests/unit/hive/commands/BaseCommand.test.ts b/tests/unit/hive/commands/BaseCommand.test.ts index b1514775..df372a05 100644 --- a/tests/unit/hive/commands/BaseCommand.test.ts +++ b/tests/unit/hive/commands/BaseCommand.test.ts @@ -1,6 +1,6 @@ import { expect, AssertionError } from 'chai'; import sinon from 'sinon'; -import { Request, Response } from 'node-fetch'; +import { FetchError, Request, Response } from 'node-fetch'; import { Thrift } from 'thrift'; import HiveDriverError from '../../../../lib/errors/HiveDriverError'; import BaseCommand from '../../../../lib/hive/Commands/BaseCommand'; @@ -211,6 +211,31 @@ describe('BaseCommand', () => { }); }); + it('should name the network error code once retries run out', async () => { + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); + const command = new CustomCommand( + new ThriftClientStub(context, async () => { + throw new FetchError('request to http://localhost/ failed, reason: connect ECONNREFUSED', 'system', { + code: 'ECONNREFUSED', + } as unknown as NodeJS.ErrnoException); + }), + context, + ); + + try { + await command.execute({}); + expect.fail('It should throw an error'); + } catch (error) { + if (error instanceof AssertionError || !(error instanceof Error)) { + throw error; + } + expect(error).to.be.instanceof(HiveDriverError); + expect(error.message).to.equal( + 'Hive driver: ECONNREFUSED when connecting to resource. Max retry count exceeded.', + ); + } + }); + it(`should re-throw unrecognized HTTP errors`, async () => { const context = new ClientContextStub(); diff --git a/tests/unit/result/CloudFetchResultHandler.test.ts b/tests/unit/result/CloudFetchResultHandler.test.ts index 4498d177..47a92143 100644 --- a/tests/unit/result/CloudFetchResultHandler.test.ts +++ b/tests/unit/result/CloudFetchResultHandler.test.ts @@ -748,9 +748,12 @@ describe('CloudFetchResultHandler', () => { // Real `HttpRetryPolicy` and node-fetch against a local server, so the deadline is exercised // across actual attempts, backoff sleeps and sockets. - function createHandler(fileLink: string, configOverrides: Partial) { + function createHandler(fileLink: string, configOverrides: Partial, agent?: http.Agent) { const context = new BaseClientContextStub({ cloudFetchConcurrentDownloads: 1, ...configOverrides }); sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); + if (agent) { + sinon.stub(context.connectionProvider, 'getAgent').resolves(agent); + } const rowSet: TRowSet = { startRowOffset: new Int64(0), rows: [], @@ -789,6 +792,22 @@ describe('CloudFetchResultHandler', () => { return { ...outcome, elapsedMs: Date.now() - startedAt }; } + it('should not leak the presigned URL from a network error that is not retried', async () => { + // Every lookup fails as for a host that doesn't resolve, which is thrown without retrying + const agent = new http.Agent({ + lookup: (hostname: string, options: unknown, callback: Function) => + callback(Object.assign(new Error(`getaddrinfo ENOTFOUND ${hostname}`), { code: 'ENOTFOUND' })), + } as unknown as http.AgentOptions); + const handler = createHandler('http://storage.example.invalid/chunk?sig=SECRET_SIGNATURE', {}, agent); + + const { error } = await fetchWithin(handler, 1500); + agent.destroy(); + + expect(error.code).to.be.equal('ENOTFOUND'); + expect(error.message).to.be.equal('CloudFetch result download failed: network error ENOTFOUND'); + expect(error.stack).to.not.contain('SECRET_SIGNATURE'); + }); + it('should fail a download whose response headers never arrive with a timeout error', async () => { const closedRequests: Array> = []; const url = await startServer((req) => { From c6b36062b91bd110b0a0a0087908b745ee1bdd7e Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 03:37:09 +0000 Subject: [PATCH 02/15] fix: wait as long as Retry-After asks, except for the feature-flag fetch getRetryAfterHeader compared the header's seconds with retryDelayMin in milliseconds, so a Retry-After below 1000 s had no effect, including on the new 503-with-Retry-After retry of ExecuteStatement. Read it in seconds and use it as the wait, clamped to [retryDelayMin, retryDelayMax]; a wait that would end after retriesTimeout isn't started, as in the kernel. connect() waits for the feature-flag fetch, so that fetch drops Retry-After and keeps the exponential backoff instead of waiting up to retryDelayMax per retry. ENOTFOUND is no longer retried for ExecuteStatement either, as in the kernel and the shared classifier. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- lib/FeatureFlagCache.ts | 6 +- lib/connection/connections/HttpRetryPolicy.ts | 20 +++-- .../connections/UnsentRequestRetryPolicy.ts | 3 +- .../connections/HttpRetryPolicy.test.ts | 78 +++++++++++++------ .../UnsentRequestRetryPolicy.test.ts | 12 ++- tests/unit/telemetry/FeatureFlagCache.test.ts | 39 ++++++++++ 7 files changed, 125 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 88c092bb..f5fc710a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift `OpenSession` is now retried like other idempotent requests: on transient HTTP errors (408, 429, 5xx except 501) and transient network errors such as a refused or reset connection. HTTP 408 is now retried on every idempotent request. Thrift `ExecuteStatement` is now also retried on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried. Server certificate verification failures and a host name that doesn't resolve (`ENOTFOUND`) are no longer retried by idempotent Thrift requests, CloudFetch downloads or the feature-flag fetch: the original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget. **Behavior change:** when it runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift `OpenSession` is now retried like other idempotent requests: on transient HTTP errors (408, 429, 5xx except 501) and transient network errors such as a refused or reset connection. HTTP 408 is now retried on every idempotent request. Thrift `ExecuteStatement` is now also retried on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried. Server certificate verification failures and a host name that doesn't resolve (`ENOTFOUND`) are no longer retried by Thrift requests, CloudFetch downloads or the feature-flag fetch: the original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for keeps the exponential backoff. **Behavior change:** when it runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/FeatureFlagCache.ts b/lib/FeatureFlagCache.ts index 8e7f323c..cc6998a7 100644 --- a/lib/FeatureFlagCache.ts +++ b/lib/FeatureFlagCache.ts @@ -239,7 +239,11 @@ export default class FeatureFlagCache { const requestConfig: RequestInit = { agent, ...init }; const result = await retryPolicy.invokeWithRetry(() => { const request = new Request(url, requestConfig); - return fetch(request).then((response) => ({ request, response })); + return fetch(request).then((response) => { + // connect() waits for the flags, so back off as usual instead of for as long as `Retry-After` asks + response.headers.delete('Retry-After'); + return { request, response }; + }); }); return result.response; } diff --git a/lib/connection/connections/HttpRetryPolicy.ts b/lib/connection/connections/HttpRetryPolicy.ts index ff6d4211..15520f78 100644 --- a/lib/connection/connections/HttpRetryPolicy.ts +++ b/lib/connection/connections/HttpRetryPolicy.ts @@ -163,13 +163,17 @@ export default class HttpRetryPolicy implements IRetryPolicy clientConfig.retriesTimeout) { + throw new RetryError(RetryErrorCode.TimeoutExceeded, details); + } + return { shouldRetry: true, retryAfter }; + } + const retryAfter = this.getBackoffDelay(this.attempt, clientConfig.retryDelayMin, clientConfig.retryDelayMax); return { shouldRetry: true, retryAfter }; } @@ -243,12 +247,12 @@ export default class HttpRetryPolicy implements IRetryPolicy 0) { - return Math.max(delayMin, value); + return Math.max(delayMin, value * 1000); } } return undefined; diff --git a/lib/connection/connections/UnsentRequestRetryPolicy.ts b/lib/connection/connections/UnsentRequestRetryPolicy.ts index 1f70521a..c2d11ce2 100644 --- a/lib/connection/connections/UnsentRequestRetryPolicy.ts +++ b/lib/connection/connections/UnsentRequestRetryPolicy.ts @@ -5,7 +5,8 @@ import RetryError from '../../errors/RetryError'; // Raised before a connection exists, so no request bytes were written. Failures on a connected // socket (ECONNRESET, EPIPE, timeouts, ...) are excluded: the server may already have the request. -const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']); +// ENOTFOUND is permanent, as in HttpRetryPolicy. +const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'EAI_AGAIN']); // For operations that are not safe to repeat: retries only requests that never left the client, and a // 503 with `Retry-After`, where the server asks for a retry without running the request (as in the kernel). diff --git a/tests/unit/connection/connections/HttpRetryPolicy.test.ts b/tests/unit/connection/connections/HttpRetryPolicy.test.ts index 870ce6df..60a8b4a4 100644 --- a/tests/unit/connection/connections/HttpRetryPolicy.test.ts +++ b/tests/unit/connection/connections/HttpRetryPolicy.test.ts @@ -34,18 +34,20 @@ describe('HttpRetryPolicy', () => { // Missing `Retry-After` header expect(policy['getRetryAfterHeader'](createStub({}), retryDelayMin)).to.be.undefined; - // Valid `Retry-After`, several header name variants - expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '10' }), retryDelayMin)).to.equal(10); - expect(policy['getRetryAfterHeader'](createStub({ 'retry-after': '10' }), retryDelayMin)).to.equal(10); - expect(policy['getRetryAfterHeader'](createStub({ 'RETRY-AFTER': '10' }), retryDelayMin)).to.equal(10); + // Valid `Retry-After` in seconds, several header name variants, returned in milliseconds + expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '10' }), retryDelayMin)).to.equal(10_000); + expect(policy['getRetryAfterHeader'](createStub({ 'retry-after': '10' }), retryDelayMin)).to.equal(10_000); + expect(policy['getRetryAfterHeader'](createStub({ 'RETRY-AFTER': '10' }), retryDelayMin)).to.equal(10_000); // Invalid header values (non-numeric, negative) expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': 'test' }), retryDelayMin)).to.be.undefined; expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '-10' }), retryDelayMin)).to.be.undefined; // It should not be smaller than min value, but can be greater than max value - expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '1' }), retryDelayMin)).to.equal(retryDelayMin); - expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '200' }), retryDelayMin)).to.equal(200); + expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '0.001' }), retryDelayMin)).to.equal( + retryDelayMin, + ); + expect(policy['getRetryAfterHeader'](createStub({ 'Retry-After': '200' }), retryDelayMin)).to.equal(200_000); }); it('should check if HTTP transaction is safe to retry', async () => { @@ -103,35 +105,65 @@ describe('HttpRetryPolicy', () => { expect(result.shouldRetry).to.be.false; }); - it('should use `Retry-After` header as a base for backoff', async () => { - const context = new ClientContextStub({ retryDelayMin: 3, retryDelayMax: 100, retryMaxAttempts: 10 }); + it('should wait as long as `Retry-After` asks, within retryDelayMin and retryDelayMax', async () => { + const context = new ClientContextStub({ retryDelayMin: 1000, retryDelayMax: 60_000, retryMaxAttempts: 10 }); const policy = new HttpRetryPolicy(context); function createStub(headers: HeadersInit) { return { request: new Request('http://localhost'), - response: new Response(undefined, { status: 500, headers }), + response: new Response(undefined, { status: 503, headers }), }; } - const result1 = await policy.shouldRetry(createStub({ 'Retry-After': '5' })); - expect(result1.shouldRetry).to.be.true; - if (result1.shouldRetry) { - expect(result1.retryAfter).to.equal(10); - } + // The header is the wait, whatever the attempt (no exponential growth) + expect(await policy.shouldRetry(createStub({ 'Retry-After': '5' }))).to.deep.equal({ + shouldRetry: true, + retryAfter: 5_000, + }); + expect(await policy.shouldRetry(createStub({ 'Retry-After': '30' }))).to.deep.equal({ + shouldRetry: true, + retryAfter: 30_000, + }); + // Clamped up to retryDelayMin and down to retryDelayMax + expect(await policy.shouldRetry(createStub({ 'Retry-After': '0.2' }))).to.deep.equal({ + shouldRetry: true, + retryAfter: 1_000, + }); + expect(await policy.shouldRetry(createStub({ 'Retry-After': '600' }))).to.deep.equal({ + shouldRetry: true, + retryAfter: 60_000, + }); + }); - const result2 = await policy.shouldRetry(createStub({ 'Retry-After': '8' })); - expect(result2.shouldRetry).to.be.true; - if (result2.shouldRetry) { - expect(result2.retryAfter).to.equal(32); + it('should not wait for a `Retry-After` that would end after retriesTimeout', async () => { + const shortBudget = { retryDelayMin: 1000, retryDelayMax: 60_000, retriesTimeout: 5000 }; + + function createStub(retryAfter: string) { + return { + request: new Request('http://localhost', { method: 'POST' }), + response: new Response(undefined, { status: 429, headers: { 'Retry-After': retryAfter } }), + }; } - policy['attempt'] = 4; - const result3 = await policy.shouldRetry(createStub({ 'Retry-After': '10' })); - expect(result3.shouldRetry).to.be.true; - if (result3.shouldRetry) { - expect(result3.retryAfter).to.equal(100); + const operation = sinon.stub().resolves(createStub('60')); + const started = Date.now(); + try { + await new HttpRetryPolicy(new ClientContextStub(shortBudget)).invokeWithRetry(operation); + expect.fail('It should throw an error'); + } catch (error) { + if (error instanceof AssertionError || !(error instanceof Error)) { + throw error; + } + expect(error).to.be.instanceOf(RetryError); + expect((error as RetryError).errorCode).to.equal(RetryErrorCode.TimeoutExceeded); } + expect(operation.callCount).to.equal(1); + expect(Date.now() - started).to.be.lt(1000); + + // A wait that fits is kept + const fitting = new HttpRetryPolicy(new ClientContextStub(shortBudget)); + expect(await fitting.shouldRetry(createStub('3'))).to.deep.equal({ shouldRetry: true, retryAfter: 3000 }); }); it('should use backoff when `Retry-After` header is missing', async () => { diff --git a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts index 653d7e14..fd62aabd 100644 --- a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts +++ b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts @@ -33,7 +33,7 @@ async function expectRejection(promise: Promise): Promise { } describe('UnsentRequestRetryPolicy', () => { - for (const code of ['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']) { + for (const code of ['ECONNREFUSED', 'EAI_AGAIN']) { it(`retries ${code}, which is raised before the request is sent`, async () => { const operation = sinon.stub(); operation.onCall(0).rejects(systemError(code)); @@ -53,6 +53,7 @@ describe('UnsentRequestRetryPolicy', () => { ['EHOSTUNREACH', systemError('EHOSTUNREACH')], ['a body timeout', new FetchError('Response timeout while trying to fetch', 'body-timeout')], ['a premature close', new Error('Premature close')], + ['ENOTFOUND, a host name that does not resolve', systemError('ENOTFOUND')], ]; for (const [label, error] of sentRequestErrors) { @@ -98,6 +99,15 @@ describe('UnsentRequestRetryPolicy', () => { expect(operation.callCount).to.equal(2); }); + it('waits the Retry-After delay before retrying a 503, with the default config', async () => { + const policy = new UnsentRequestRetryPolicy(new ClientContextStub()); + + expect(await policy.shouldRetry(statusResponse(503, { 'Retry-After': '5' }))).to.deep.equal({ + shouldRetry: true, + retryAfter: 5_000, + }); + }); + it('throws a RetryError once a persistent 503 with Retry-After runs out of attempts', async () => { const operation = sinon.stub().resolves(statusResponse(503, { 'Retry-After': '1' })); diff --git a/tests/unit/telemetry/FeatureFlagCache.test.ts b/tests/unit/telemetry/FeatureFlagCache.test.ts index 7f1ea951..da179f6f 100644 --- a/tests/unit/telemetry/FeatureFlagCache.test.ts +++ b/tests/unit/telemetry/FeatureFlagCache.test.ts @@ -16,8 +16,11 @@ import { expect } from 'chai'; import sinon from 'sinon'; +import http from 'http'; +import { AddressInfo } from 'net'; import { Response } from 'node-fetch'; import FeatureFlagCache from '../../../lib/FeatureFlagCache'; +import HttpRetryPolicy from '../../../lib/connection/connections/HttpRetryPolicy'; import ClientContextStub from '../.stubs/ClientContextStub'; import { LogLevel } from '../../../lib/contracts/IDBSQLLogger'; @@ -273,6 +276,42 @@ describe('FeatureFlagCache', () => { fetchStub.restore(); }); + + it('should not wait for Retry-After before retrying the flags request', async () => { + // connect() waits for the flags, so a Retry-After from the endpoint must not stretch the retries + clock.restore(); + let hits = 0; + const server = http.createServer((req, res) => { + hits += 1; + if (hits === 1) { + res.writeHead(503, { 'Retry-After': '60' }); + res.end(); + } else { + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify({ flags: [] })); + } + }); + await new Promise((resolve) => { + server.listen(0, '127.0.0.1', resolve); + }); + try { + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 60_000 }); + sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); + const cache = new FeatureFlagCache(context); + const { port } = server.address() as AddressInfo; + + const started = Date.now(); + const response = await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET' }); + + expect(response.status).to.equal(200); + expect(hits).to.equal(2); + expect(Date.now() - started).to.be.lt(1000); + } finally { + server.closeAllConnections(); + server.close(); + clock = sinon.useFakeTimers(); + } + }); }); describe('Integration scenarios', () => { From ab623a9d99a6f094ee888dc58dfca1ef7a2e230c Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 04:26:48 +0000 Subject: [PATCH 03/15] fix: retry the same statuses and network errors as the kernel backend Which Thrift requests are retried now matches the kernel (src/client/retry.rs and the CloudFetch chunk retry), in one shared classification; backoff and the retry budget stay Node's: - Idempotent RPCs (OpenSession and the other idempotent Thrift methods) and CloudFetch downloads retry every status except 400, 401, 403-405 and 409-416, so 501 and the other 4xx are retried too. Thrift RPCs also retry a request timeout. - ExecuteStatement is retried on a 429, a 503 with Retry-After, and any failure to connect (connect timeouts, unreachable hosts and a socket closed during the TLS handshake too, not only ECONNREFUSED and EAI_AGAIN). - CloudFetch downloads retry ENOTFOUND again, as on main and in the kernel; certificate failures stay unretried everywhere, ENOTFOUND on RPCs. - The feature-flag fetch makes a single attempt, as in the kernel, which also replaces the Retry-After strip. A retry-exhaustion error after request timeouts reads "Hive driver: request-timeout when connecting to resource ...". Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- lib/FeatureFlagCache.ts | 18 +--- lib/connection/connections/HttpConnection.ts | 6 +- lib/connection/connections/HttpRetryPolicy.ts | 82 +++++++++++----- .../connections/UnsentRequestRetryPolicy.ts | 19 ++-- .../contracts/IConnectionProvider.ts | 4 +- lib/connection/contracts/IRetryPolicy.ts | 3 + lib/hive/Commands/BaseCommand.ts | 7 +- lib/result/CloudFetchResultHandler.ts | 2 +- .../telemetry/telemetry-integration.test.ts | 2 +- tests/unit/.stubs/ConnectionProviderStub.ts | 5 +- tests/unit/DBSQLClient.test.ts | 4 +- .../connections/HttpRetryPolicy.test.ts | 98 ++++++++++--------- .../ThriftHttpConnectionRetry.test.ts | 40 ++++++-- .../UnsentRequestRetryPolicy.test.ts | 51 +++++++--- tests/unit/hive/commands/BaseCommand.test.ts | 23 +++++ .../result/CloudFetchResultHandler.test.ts | 49 ++++++++-- tests/unit/telemetry/FeatureFlagCache.test.ts | 37 +++---- 18 files changed, 286 insertions(+), 166 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f5fc710a..d2ce645e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift `OpenSession` is now retried like other idempotent requests: on transient HTTP errors (408, 429, 5xx except 501) and transient network errors such as a refused or reset connection. HTTP 408 is now retried on every idempotent request. Thrift `ExecuteStatement` is now also retried on HTTP 503 with a `Retry-After` header, which the server returns without running the statement; other HTTP errors on `ExecuteStatement` are still not retried. Server certificate verification failures and a host name that doesn't resolve (`ENOTFOUND`) are no longer retried by Thrift requests, CloudFetch downloads or the feature-flag fetch: the original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). These retries use the existing `retryMaxAttempts` / `retriesTimeout` budget. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for keeps the exponential backoff. **Behavior change:** when it runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException`, network error or `Request timed out`, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/FeatureFlagCache.ts b/lib/FeatureFlagCache.ts index cc6998a7..7cb713fc 100644 --- a/lib/FeatureFlagCache.ts +++ b/lib/FeatureFlagCache.ts @@ -195,7 +195,7 @@ export default class FeatureFlagCache { logger.log(LogLevel.debug, `Fetching feature flags from ${endpoint}`); - const response = await this.fetchWithRetry(endpoint, { + const response = await this.sendRequest(endpoint, { method: 'GET', headers, timeout: 10000, @@ -231,21 +231,11 @@ export default class FeatureFlagCache { } } - /** Retries transient failures using the connection's retry policy. */ - private async fetchWithRetry(url: string, init: RequestInit): Promise { + // A single attempt, as in the kernel: connect() waits for the flags, which fall back to defaults on failure. + private async sendRequest(url: string, init: RequestInit): Promise { const connectionProvider = await this.context.getConnectionProvider(); const agent = await connectionProvider.getAgent(url); - const retryPolicy = await connectionProvider.getRetryPolicy(); - const requestConfig: RequestInit = { agent, ...init }; - const result = await retryPolicy.invokeWithRetry(() => { - const request = new Request(url, requestConfig); - return fetch(request).then((response) => { - // connect() waits for the flags, so back off as usual instead of for as long as `Retry-After` asks - response.headers.delete('Retry-After'); - return { request, response }; - }); - }); - return result.response; + return fetch(new Request(url, { agent, ...init })); } private async getAuthHeaders(): Promise> { diff --git a/lib/connection/connections/HttpConnection.ts b/lib/connection/connections/HttpConnection.ts index 6472265a..f51db386 100644 --- a/lib/connection/connections/HttpConnection.ts +++ b/lib/connection/connections/HttpConnection.ts @@ -11,7 +11,7 @@ import { LogLevel } from '../../contracts/IDBSQLLogger'; import HiveDriverError from '../../errors/HiveDriverError'; import ThriftHttpConnection from './ThriftHttpConnection'; -import IRetryPolicy from '../contracts/IRetryPolicy'; +import IRetryPolicy, { RetryRequestClass } from '../contracts/IRetryPolicy'; import HttpRetryPolicy from './HttpRetryPolicy'; import UnsentRequestRetryPolicy from './UnsentRequestRetryPolicy'; @@ -227,7 +227,7 @@ export default class HttpConnection implements IConnectionProvider { return this.connection; } - public async getRetryPolicy(): Promise> { - return new HttpRetryPolicy(this.context); + public async getRetryPolicy(requestClass?: RetryRequestClass): Promise> { + return new HttpRetryPolicy(this.context, requestClass); } } diff --git a/lib/connection/connections/HttpRetryPolicy.ts b/lib/connection/connections/HttpRetryPolicy.ts index 15520f78..cfffe03a 100644 --- a/lib/connection/connections/HttpRetryPolicy.ts +++ b/lib/connection/connections/HttpRetryPolicy.ts @@ -1,4 +1,4 @@ -import IRetryPolicy, { ShouldRetryResult, RetryableOperation } from '../contracts/IRetryPolicy'; +import IRetryPolicy, { ShouldRetryResult, RetryableOperation, RetryRequestClass } from '../contracts/IRetryPolicy'; import { HttpTransactionDetails } from '../contracts/IConnectionProvider'; import IClientContext from '../../contracts/IClientContext'; import RetryError, { RetryErrorCode } from '../../errors/RetryError'; @@ -39,10 +39,8 @@ const RETRYABLE_NETWORK_ERROR_CODES = new Set([ 'EAI_AGAIN', ]); -// Retrying can't fix these, so they are thrown at once, as in the kernel: the client's certificate -// verification (OpenSSL's X509 codes as Node names them, and its hostname check) and a host name that -// doesn't resolve (`ENOTFOUND`; a temporary DNS failure is `EAI_AGAIN`). -const PERMANENT_NETWORK_ERROR_CODES = new Set([ +// The client's certificate verification (OpenSSL's X509 codes as Node names them, and its hostname check) +const CERTIFICATE_ERROR_CODES = [ 'CERT_CHAIN_TOO_LONG', 'CERT_HAS_EXPIRED', 'CERT_NOT_YET_VALID', @@ -65,8 +63,47 @@ const PERMANENT_NETWORK_ERROR_CODES = new Set([ 'UNABLE_TO_GET_ISSUER_CERT', 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY', 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', - 'ENOTFOUND', -]); +]; + +// Which statuses and network errors are retried matches the kernel (databricks-sql-kernel src/client/retry.rs +// and the CloudFetch chunk retry); backoff and budget stay Node's. +// Network errors retrying can't fix, thrown at once: certificate failures, and for RPCs a host name that doesn't +// resolve (`ENOTFOUND`; a temporary DNS failure is `EAI_AGAIN`). The kernel retries CloudFetch downloads on it. +const PERMANENT_NETWORK_ERROR_CODES: Record> = { + rpc: new Set([...CERTIFICATE_ERROR_CODES, 'ENOTFOUND']), + cloudFetch: new Set(CERTIFICATE_ERROR_CODES), +}; + +// Idempotent requests retry every status except these (the kernel's IDEMPOTENT_NON_RETRYABLE); non-idempotent +// ones only a 429, or a 503 with `Retry-After`. +const IDEMPOTENT_NON_RETRYABLE_STATUSES = new Set([400, 401, 403, 404, 405, 409, 410, 411, 412, 413, 414, 415, 416]); + +export function isRetryableStatus({ response }: HttpTransactionDetails, idempotent: boolean): boolean { + const { status } = response; + if (status >= 200 && status < 400) { + return false; + } + if (idempotent) { + return !IDEMPOTENT_NON_RETRYABLE_STATUSES.has(status); + } + return status === 429 || (status === 503 && response.headers.has('Retry-After')); +} + +// A failure to connect (the kernel's `is_connect`), so the request never left the client and even a +// non-idempotent one can be retried. Node reports these as `connect
`, or as a socket closed +// before the TLS handshake finished. +export function isConnectError(error: unknown): boolean { + const { code, message } = (error ?? {}) as { code?: unknown; message?: unknown }; + if (typeof code !== 'string' || PERMANENT_NETWORK_ERROR_CODES.rpc.has(code)) { + return false; + } + return ( + code === 'ECONNREFUSED' || + code === 'EAI_AGAIN' || + (typeof message === 'string' && + (message.includes(`connect ${code}`) || message.includes('before secure TLS connection was established'))) + ); +} // Fallback message patterns for errors that don't carry an errno. node-fetch surfaces // "socket hang up" as a generic FetchError, and "Premature close" when the response @@ -81,10 +118,13 @@ export default class HttpRetryPolicy implements IRetryPolicy { @@ -204,15 +244,14 @@ export default class HttpRetryPolicy implements IRetryPolicy= 500 && statusCode !== 501); - - return result; + protected isRetryable(details: HttpTransactionDetails): boolean { + return isRetryableStatus(details, true); } protected getRetryAfterHeader({ response }: HttpTransactionDetails, delayMin: number): number | undefined { diff --git a/lib/connection/connections/UnsentRequestRetryPolicy.ts b/lib/connection/connections/UnsentRequestRetryPolicy.ts index c2d11ce2..0ecb97c5 100644 --- a/lib/connection/connections/UnsentRequestRetryPolicy.ts +++ b/lib/connection/connections/UnsentRequestRetryPolicy.ts @@ -1,15 +1,11 @@ -import HttpRetryPolicy from './HttpRetryPolicy'; +import HttpRetryPolicy, { isConnectError, isRetryableStatus } from './HttpRetryPolicy'; import { HttpTransactionDetails } from '../contracts/IConnectionProvider'; import { RetryableOperation } from '../contracts/IRetryPolicy'; import RetryError from '../../errors/RetryError'; -// Raised before a connection exists, so no request bytes were written. Failures on a connected -// socket (ECONNRESET, EPIPE, timeouts, ...) are excluded: the server may already have the request. -// ENOTFOUND is permanent, as in HttpRetryPolicy. -const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'EAI_AGAIN']); - -// For operations that are not safe to repeat: retries only requests that never left the client, and a -// 503 with `Retry-After`, where the server asks for a retry without running the request (as in the kernel). +// For operations that are not safe to repeat: retries only requests the server didn't run, as the kernel's +// non-idempotent strategy does: a failure to connect, a 429, or a 503 with `Retry-After`. Failures on a +// connected socket (ECONNRESET, EPIPE, timeouts, ...) are not retried: the server may already have the request. export default class UnsentRequestRetryPolicy extends HttpRetryPolicy { public async invokeWithRetry(operation: RetryableOperation): Promise { try { @@ -23,12 +19,11 @@ export default class UnsentRequestRetryPolicy extends HttpRetryPolicy { } } - protected isRetryable({ response }: HttpTransactionDetails): boolean { - return response.status === 503 && response.headers.has('Retry-After'); + protected isRetryable(details: HttpTransactionDetails): boolean { + return isRetryableStatus(details, false); } protected isRetryableNetworkError(error: unknown): boolean { - const code = (error as { code?: unknown } | undefined)?.code; - return typeof code === 'string' && UNSENT_REQUEST_ERROR_CODES.has(code); + return isConnectError(error); } } diff --git a/lib/connection/contracts/IConnectionProvider.ts b/lib/connection/contracts/IConnectionProvider.ts index a81bbe82..ff24f552 100644 --- a/lib/connection/contracts/IConnectionProvider.ts +++ b/lib/connection/contracts/IConnectionProvider.ts @@ -1,6 +1,6 @@ import http from 'http'; import { HeadersInit, Request, Response } from 'node-fetch'; -import IRetryPolicy from './IRetryPolicy'; +import IRetryPolicy, { RetryRequestClass } from './IRetryPolicy'; export interface HttpTransactionDetails { request: Request; @@ -18,5 +18,5 @@ export default interface IConnectionProvider { setHeaders(headers: HeadersInit): void; - getRetryPolicy(): Promise>; + getRetryPolicy(requestClass?: RetryRequestClass): Promise>; } diff --git a/lib/connection/contracts/IRetryPolicy.ts b/lib/connection/contracts/IRetryPolicy.ts index 900671f0..ff022950 100644 --- a/lib/connection/contracts/IRetryPolicy.ts +++ b/lib/connection/contracts/IRetryPolicy.ts @@ -9,6 +9,9 @@ export type ShouldRetryResult = export type RetryableOperation = () => Promise; +// Request classes whose retried network errors differ (see HttpRetryPolicy) +export type RetryRequestClass = 'rpc' | 'cloudFetch'; + export default interface IRetryPolicy { shouldRetry(details: R): Promise; diff --git a/lib/hive/Commands/BaseCommand.ts b/lib/hive/Commands/BaseCommand.ts index 0db2b029..81d46b46 100644 --- a/lib/hive/Commands/BaseCommand.ts +++ b/lib/hive/Commands/BaseCommand.ts @@ -27,9 +27,10 @@ export default abstract class BaseCommand { ) { statusCode = error.payload.response.status; } - // After network errors, name the last one's code (e.g. ECONNREFUSED) instead of a status - const code = (error.payload as { code?: unknown } | null | undefined)?.code; - const cause = statusCode ?? (typeof code === 'string' && code ? code : 'Error'); + // After network errors, name the last one's code (e.g. ECONNREFUSED) or type (request-timeout) + const { code, type } = (error.payload ?? {}) as { code?: unknown; type?: unknown }; + const networkCause = [code, type].find((value) => typeof value === 'string' && value); + const cause = statusCode ?? networkCause ?? 'Error'; switch (error.errorCode) { case RetryErrorCode.AttemptsExceeded: diff --git a/lib/result/CloudFetchResultHandler.ts b/lib/result/CloudFetchResultHandler.ts index 1e30363b..9be28825 100644 --- a/lib/result/CloudFetchResultHandler.ts +++ b/lib/result/CloudFetchResultHandler.ts @@ -156,7 +156,7 @@ export default class CloudFetchResultHandler implements IResultsProvider { const client = new DBSQLClient(); const fetchStub = sinon - .stub(FeatureFlagCache.prototype as any, 'fetchWithRetry') + .stub(FeatureFlagCache.prototype as any, 'sendRequest') .rejects(new Error('Feature flag fetch failed')); try { diff --git a/tests/unit/.stubs/ConnectionProviderStub.ts b/tests/unit/.stubs/ConnectionProviderStub.ts index 20b54dfd..336a17d5 100644 --- a/tests/unit/.stubs/ConnectionProviderStub.ts +++ b/tests/unit/.stubs/ConnectionProviderStub.ts @@ -1,7 +1,7 @@ import http from 'http'; import { HeadersInit } from 'node-fetch'; import IConnectionProvider, { HttpTransactionDetails } from '../../../lib/connection/contracts/IConnectionProvider'; -import IRetryPolicy from '../../../lib/connection/contracts/IRetryPolicy'; +import IRetryPolicy, { RetryRequestClass } from '../../../lib/connection/contracts/IRetryPolicy'; import NullRetryPolicy from '../../../lib/connection/connections/NullRetryPolicy'; export default class ConnectionProviderStub implements IConnectionProvider { @@ -19,7 +19,8 @@ export default class ConnectionProviderStub implements IConnectionProvider { this.headers = headers; } - public async getRetryPolicy(): Promise> { + // eslint-disable-next-line @typescript-eslint/no-unused-vars + public async getRetryPolicy(requestClass?: RetryRequestClass): Promise> { return new NullRetryPolicy(); } } diff --git a/tests/unit/DBSQLClient.test.ts b/tests/unit/DBSQLClient.test.ts index b8e007ba..507a06ad 100644 --- a/tests/unit/DBSQLClient.test.ts +++ b/tests/unit/DBSQLClient.test.ts @@ -1081,7 +1081,7 @@ describe('DBSQLClient feature flag lifecycle', () => { .stub(DatabricksOAuth.prototype, 'authenticate') .resolves({ Authorization: 'Bearer oauth-token' }); const fetch = sinon - .stub(FeatureFlagCache.prototype as any, 'fetchWithRetry') + .stub(FeatureFlagCache.prototype as any, 'sendRequest') .resolves(new Response(JSON.stringify({ flags: [{ name: 'sampleLimit', value: '42' }] }))); const connect = sinon.stub(ThriftBackend.prototype, 'connect').resolves(); sinon.stub(ThriftBackend.prototype, 'close').resolves(); @@ -1114,7 +1114,7 @@ describe('DBSQLClient feature flag lifecycle', () => { } it('continues connecting when the flag GET fails', async () => { - const fetch = sinon.stub(FeatureFlagCache.prototype as any, 'fetchWithRetry').rejects(new Error('offline')); + const fetch = sinon.stub(FeatureFlagCache.prototype as any, 'sendRequest').rejects(new Error('offline')); const connect = sinon.stub(ThriftBackend.prototype, 'connect').resolves(); sinon.stub(ThriftBackend.prototype, 'close').resolves(); const client = new DBSQLClient(); diff --git a/tests/unit/connection/connections/HttpRetryPolicy.test.ts b/tests/unit/connection/connections/HttpRetryPolicy.test.ts index 60a8b4a4..1bbf678d 100644 --- a/tests/unit/connection/connections/HttpRetryPolicy.test.ts +++ b/tests/unit/connection/connections/HttpRetryPolicy.test.ts @@ -60,22 +60,16 @@ describe('HttpRetryPolicy', () => { }; } - // Status codes below 100 can be retried - for (let status = 1; status < 100; status += 1) { - expect(policy['isRetryable'](createStub(status))).to.be.true; + // As the kernel's idempotent strategy: every status except success, redirects and these rejections + const nonRetryable = [400, 401, 403, 404, 405, 409, 410, 411, 412, 413, 414, 415, 416]; + for (let status = 1; status < 1000; status += 1) { + const expectedResult = !(status >= 200 && status < 400) && !nonRetryable.includes(status); + expect(policy['isRetryable'](createStub(status)), `status ${status}`).to.equal(expectedResult); } - // Status codes between 100 (including) and 500 (excluding) should not be retried - // The only exceptions are 408 (Request Timeout) and 429 (Too many requests) - for (let status = 100; status < 500; status += 1) { - const expectedResult = status === 408 || status === 429; - expect(policy['isRetryable'](createStub(status))).to.equal(expectedResult); - } - - // Status codes above 500 can be retried, except for 501 - for (let status = 500; status < 1000; status += 1) { - const expectedResult = status === 501 ? false : true; - expect(policy['isRetryable'](createStub(status))).to.equal(expectedResult); + // Statuses whose class changed to match the kernel + for (const status of [402, 406, 407, 417, 422, 425, 431, 451, 501]) { + expect(policy['isRetryable'](createStub(status)), `status ${status}`).to.be.true; } }); @@ -440,11 +434,42 @@ describe('HttpRetryPolicy', () => { } as unknown as NodeJS.ErrnoException); } - function createPolicy() { - return new HttpRetryPolicy(new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 5 })); + function createPolicy(requestClass?: 'rpc' | 'cloudFetch') { + return new HttpRetryPolicy( + new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 5 }), + requestClass, + ); + } + + async function expectThrownAtOnce(error: Error, requestClass?: 'rpc' | 'cloudFetch') { + const operation = sinon.stub().rejects(error); + try { + await createPolicy(requestClass).invokeWithRetry(operation); + expect.fail('It should throw the error'); + } catch (e) { + if (e instanceof AssertionError) { + throw e; + } + expect(e).to.equal(error); + } + expect(operation.callCount).to.equal(1); + } + + async function expectRetried(error: Error, requestClass?: 'rpc' | 'cloudFetch') { + const operation = sinon.stub(); + operation.onCall(0).rejects(error); + operation.onCall(1).resolves({ + request: new Request('http://localhost'), + response: new Response(undefined, { status: 200 }), + }); + + const result = await createPolicy(requestClass).invokeWithRetry(operation); + + expect(result.response.status).to.equal(200); + expect(operation.callCount).to.equal(2); } - // Certificate verification failures and a host name that doesn't resolve + // Certificate verification failures, and for RPCs a host name that doesn't resolve for (const code of [ 'DEPTH_ZERO_SELF_SIGNED_CERT', 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', @@ -453,21 +478,7 @@ describe('HttpRetryPolicy', () => { 'ERR_TLS_CERT_ALTNAME_INVALID', 'ENOTFOUND', ]) { - it(`throws ${code} at once, without retrying`, async () => { - const error = systemError(code); - const operation = sinon.stub().rejects(error); - - try { - await createPolicy().invokeWithRetry(operation); - expect.fail('It should throw the error'); - } catch (e) { - if (e instanceof AssertionError) { - throw e; - } - expect(e).to.equal(error); - } - expect(operation.callCount).to.equal(1); - }); + it(`throws ${code} at once on an RPC, without retrying`, () => expectThrownAtOnce(systemError(code))); } // Transient connect/DNS failures, a TLS alert sent by the server, and unavailable revocation data @@ -478,20 +489,17 @@ describe('HttpRetryPolicy', () => { 'ERR_SSL_TLSV13_ALERT_CERTIFICATE_REQUIRED', 'UNABLE_TO_GET_CRL', ]) { - it(`retries ${code}`, async () => { - const operation = sinon.stub(); - operation.onCall(0).rejects(systemError(code)); - operation.onCall(1).resolves({ - request: new Request('http://localhost'), - response: new Response(undefined, { status: 200 }), - }); - - const result = await createPolicy().invokeWithRetry(operation); - - expect(result.response.status).to.equal(200); - expect(operation.callCount).to.equal(2); - }); + it(`retries ${code}`, () => expectRetried(systemError(code))); } + + it('retries a request timeout on an RPC, as the kernel', () => + expectRetried(new FetchError('network timeout at: https://example.com', 'request-timeout'))); + + // As the kernel's CloudFetch chunk retry: everything but certificate failures + it('retries ENOTFOUND on a CloudFetch download', () => expectRetried(systemError('ENOTFOUND'), 'cloudFetch')); + + it('throws a certificate failure at once on a CloudFetch download', () => + expectThrownAtOnce(systemError('CERT_HAS_EXPIRED'), 'cloudFetch')); }); it('should cancel a pending backoff sleep when the signal is aborted', async () => { diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index c5849ec1..6a12485a 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -23,6 +23,7 @@ const REPLY = 2; type ServerAction = | 'reply' | 'close-after-request' + | 'stall' | 'status-408' | 'status-429' | 'status-503' @@ -45,6 +46,8 @@ async function startServer(actions: ServerAction[]): Promise { if (action === 'close-after-request') { // The full request has been received; drop the connection without a response req.socket.destroy(); + } else if (action === 'stall') { + // Never respond; the client's request timeout ends it } else if (action === 'status-503-retry-after') { res.writeHead(503, { 'Retry-After': '1' }); res.end(); @@ -106,7 +109,8 @@ class RefusingAgent extends http.Agent { async function createThriftConnection(port: number, agent: http.Agent): Promise { const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); - const httpConnection = new HttpConnection({ host: '127.0.0.1', port, path: '/' }, context); + // A short socketTimeout, so a stalled response becomes a request timeout quickly + const httpConnection = new HttpConnection({ host: '127.0.0.1', port, path: '/', socketTimeout: 300 }, context); httpConnection['httpAgent'] = agent; return httpConnection.getThriftConnection(); } @@ -188,7 +192,7 @@ describe('ThriftHttpConnection retries', () => { expect(server.requests).to.equal(2); }); - for (const status of ['status-503', 'status-429', 'status-408'] as const) { + for (const status of ['status-503', 'status-408'] as const) { it(`does not retry ExecuteStatement on an HTTP error status (${status})`, async () => { const { server, connection } = await setup([status], 0); @@ -200,14 +204,16 @@ describe('ThriftHttpConnection retries', () => { }); } - it('retries ExecuteStatement on a 503 with Retry-After, which the server did not run', async () => { - const { server, connection } = await setup(['status-503-retry-after', 'reply'], 0); + for (const status of ['status-503-retry-after', 'status-429'] as const) { + it(`retries ExecuteStatement when the server rejects it without running it (${status})`, async () => { + const { server, connection } = await setup([status, 'reply'], 0); - const result = await send(connection, 'ExecuteStatement', 1); + const result = await send(connection, 'ExecuteStatement', 1); - expect(result).to.equal('ExecuteStatement result'); - expect(server.requests).to.equal(2); - }); + expect(result).to.equal('ExecuteStatement result'); + expect(server.requests).to.equal(2); + }); + } for (const status of ['status-503', 'status-429', 'status-408'] as const) { it(`retries OpenSession on a transient HTTP error (${status})`, async () => { @@ -220,6 +226,24 @@ describe('ThriftHttpConnection retries', () => { }); } + it('retries OpenSession after a request timeout, as the kernel', async () => { + const { server, connection } = await setup(['stall', 'reply'], 0); + + const result = await send(connection, 'OpenSession', 1); + + expect(result).to.equal('OpenSession result'); + expect(server.requests).to.equal(2); + }); + + it('does not retry ExecuteStatement after a request timeout', async () => { + const { server, connection } = await setup(['stall', 'reply'], 0); + + const error = await expectRejection(send(connection, 'ExecuteStatement', 1)); + + expect(error.message).to.equal('Request timed out'); + expect(server.requests).to.equal(1); + }); + it('retries OpenSession when the connection is refused', async () => { const { server, agent, connection } = await setup(['reply'], 1); diff --git a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts index fd62aabd..610b359c 100644 --- a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts +++ b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts @@ -33,10 +33,23 @@ async function expectRejection(promise: Promise): Promise { } describe('UnsentRequestRetryPolicy', () => { - for (const code of ['ECONNREFUSED', 'EAI_AGAIN']) { - it(`retries ${code}, which is raised before the request is sent`, async () => { + // Failures to connect (the kernel's `is_connect`): the request never left the client + const connectErrors: Array<[string, Error]> = [ + ['ECONNREFUSED', systemError('ECONNREFUSED', 'connect ECONNREFUSED 10.0.0.1:443')], + ['EAI_AGAIN', systemError('EAI_AGAIN', 'getaddrinfo EAI_AGAIN example.com')], + ['a connect timeout', systemError('ETIMEDOUT', 'connect ETIMEDOUT 10.0.0.1:443')], + ['an unreachable host', systemError('EHOSTUNREACH', 'connect EHOSTUNREACH 10.0.0.1:443')], + ['an unreachable network', systemError('ENETUNREACH', 'connect ENETUNREACH 10.0.0.1:443')], + [ + 'a socket closed during the TLS handshake', + systemError('ECONNRESET', 'Client network socket disconnected before secure TLS connection was established'), + ], + ]; + + for (const [label, error] of connectErrors) { + it(`retries ${label}, which is raised before the request is sent`, async () => { const operation = sinon.stub(); - operation.onCall(0).rejects(systemError(code)); + operation.onCall(0).rejects(error); operation.onCall(1).resolves(okResponse()); const result = await createPolicy().invokeWithRetry(operation); @@ -49,11 +62,12 @@ describe('UnsentRequestRetryPolicy', () => { const sentRequestErrors: Array<[string, Error]> = [ ['ECONNRESET (socket hang up)', systemError('ECONNRESET', 'socket hang up')], ['EPIPE', systemError('EPIPE')], - ['ETIMEDOUT', systemError('ETIMEDOUT')], - ['EHOSTUNREACH', systemError('EHOSTUNREACH')], + ['ETIMEDOUT on a connected socket', systemError('ETIMEDOUT', 'read ETIMEDOUT')], + ['EHOSTUNREACH on a connected socket', systemError('EHOSTUNREACH', 'write EHOSTUNREACH')], ['a body timeout', new FetchError('Response timeout while trying to fetch', 'body-timeout')], ['a premature close', new Error('Premature close')], ['ENOTFOUND, a host name that does not resolve', systemError('ENOTFOUND')], + ['a request timeout', new FetchError('network timeout at: https://example.com', 'request-timeout')], ]; for (const [label, error] of sentRequestErrors) { @@ -70,9 +84,12 @@ describe('UnsentRequestRetryPolicy', () => { } for (const [status, headers] of [ - [429, {}], [500, {}], + [501, {}], + [502, {}], [503, {}], + [504, {}], + [408, {}], [500, { 'Retry-After': '1' }], ] as const) { it(`does not retry an HTTP ${status} response ${ @@ -88,16 +105,22 @@ describe('UnsentRequestRetryPolicy', () => { }); } - it('retries an HTTP 503 with Retry-After, which the server did not run', async () => { - const operation = sinon.stub(); - operation.onCall(0).resolves(statusResponse(503, { 'Retry-After': '1' })); - operation.onCall(1).resolves(okResponse()); + for (const [label, rejection] of [ + ['an HTTP 503 with Retry-After', statusResponse(503, { 'Retry-After': '1' })], + ['an HTTP 429', statusResponse(429)], + ['an HTTP 429 with Retry-After', statusResponse(429, { 'Retry-After': '1' })], + ] as const) { + it(`retries ${label}, which the server did not run`, async () => { + const operation = sinon.stub(); + operation.onCall(0).resolves(rejection); + operation.onCall(1).resolves(okResponse()); - const result = await createPolicy().invokeWithRetry(operation); + const result = await createPolicy().invokeWithRetry(operation); - expect(result.response.status).to.equal(200); - expect(operation.callCount).to.equal(2); - }); + expect(result.response.status).to.equal(200); + expect(operation.callCount).to.equal(2); + }); + } it('waits the Retry-After delay before retrying a 503, with the default config', async () => { const policy = new UnsentRequestRetryPolicy(new ClientContextStub()); diff --git a/tests/unit/hive/commands/BaseCommand.test.ts b/tests/unit/hive/commands/BaseCommand.test.ts index df372a05..49574ad6 100644 --- a/tests/unit/hive/commands/BaseCommand.test.ts +++ b/tests/unit/hive/commands/BaseCommand.test.ts @@ -211,6 +211,29 @@ describe('BaseCommand', () => { }); }); + it('should name a request timeout once retries run out', async () => { + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); + const command = new CustomCommand( + new ThriftClientStub(context, async () => { + throw new FetchError('network timeout at: http://localhost/', 'request-timeout'); + }), + context, + ); + + try { + await command.execute({}); + expect.fail('It should throw an error'); + } catch (error) { + if (error instanceof AssertionError || !(error instanceof Error)) { + throw error; + } + expect(error).to.be.instanceof(HiveDriverError); + expect(error.message).to.equal( + 'Hive driver: request-timeout when connecting to resource. Max retry count exceeded.', + ); + } + }); + it('should name the network error code once retries run out', async () => { const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); const command = new CustomCommand( diff --git a/tests/unit/result/CloudFetchResultHandler.test.ts b/tests/unit/result/CloudFetchResultHandler.test.ts index 47a92143..2ded0970 100644 --- a/tests/unit/result/CloudFetchResultHandler.test.ts +++ b/tests/unit/result/CloudFetchResultHandler.test.ts @@ -5,7 +5,7 @@ import { AddressInfo } from 'net'; import Int64 from 'node-int64'; import { compressFrameSync } from 'lz4-napi'; import { Request, Response } from 'node-fetch'; -import { ShouldRetryResult } from '../../../lib/connection/contracts/IRetryPolicy'; +import { RetryRequestClass, ShouldRetryResult } from '../../../lib/connection/contracts/IRetryPolicy'; import { HttpTransactionDetails } from '../../../lib/connection/contracts/IConnectionProvider'; import { LogLevel } from '../../../lib/contracts/IDBSQLLogger'; import CloudFetchResultHandler from '../../../lib/result/CloudFetchResultHandler'; @@ -750,7 +750,9 @@ describe('CloudFetchResultHandler', () => { // across actual attempts, backoff sleeps and sockets. function createHandler(fileLink: string, configOverrides: Partial, agent?: http.Agent) { const context = new BaseClientContextStub({ cloudFetchConcurrentDownloads: 1, ...configOverrides }); - sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); + sinon + .stub(context.connectionProvider, 'getRetryPolicy') + .callsFake(async (requestClass?: RetryRequestClass) => new HttpRetryPolicy(context, requestClass)); if (agent) { sinon.stub(context.connectionProvider, 'getAgent').resolves(agent); } @@ -792,19 +794,48 @@ describe('CloudFetchResultHandler', () => { return { ...outcome, elapsedMs: Date.now() - startedAt }; } - it('should not leak the presigned URL from a network error that is not retried', async () => { - // Every lookup fails as for a host that doesn't resolve, which is thrown without retrying - const agent = new http.Agent({ - lookup: (hostname: string, options: unknown, callback: Function) => - callback(Object.assign(new Error(`getaddrinfo ENOTFOUND ${hostname}`), { code: 'ENOTFOUND' })), + // An agent whose every connection fails with `code` before any request is sent + function failingAgent(code: string, onLookup = () => {}) { + return new http.Agent({ + lookup: (hostname: string, options: unknown, callback: Function) => { + onLookup(); + callback(Object.assign(new Error(`${code} ${hostname}`), { code })); + }, } as unknown as http.AgentOptions); + } + + it('should not leak the presigned URL from a network error that is not retried', async () => { + // A certificate failure is thrown without retrying + const agent = failingAgent('CERT_HAS_EXPIRED'); const handler = createHandler('http://storage.example.invalid/chunk?sig=SECRET_SIGNATURE', {}, agent); const { error } = await fetchWithin(handler, 1500); agent.destroy(); - expect(error.code).to.be.equal('ENOTFOUND'); - expect(error.message).to.be.equal('CloudFetch result download failed: network error ENOTFOUND'); + expect(error.code).to.be.equal('CERT_HAS_EXPIRED'); + expect(error.message).to.be.equal('CloudFetch result download failed: network error CERT_HAS_EXPIRED'); + expect(error.stack).to.not.contain('SECRET_SIGNATURE'); + }); + + it('should retry a download whose host does not resolve, as the kernel', async () => { + let lookups = 0; + const agent = failingAgent('ENOTFOUND', () => { + lookups += 1; + }); + const handler = createHandler( + 'http://storage.example.invalid/chunk?sig=SECRET_SIGNATURE', + { retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }, + agent, + ); + + const { error } = await fetchWithin(handler, 1500); + agent.destroy(); + + expect(lookups).to.be.equal(3); + expect(error).to.be.instanceOf(RetryError); + expect(error.message).to.be.equal( + 'CloudFetch result download failed: Max retry count exceeded (last error: network error ENOTFOUND)', + ); expect(error.stack).to.not.contain('SECRET_SIGNATURE'); }); diff --git a/tests/unit/telemetry/FeatureFlagCache.test.ts b/tests/unit/telemetry/FeatureFlagCache.test.ts index da179f6f..05fdbbeb 100644 --- a/tests/unit/telemetry/FeatureFlagCache.test.ts +++ b/tests/unit/telemetry/FeatureFlagCache.test.ts @@ -263,12 +263,12 @@ describe('FeatureFlagCache', () => { const host = 'test-host.databricks.com'; // Stub the network seam so the test is deterministic. The real - // `fetchWithRetry` makes an HTTP call with a 10s timeout to the + // `sendRequest` makes an HTTP call with a 10s timeout to the // (bogus) host; under mocha's 2s default this passed only when the // DNS failure happened to resolve quickly — flaky across runners / // Node versions (it timed out on Node 14/16/18 in CI). The behavior // under test is just that `fetchFeatureFlags` resolves to `false`. - const fetchStub = sinon.stub(cache as any, 'fetchWithRetry').rejects(new Error('network disabled in test')); + const fetchStub = sinon.stub(cache as any, 'sendRequest').rejects(new Error('network disabled in test')); cache.getOrCreateContext(host); const result = await cache.getBoolean(host, 'flag'); @@ -277,35 +277,28 @@ describe('FeatureFlagCache', () => { fetchStub.restore(); }); - it('should not wait for Retry-After before retrying the flags request', async () => { - // connect() waits for the flags, so a Retry-After from the endpoint must not stretch the retries + it('should make a single attempt, even when the endpoint asks for a retry', async () => { + // As in the kernel: connect() waits for the flags, and a failure falls back to the defaults clock.restore(); let hits = 0; const server = http.createServer((req, res) => { hits += 1; - if (hits === 1) { - res.writeHead(503, { 'Retry-After': '60' }); - res.end(); - } else { - res.writeHead(200, { 'Content-Type': 'application/json' }); - res.end(JSON.stringify({ flags: [] })); - } + res.writeHead(503, { 'Retry-After': '1' }); + res.end(); }); await new Promise((resolve) => { server.listen(0, '127.0.0.1', resolve); }); try { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 60_000 }); + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2 }); sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); const cache = new FeatureFlagCache(context); const { port } = server.address() as AddressInfo; - const started = Date.now(); - const response = await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET' }); + const response = await (cache as any).sendRequest(`http://127.0.0.1:${port}/`, { method: 'GET' }); - expect(response.status).to.equal(200); - expect(hits).to.equal(2); - expect(Date.now() - started).to.be.lt(1000); + expect(response.status).to.equal(503); + expect(hits).to.equal(1); } finally { server.closeAllConnections(); server.close(); @@ -385,7 +378,7 @@ describe('FeatureFlagCache', () => { customHeaders: { 'x-databricks-org-id': '12345678901234' }, } as any); const cache = new FeatureFlagCache(context); - const stub = sinon.stub(cache as any, 'fetchWithRetry').returns(makeJsonResponse({ flags: [] })); + const stub = sinon.stub(cache as any, 'sendRequest').returns(makeJsonResponse({ flags: [] })); await (cache as any).fetchFeatureFlags('host.example.com'); @@ -398,7 +391,7 @@ describe('FeatureFlagCache', () => { it('does not set x-databricks-org-id when customHeaders is empty', async () => { const context = new ClientContextStub(); const cache = new FeatureFlagCache(context); - const stub = sinon.stub(cache as any, 'fetchWithRetry').returns(makeJsonResponse({ flags: [] })); + const stub = sinon.stub(cache as any, 'sendRequest').returns(makeJsonResponse({ flags: [] })); await (cache as any).fetchFeatureFlags('host.example.com'); @@ -430,7 +423,7 @@ describe('FeatureFlagCache', () => { ]; const cache = new FeatureFlagCache(new ClientContextStub()); cache.getOrCreateContext('test-host'); - const fetchStub = sinon.stub(cache as any, 'fetchWithRetry').resolves( + const fetchStub = sinon.stub(cache as any, 'sendRequest').resolves( new Response( JSON.stringify({ flags: cases.map(([, value], index) => ({ name: String(index), value })), @@ -460,8 +453,8 @@ describe('FeatureFlagCache', () => { ttl_seconds: 60, }), ); - const firstFetch = sinon.stub(first as any, 'fetchWithRetry').resolves(response('true')); - const currentFetch = sinon.stub(current as any, 'fetchWithRetry').resolves(response('FALSE')); + const firstFetch = sinon.stub(first as any, 'sendRequest').resolves(response('true')); + const currentFetch = sinon.stub(current as any, 'sendRequest').resolves(response('FALSE')); expect(await Promise.all([first.getBoolean('host-a', 'flag'), current.getBoolean('host-b', 'flag')])).to.deep.equal( [true, true], ); From b3b240d2a710dacafd7b4f7b831f10e6926b570c Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 04:52:56 +0000 Subject: [PATCH 04/15] fix: keep "Request timed out" when every retried attempt times out With request timeouts retried on idempotent RPCs, an OpenSession that timed out on every attempt surfaced as a retry-exhaustion HiveDriverError instead of the "Request timed out" error it has always raised (the e2e Timeouts tests pin it). Unwrap the last timeout before the existing conversion, so the error stays the same, and drop the BaseCommand fallback that named it. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- .../connections/ThriftHttpConnection.ts | 9 ++++++++ lib/hive/Commands/BaseCommand.ts | 7 +++--- .../ThriftHttpConnectionRetry.test.ts | 9 ++++++++ tests/unit/hive/commands/BaseCommand.test.ts | 23 ------------------- 5 files changed, 22 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d2ce645e..1f183a82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException`, network error or `Request timed out`, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout` (if every attempt times out, the error is still `Request timed out`). `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/connection/connections/ThriftHttpConnection.ts b/lib/connection/connections/ThriftHttpConnection.ts index 4cd64ff4..ea514e64 100644 --- a/lib/connection/connections/ThriftHttpConnection.ts +++ b/lib/connection/connections/ThriftHttpConnection.ts @@ -12,6 +12,7 @@ import InputBufferUnderrunError from 'thrift/lib/nodejs/lib/thrift/input_buffer_ import IRetryPolicy from '../contracts/IRetryPolicy'; import { HttpTransactionDetails } from '../contracts/IConnectionProvider'; import NullRetryPolicy from './NullRetryPolicy'; +import RetryError from '../../errors/RetryError'; export class THTTPException extends Thrift.TApplicationException { public readonly statusCode: unknown; @@ -201,6 +202,14 @@ export default class ThriftHttpConnection extends EventEmitter { )(buffer); }) .catch((error) => { + // A request that timed out on every attempt still fails as a timeout + if ( + error instanceof RetryError && + error.payload instanceof FetchError && + error.payload.type === 'request-timeout' + ) { + error = error.payload; + } if (error instanceof FetchError) { if (error.type === 'request-timeout') { error = new Thrift.TApplicationException( diff --git a/lib/hive/Commands/BaseCommand.ts b/lib/hive/Commands/BaseCommand.ts index 81d46b46..0db2b029 100644 --- a/lib/hive/Commands/BaseCommand.ts +++ b/lib/hive/Commands/BaseCommand.ts @@ -27,10 +27,9 @@ export default abstract class BaseCommand { ) { statusCode = error.payload.response.status; } - // After network errors, name the last one's code (e.g. ECONNREFUSED) or type (request-timeout) - const { code, type } = (error.payload ?? {}) as { code?: unknown; type?: unknown }; - const networkCause = [code, type].find((value) => typeof value === 'string' && value); - const cause = statusCode ?? networkCause ?? 'Error'; + // After network errors, name the last one's code (e.g. ECONNREFUSED) instead of a status + const code = (error.payload as { code?: unknown } | null | undefined)?.code; + const cause = statusCode ?? (typeof code === 'string' && code ? code : 'Error'); switch (error.errorCode) { case RetryErrorCode.AttemptsExceeded: diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 6a12485a..0f52084a 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -235,6 +235,15 @@ describe('ThriftHttpConnection retries', () => { expect(server.requests).to.equal(2); }); + it('fails an OpenSession that times out on every attempt with "Request timed out"', async () => { + const { server, connection } = await setup(['stall', 'stall', 'stall'], 0); + + const error = await expectRejection(send(connection, 'OpenSession', 1)); + + expect(error.message).to.equal('Request timed out'); + expect(server.requests).to.equal(3); + }); + it('does not retry ExecuteStatement after a request timeout', async () => { const { server, connection } = await setup(['stall', 'reply'], 0); diff --git a/tests/unit/hive/commands/BaseCommand.test.ts b/tests/unit/hive/commands/BaseCommand.test.ts index 49574ad6..df372a05 100644 --- a/tests/unit/hive/commands/BaseCommand.test.ts +++ b/tests/unit/hive/commands/BaseCommand.test.ts @@ -211,29 +211,6 @@ describe('BaseCommand', () => { }); }); - it('should name a request timeout once retries run out', async () => { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); - const command = new CustomCommand( - new ThriftClientStub(context, async () => { - throw new FetchError('network timeout at: http://localhost/', 'request-timeout'); - }), - context, - ); - - try { - await command.execute({}); - expect.fail('It should throw an error'); - } catch (error) { - if (error instanceof AssertionError || !(error instanceof Error)) { - throw error; - } - expect(error).to.be.instanceof(HiveDriverError); - expect(error.message).to.equal( - 'Hive driver: request-timeout when connecting to resource. Max retry count exceeded.', - ); - } - }); - it('should name the network error code once retries run out', async () => { const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); const command = new CustomCommand( From 111ee193fc2125f1ae7f66224f3e8dd8a4fd377a Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 05:04:21 +0000 Subject: [PATCH 05/15] test: pin "Request timed out" when the first timeout uses up the budget With the defaults (socketTimeout and retriesTimeout both 15 minutes) a timed-out request has no time left for a retry. Test that it still fails with "Request timed out", and say in the CHANGELOG when the timeout retry applies. Note why the flags single-attempt test stubs a real retry policy. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- .../ThriftHttpConnectionRetry.test.ts | 22 +++++++++++++++---- tests/unit/telemetry/FeatureFlagCache.test.ts | 1 + 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f183a82..ad3e0ef2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout` (if every attempt times out, the error is still `Request timed out`). `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 0f52084a..1943fc09 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -4,6 +4,7 @@ import { expect } from 'chai'; import { FetchError } from 'node-fetch'; import HttpConnection from '../../../../lib/connection/connections/HttpConnection'; import ThriftHttpConnection, { THTTPException } from '../../../../lib/connection/connections/ThriftHttpConnection'; +import { ClientConfig } from '../../../../lib/contracts/IClientContext'; import ClientContextStub from '../../.stubs/ClientContextStub'; // Header-only Thrift binary messages: enough for method-name detection and for the fake `recv_*` below. @@ -107,8 +108,12 @@ class RefusingAgent extends http.Agent { } } -async function createThriftConnection(port: number, agent: http.Agent): Promise { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); +async function createThriftConnection( + port: number, + agent: http.Agent, + config: Partial, +): Promise { + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3, ...config }); // A short socketTimeout, so a stalled response becomes a request timeout quickly const httpConnection = new HttpConnection({ host: '127.0.0.1', port, path: '/', socketTimeout: 300 }, context); httpConnection['httpAgent'] = agent; @@ -148,7 +153,7 @@ describe('ThriftHttpConnection retries', () => { cleanup.splice(0).forEach((fn) => fn()); }); - async function setup(actions: ServerAction[], refusals: number) { + async function setup(actions: ServerAction[], refusals: number, config: Partial = {}) { const closedPort = await getClosedPort(); const server = await startServer(actions); const agent = new RefusingAgent(refusals, closedPort); @@ -156,7 +161,7 @@ describe('ThriftHttpConnection retries', () => { () => server.close(), () => agent.destroy(), ); - const connection = await createThriftConnection(server.port, agent); + const connection = await createThriftConnection(server.port, agent, config); return { server, agent, connection }; } @@ -244,6 +249,15 @@ describe('ThriftHttpConnection retries', () => { expect(server.requests).to.equal(3); }); + it('fails with "Request timed out" when the first timeout uses up retriesTimeout, as with the defaults', async () => { + const { server, connection } = await setup(['stall', 'reply'], 0, { retriesTimeout: 300 }); + + const error = await expectRejection(send(connection, 'OpenSession', 1)); + + expect(error.message).to.equal('Request timed out'); + expect(server.requests).to.equal(1); + }); + it('does not retry ExecuteStatement after a request timeout', async () => { const { server, connection } = await setup(['stall', 'reply'], 0); diff --git a/tests/unit/telemetry/FeatureFlagCache.test.ts b/tests/unit/telemetry/FeatureFlagCache.test.ts index 05fdbbeb..7176cdda 100644 --- a/tests/unit/telemetry/FeatureFlagCache.test.ts +++ b/tests/unit/telemetry/FeatureFlagCache.test.ts @@ -291,6 +291,7 @@ describe('FeatureFlagCache', () => { }); try { const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2 }); + // A real retry policy, so the test fails if the fetch goes back to retrying through it sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); const cache = new FeatureFlagCache(context); const { port } = server.address() as AddressInfo; From be73bde3e6062d7e9ea2ec1975c81ca01ec06b76 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 05:20:47 +0000 Subject: [PATCH 06/15] fix: keep the feature-flag fetch's retries, without stretching connect() The flags fetch has retried since main, so it keeps doing so instead of making a single attempt. As before, it backs off without waiting for Retry-After. It also doesn't repeat a request that timed out: request timeouts are now retryable, and each attempt would wait out the 10 s timeout while connect() waits for the flags. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- lib/FeatureFlagCache.ts | 29 ++++++-- .../telemetry/telemetry-integration.test.ts | 2 +- tests/unit/DBSQLClient.test.ts | 4 +- tests/unit/telemetry/FeatureFlagCache.test.ts | 73 +++++++++++++++---- 5 files changed, 86 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad3e0ef2..04fbe45d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). The feature-flag fetch that `connect()` waits for now makes a single attempt, as in the kernel, and falls back to defaults on failure. A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for keeps the exponential backoff instead, and doesn't repeat a request that timed out. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/FeatureFlagCache.ts b/lib/FeatureFlagCache.ts index 7cb713fc..e2e61541 100644 --- a/lib/FeatureFlagCache.ts +++ b/lib/FeatureFlagCache.ts @@ -14,8 +14,9 @@ * limitations under the License. */ -import fetch, { RequestInit, Response, Request } from 'node-fetch'; +import fetch, { RequestInit, Response, Request, FetchError } from 'node-fetch'; import IClientContext from './contracts/IClientContext'; +import RetryError, { RetryErrorCode } from './errors/RetryError'; import { LogLevel } from './contracts/IDBSQLLogger'; import IAuthentication from './connection/contracts/IAuthentication'; import { buildTelemetryUrl, normalizeHeaders } from './telemetry/telemetryUtils'; @@ -195,7 +196,7 @@ export default class FeatureFlagCache { logger.log(LogLevel.debug, `Fetching feature flags from ${endpoint}`); - const response = await this.sendRequest(endpoint, { + const response = await this.fetchWithRetry(endpoint, { method: 'GET', headers, timeout: 10000, @@ -231,11 +232,29 @@ export default class FeatureFlagCache { } } - // A single attempt, as in the kernel: connect() waits for the flags, which fall back to defaults on failure. - private async sendRequest(url: string, init: RequestInit): Promise { + /** Retries transient failures using the connection's retry policy. */ + private async fetchWithRetry(url: string, init: RequestInit): Promise { const connectionProvider = await this.context.getConnectionProvider(); const agent = await connectionProvider.getAgent(url); - return fetch(new Request(url, { agent, ...init })); + const retryPolicy = await connectionProvider.getRetryPolicy(); + const requestConfig: RequestInit = { agent, ...init }; + const result = await retryPolicy.invokeWithRetry(() => { + const request = new Request(url, requestConfig); + // connect() waits for the flags, so back off as usual instead of for as long as `Retry-After` asks, and + // don't repeat a request that already waited out its timeout (the policy rethrows a RetryError as is) + return fetch(request).then( + (response) => { + response.headers.delete('Retry-After'); + return { request, response }; + }, + (error) => { + throw error instanceof FetchError && error.type === 'request-timeout' + ? new RetryError(RetryErrorCode.TimeoutExceeded, error) + : error; + }, + ); + }); + return result.response; } private async getAuthHeaders(): Promise> { diff --git a/tests/e2e/telemetry/telemetry-integration.test.ts b/tests/e2e/telemetry/telemetry-integration.test.ts index 0c487205..6ff02cff 100644 --- a/tests/e2e/telemetry/telemetry-integration.test.ts +++ b/tests/e2e/telemetry/telemetry-integration.test.ts @@ -313,7 +313,7 @@ describe('Telemetry Integration', () => { const client = new DBSQLClient(); const fetchStub = sinon - .stub(FeatureFlagCache.prototype as any, 'sendRequest') + .stub(FeatureFlagCache.prototype as any, 'fetchWithRetry') .rejects(new Error('Feature flag fetch failed')); try { diff --git a/tests/unit/DBSQLClient.test.ts b/tests/unit/DBSQLClient.test.ts index 507a06ad..b8e007ba 100644 --- a/tests/unit/DBSQLClient.test.ts +++ b/tests/unit/DBSQLClient.test.ts @@ -1081,7 +1081,7 @@ describe('DBSQLClient feature flag lifecycle', () => { .stub(DatabricksOAuth.prototype, 'authenticate') .resolves({ Authorization: 'Bearer oauth-token' }); const fetch = sinon - .stub(FeatureFlagCache.prototype as any, 'sendRequest') + .stub(FeatureFlagCache.prototype as any, 'fetchWithRetry') .resolves(new Response(JSON.stringify({ flags: [{ name: 'sampleLimit', value: '42' }] }))); const connect = sinon.stub(ThriftBackend.prototype, 'connect').resolves(); sinon.stub(ThriftBackend.prototype, 'close').resolves(); @@ -1114,7 +1114,7 @@ describe('DBSQLClient feature flag lifecycle', () => { } it('continues connecting when the flag GET fails', async () => { - const fetch = sinon.stub(FeatureFlagCache.prototype as any, 'sendRequest').rejects(new Error('offline')); + const fetch = sinon.stub(FeatureFlagCache.prototype as any, 'fetchWithRetry').rejects(new Error('offline')); const connect = sinon.stub(ThriftBackend.prototype, 'connect').resolves(); sinon.stub(ThriftBackend.prototype, 'close').resolves(); const client = new DBSQLClient(); diff --git a/tests/unit/telemetry/FeatureFlagCache.test.ts b/tests/unit/telemetry/FeatureFlagCache.test.ts index 7176cdda..daf69964 100644 --- a/tests/unit/telemetry/FeatureFlagCache.test.ts +++ b/tests/unit/telemetry/FeatureFlagCache.test.ts @@ -21,6 +21,7 @@ import { AddressInfo } from 'net'; import { Response } from 'node-fetch'; import FeatureFlagCache from '../../../lib/FeatureFlagCache'; import HttpRetryPolicy from '../../../lib/connection/connections/HttpRetryPolicy'; +import RetryError from '../../../lib/errors/RetryError'; import ClientContextStub from '../.stubs/ClientContextStub'; import { LogLevel } from '../../../lib/contracts/IDBSQLLogger'; @@ -263,12 +264,12 @@ describe('FeatureFlagCache', () => { const host = 'test-host.databricks.com'; // Stub the network seam so the test is deterministic. The real - // `sendRequest` makes an HTTP call with a 10s timeout to the + // `fetchWithRetry` makes an HTTP call with a 10s timeout to the // (bogus) host; under mocha's 2s default this passed only when the // DNS failure happened to resolve quickly — flaky across runners / // Node versions (it timed out on Node 14/16/18 in CI). The behavior // under test is just that `fetchFeatureFlags` resolves to `false`. - const fetchStub = sinon.stub(cache as any, 'sendRequest').rejects(new Error('network disabled in test')); + const fetchStub = sinon.stub(cache as any, 'fetchWithRetry').rejects(new Error('network disabled in test')); cache.getOrCreateContext(host); const result = await cache.getBoolean(host, 'flag'); @@ -277,32 +278,74 @@ describe('FeatureFlagCache', () => { fetchStub.restore(); }); - it('should make a single attempt, even when the endpoint asks for a retry', async () => { - // As in the kernel: connect() waits for the flags, and a failure falls back to the defaults + it('should not wait for Retry-After before retrying the flags request', async () => { + // connect() waits for the flags, so a Retry-After from the endpoint must not stretch the retries clock.restore(); let hits = 0; const server = http.createServer((req, res) => { hits += 1; - res.writeHead(503, { 'Retry-After': '1' }); - res.end(); + if (hits === 1) { + res.writeHead(503, { 'Retry-After': '60' }); + res.end(); + } else { + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify({ flags: [] })); + } }); await new Promise((resolve) => { server.listen(0, '127.0.0.1', resolve); }); try { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2 }); - // A real retry policy, so the test fails if the fetch goes back to retrying through it + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 60_000 }); sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); const cache = new FeatureFlagCache(context); const { port } = server.address() as AddressInfo; - const response = await (cache as any).sendRequest(`http://127.0.0.1:${port}/`, { method: 'GET' }); + const started = Date.now(); + const response = await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET' }); - expect(response.status).to.equal(503); + expect(response.status).to.equal(200); + expect(hits).to.equal(2); + expect(Date.now() - started).to.be.lt(1000); + } finally { + server.closeAllConnections(); + server.close(); + clock = sinon.useFakeTimers(); + } + }); + + it('should not repeat a flags request that timed out', async () => { + // Each attempt waits out the request timeout, so retrying it would stretch connect() + clock.restore(); + let hits = 0; + const server = http.createServer(() => { + hits += 1; + }); + await new Promise((resolve) => { + server.listen(0, '127.0.0.1', resolve); + }); + try { + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); + sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); + const cache = new FeatureFlagCache(context); + const { port } = server.address() as AddressInfo; + + let error: unknown; + try { + await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET', timeout: 100 }); + } catch (e) { + error = e; + } + + expect(error).to.be.instanceOf(RetryError); expect(hits).to.equal(1); } finally { server.closeAllConnections(); server.close(); + // Let node-fetch finish the aborted request (it clears its timer) before faking timers again + await new Promise((resolve) => { + setTimeout(resolve, 10); + }); clock = sinon.useFakeTimers(); } }); @@ -379,7 +422,7 @@ describe('FeatureFlagCache', () => { customHeaders: { 'x-databricks-org-id': '12345678901234' }, } as any); const cache = new FeatureFlagCache(context); - const stub = sinon.stub(cache as any, 'sendRequest').returns(makeJsonResponse({ flags: [] })); + const stub = sinon.stub(cache as any, 'fetchWithRetry').returns(makeJsonResponse({ flags: [] })); await (cache as any).fetchFeatureFlags('host.example.com'); @@ -392,7 +435,7 @@ describe('FeatureFlagCache', () => { it('does not set x-databricks-org-id when customHeaders is empty', async () => { const context = new ClientContextStub(); const cache = new FeatureFlagCache(context); - const stub = sinon.stub(cache as any, 'sendRequest').returns(makeJsonResponse({ flags: [] })); + const stub = sinon.stub(cache as any, 'fetchWithRetry').returns(makeJsonResponse({ flags: [] })); await (cache as any).fetchFeatureFlags('host.example.com'); @@ -424,7 +467,7 @@ describe('FeatureFlagCache', () => { ]; const cache = new FeatureFlagCache(new ClientContextStub()); cache.getOrCreateContext('test-host'); - const fetchStub = sinon.stub(cache as any, 'sendRequest').resolves( + const fetchStub = sinon.stub(cache as any, 'fetchWithRetry').resolves( new Response( JSON.stringify({ flags: cases.map(([, value], index) => ({ name: String(index), value })), @@ -454,8 +497,8 @@ describe('FeatureFlagCache', () => { ttl_seconds: 60, }), ); - const firstFetch = sinon.stub(first as any, 'sendRequest').resolves(response('true')); - const currentFetch = sinon.stub(current as any, 'sendRequest').resolves(response('FALSE')); + const firstFetch = sinon.stub(first as any, 'fetchWithRetry').resolves(response('true')); + const currentFetch = sinon.stub(current as any, 'fetchWithRetry').resolves(response('FALSE')); expect(await Promise.all([first.getBoolean('host-a', 'flag'), current.getBoolean('host-b', 'flag')])).to.deep.equal( [true, true], ); From 26abeb42f69975600e94cc377a57563c74389735 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Mon, 16 Nov 2026 05:41:49 +0000 Subject: [PATCH 07/15] fix: cap the feature-flag fetch's total time, and retry a multi-address connect failure The feature-flag fetch keeps the shared retry policy, as it has since main, with one deadline for its attempts, the waits between them and the body read (10 s, what a single attempt could already take). A failing or stalled flags endpoint can no longer hold connect() longer, whatever Retry-After asks and however often a request times out. This replaces the Retry-After strip and the timeout special case. When a host has several addresses and connecting to all of them fails, Node reports an AggregateError that has a code but no message, so node-fetch's reason is empty. Treat that as a connect failure, so ExecuteStatement is retried on a connect timeout or an unreachable host/network there too. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- lib/FeatureFlagCache.ts | 26 ++-- lib/connection/connections/HttpRetryPolicy.ts | 9 +- .../ThriftHttpConnectionRetry.test.ts | 50 ++++++++ .../UnsentRequestRetryPolicy.test.ts | 4 + tests/unit/telemetry/FeatureFlagCache.test.ts | 115 ++++++++---------- 6 files changed, 118 insertions(+), 88 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 04fbe45d..b993b2aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for keeps the exponential backoff instead, and doesn't repeat a request that timed out. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the HTTP statuses and network errors the kernel backend retries, with the exceptions listed in #605; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests and the feature-flag fetch; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total, retries included, and falls back to defaults. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. diff --git a/lib/FeatureFlagCache.ts b/lib/FeatureFlagCache.ts index e2e61541..6fe90247 100644 --- a/lib/FeatureFlagCache.ts +++ b/lib/FeatureFlagCache.ts @@ -14,9 +14,8 @@ * limitations under the License. */ -import fetch, { RequestInit, Response, Request, FetchError } from 'node-fetch'; +import fetch, { RequestInit, Response, Request } from 'node-fetch'; import IClientContext from './contracts/IClientContext'; -import RetryError, { RetryErrorCode } from './errors/RetryError'; import { LogLevel } from './contracts/IDBSQLLogger'; import IAuthentication from './connection/contracts/IAuthentication'; import { buildTelemetryUrl, normalizeHeaders } from './telemetry/telemetryUtils'; @@ -54,6 +53,9 @@ export default class FeatureFlagCache { private readonly MAX_CACHE_DURATION_S = 3600; + // The flags fetch's total time, retries included: the 10 s a single attempt could already take + private readonly FETCH_DEADLINE_MS = 10 * 1000; + constructor(private context: IClientContext, private authProvider?: IAuthentication) { this.userAgent = buildUserAgentString(this.context.getConfig().userAgentEntry); } @@ -237,23 +239,13 @@ export default class FeatureFlagCache { const connectionProvider = await this.context.getConnectionProvider(); const agent = await connectionProvider.getAgent(url); const retryPolicy = await connectionProvider.getRetryPolicy(); - const requestConfig: RequestInit = { agent, ...init }; + // connect() waits for the flags, so the attempts, the waits between them and reading the body share one deadline + const signal = AbortSignal.timeout(this.FETCH_DEADLINE_MS); + const requestConfig: RequestInit = { agent, ...init, signal: signal as unknown as RequestInit['signal'] }; const result = await retryPolicy.invokeWithRetry(() => { const request = new Request(url, requestConfig); - // connect() waits for the flags, so back off as usual instead of for as long as `Retry-After` asks, and - // don't repeat a request that already waited out its timeout (the policy rethrows a RetryError as is) - return fetch(request).then( - (response) => { - response.headers.delete('Retry-After'); - return { request, response }; - }, - (error) => { - throw error instanceof FetchError && error.type === 'request-timeout' - ? new RetryError(RetryErrorCode.TimeoutExceeded, error) - : error; - }, - ); - }); + return fetch(request).then((response) => ({ request, response })); + }, signal); return result.response; } diff --git a/lib/connection/connections/HttpRetryPolicy.ts b/lib/connection/connections/HttpRetryPolicy.ts index cfffe03a..b7b2283b 100644 --- a/lib/connection/connections/HttpRetryPolicy.ts +++ b/lib/connection/connections/HttpRetryPolicy.ts @@ -90,8 +90,9 @@ export function isRetryableStatus({ response }: HttpTransactionDetails, idempote } // A failure to connect (the kernel's `is_connect`), so the request never left the client and even a -// non-idempotent one can be retried. Node reports these as `connect
`, or as a socket closed -// before the TLS handshake finished. +// non-idempotent one can be retried. Node reports these as `connect
`, as a socket closed +// before the TLS handshake finished, or, for a host with several addresses, as an AggregateError of the +// attempts, which has a code but no message (node-fetch's reason is then empty). export function isConnectError(error: unknown): boolean { const { code, message } = (error ?? {}) as { code?: unknown; message?: unknown }; if (typeof code !== 'string' || PERMANENT_NETWORK_ERROR_CODES.rpc.has(code)) { @@ -101,7 +102,9 @@ export function isConnectError(error: unknown): boolean { code === 'ECONNREFUSED' || code === 'EAI_AGAIN' || (typeof message === 'string' && - (message.includes(`connect ${code}`) || message.includes('before secure TLS connection was established'))) + (message.includes(`connect ${code}`) || + message.includes('before secure TLS connection was established') || + message.endsWith('reason: '))) ); } diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 1943fc09..4f4eb8f5 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -108,6 +108,40 @@ class RefusingAgent extends http.Agent { } } +// Keep-alive agent whose first connection tries two closed addresses, as for a host with several addresses. +// Node fails it with an AggregateError that has a code but no message. The code is set to ETIMEDOUT, as when +// every address times out, which a test can't wait for. +class MultiAddressTimeoutAgent extends http.Agent { + public connectAttempts = 0; + + constructor(private readonly closedPort: number) { + super({ keepAlive: true }); + } + + createConnection(options: net.NetConnectOpts): net.Socket { + this.connectAttempts += 1; + if (this.connectAttempts > 1) { + return net.createConnection(options); + } + const addresses = [ + { address: '127.0.0.1', family: 4 }, + { address: '::1', family: 6 }, + ]; + const connectOptions = { + host: 'multi-address.test', + port: this.closedPort, + autoSelectFamily: true, + lookup: (hostname: string, lookupOptions: unknown, callback: (error: null, results: typeof addresses) => void) => + callback(null, addresses), + }; + const socket = net.createConnection(connectOptions as unknown as net.NetConnectOpts); + socket.prependListener('error', (error: NodeJS.ErrnoException) => { + error.code = 'ETIMEDOUT'; + }); + return socket; + } +} + async function createThriftConnection( port: number, agent: http.Agent, @@ -175,6 +209,22 @@ describe('ThriftHttpConnection retries', () => { expect(server.requests).to.equal(1); }); + it('retries ExecuteStatement when connecting to every address of the host times out', async () => { + const server = await startServer(['reply']); + const agent = new MultiAddressTimeoutAgent(await getClosedPort()); + cleanup.push( + () => server.close(), + () => agent.destroy(), + ); + const connection = await createThriftConnection(server.port, agent, {}); + + const result = await send(connection, 'ExecuteStatement', 1); + + expect(result).to.equal('ExecuteStatement result'); + expect(agent.connectAttempts).to.equal(2); + expect(server.requests).to.equal(1); + }); + it('surfaces the original error once retries for a refused ExecuteStatement are exhausted', async () => { const { server, agent, connection } = await setup([], Infinity); diff --git a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts index 610b359c..d543c8c6 100644 --- a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts +++ b/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts @@ -44,6 +44,10 @@ describe('UnsentRequestRetryPolicy', () => { 'a socket closed during the TLS handshake', systemError('ECONNRESET', 'Client network socket disconnected before secure TLS connection was established'), ], + // A host with several addresses: Node's AggregateError has the code but no message + ['a connect timeout on every address', systemError('ETIMEDOUT', '')], + ['an unreachable host on every address', systemError('EHOSTUNREACH', '')], + ['an unreachable network on every address', systemError('ENETUNREACH', '')], ]; for (const [label, error] of connectErrors) { diff --git a/tests/unit/telemetry/FeatureFlagCache.test.ts b/tests/unit/telemetry/FeatureFlagCache.test.ts index daf69964..227854ad 100644 --- a/tests/unit/telemetry/FeatureFlagCache.test.ts +++ b/tests/unit/telemetry/FeatureFlagCache.test.ts @@ -21,7 +21,6 @@ import { AddressInfo } from 'net'; import { Response } from 'node-fetch'; import FeatureFlagCache from '../../../lib/FeatureFlagCache'; import HttpRetryPolicy from '../../../lib/connection/connections/HttpRetryPolicy'; -import RetryError from '../../../lib/errors/RetryError'; import ClientContextStub from '../.stubs/ClientContextStub'; import { LogLevel } from '../../../lib/contracts/IDBSQLLogger'; @@ -278,77 +277,59 @@ describe('FeatureFlagCache', () => { fetchStub.restore(); }); - it('should not wait for Retry-After before retrying the flags request', async () => { - // connect() waits for the flags, so a Retry-After from the endpoint must not stretch the retries - clock.restore(); - let hits = 0; - const server = http.createServer((req, res) => { - hits += 1; - if (hits === 1) { + // connect() waits for the flags, so a failing or stalled endpoint must not hold it past the fetch deadline + const slowEndpoints: Array<[string, http.RequestListener]> = [ + [ + 'keeps failing and asks for long waits', + (req, res) => { res.writeHead(503, { 'Retry-After': '60' }); res.end(); - } else { - res.writeHead(200, { 'Content-Type': 'application/json' }); - res.end(JSON.stringify({ flags: [] })); - } - }); - await new Promise((resolve) => { - server.listen(0, '127.0.0.1', resolve); - }); - try { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 60_000 }); - sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); - const cache = new FeatureFlagCache(context); - const { port } = server.address() as AddressInfo; - - const started = Date.now(); - const response = await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET' }); - - expect(response.status).to.equal(200); - expect(hits).to.equal(2); - expect(Date.now() - started).to.be.lt(1000); - } finally { - server.closeAllConnections(); - server.close(); - clock = sinon.useFakeTimers(); - } - }); - - it('should not repeat a flags request that timed out', async () => { - // Each attempt waits out the request timeout, so retrying it would stretch connect() - clock.restore(); - let hits = 0; - const server = http.createServer(() => { - hits += 1; - }); - await new Promise((resolve) => { - server.listen(0, '127.0.0.1', resolve); - }); - try { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); - sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); - const cache = new FeatureFlagCache(context); - const { port } = server.address() as AddressInfo; - - let error: unknown; - try { - await (cache as any).fetchWithRetry(`http://127.0.0.1:${port}/`, { method: 'GET', timeout: 100 }); - } catch (e) { - error = e; - } + }, + ], + ['never responds', () => {}], + ]; - expect(error).to.be.instanceOf(RetryError); - expect(hits).to.equal(1); - } finally { - server.closeAllConnections(); - server.close(); - // Let node-fetch finish the aborted request (it clears its timer) before faking timers again + for (const [label, handler] of slowEndpoints) { + it(`should fall back to defaults at the fetch deadline when the endpoint ${label}`, async () => { + clock.restore(); + let hits = 0; + const server = http.createServer((req, res) => { + hits += 1; + handler(req, res); + }); await new Promise((resolve) => { - setTimeout(resolve, 10); + server.listen(0, '127.0.0.1', resolve); }); - clock = sinon.useFakeTimers(); - } - }); + try { + // The default retry settings, with a real retry policy + const context = new ClientContextStub(); + sinon.stub(context.connectionProvider, 'getRetryPolicy').callsFake(async () => new HttpRetryPolicy(context)); + const cache = new FeatureFlagCache(context); + (cache as any).FETCH_DEADLINE_MS = 300; + // The flags URL is always https, so point the real fetch at the local endpoint + const { port } = server.address() as AddressInfo; + const fetchWithRetry = (cache as any).fetchWithRetry.bind(cache); + sinon + .stub(cache as any, 'fetchWithRetry') + .callsFake((url: unknown, init: unknown) => fetchWithRetry(`http://127.0.0.1:${port}/`, init)); + cache.getOrCreateContext('test-host'); + + const started = Date.now(); + expect(await cache.getBoolean('test-host', 'flag', true)).to.equal(true); + + expect(Date.now() - started).to.be.within(250, 1000); + expect(hits).to.equal(1); + } finally { + server.closeAllConnections(); + server.close(); + // Let node-fetch finish the aborted request (it clears its timer) before faking timers again + await new Promise((resolve) => { + setTimeout(resolve, 10); + }); + clock = sinon.useFakeTimers(); + } + }); + } }); describe('Integration scenarios', () => { From fa5e6e41478055f5b847d85a23de7f168b65a2a6 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 04:42:15 +0000 Subject: [PATCH 08/15] refactor: rename UnsentRequestRetryPolicy to NonIdempotentRequestRetryPolicy The policy now retries a non-idempotent request whenever the server didn't run it: a failure to connect, a 429, or a 503 with Retry-After. That's more than an unsent request, so name it for the requests it serves. It isn't exported and hasn't been released (#598), so no alias is needed. No behavior change. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- lib/connection/connections/HttpConnection.ts | 4 ++-- ...yPolicy.ts => NonIdempotentRequestRetryPolicy.ts} | 2 +- lib/connection/connections/ThriftHttpConnection.ts | 12 ++++++------ ...st.ts => NonIdempotentRequestRetryPolicy.test.ts} | 12 +++++++----- 4 files changed, 16 insertions(+), 14 deletions(-) rename lib/connection/connections/{UnsentRequestRetryPolicy.ts => NonIdempotentRequestRetryPolicy.ts} (94%) rename tests/unit/connection/connections/{UnsentRequestRetryPolicy.test.ts => NonIdempotentRequestRetryPolicy.test.ts} (94%) diff --git a/lib/connection/connections/HttpConnection.ts b/lib/connection/connections/HttpConnection.ts index f51db386..5068e076 100644 --- a/lib/connection/connections/HttpConnection.ts +++ b/lib/connection/connections/HttpConnection.ts @@ -13,7 +13,7 @@ import HiveDriverError from '../../errors/HiveDriverError'; import ThriftHttpConnection from './ThriftHttpConnection'; import IRetryPolicy, { RetryRequestClass } from '../contracts/IRetryPolicy'; import HttpRetryPolicy from './HttpRetryPolicy'; -import UnsentRequestRetryPolicy from './UnsentRequestRetryPolicy'; +import NonIdempotentRequestRetryPolicy from './NonIdempotentRequestRetryPolicy'; // Set by the driver, auth provider or Thrift transport. node-fetch merges // same-named headers case-insensitively into one value, so a caller copy @@ -214,7 +214,7 @@ export default class HttpConnection implements IConnectionProvider { transport: thrift.TBufferedTransport, protocol: thrift.TBinaryProtocol, getRetryPolicy: () => this.getRetryPolicy(), - getUnsentRequestRetryPolicy: async () => new UnsentRequestRetryPolicy(this.context), + getNonIdempotentRequestRetryPolicy: async () => new NonIdempotentRequestRetryPolicy(this.context), }, { agent, diff --git a/lib/connection/connections/UnsentRequestRetryPolicy.ts b/lib/connection/connections/NonIdempotentRequestRetryPolicy.ts similarity index 94% rename from lib/connection/connections/UnsentRequestRetryPolicy.ts rename to lib/connection/connections/NonIdempotentRequestRetryPolicy.ts index 0ecb97c5..2171ffe9 100644 --- a/lib/connection/connections/UnsentRequestRetryPolicy.ts +++ b/lib/connection/connections/NonIdempotentRequestRetryPolicy.ts @@ -6,7 +6,7 @@ import RetryError from '../../errors/RetryError'; // For operations that are not safe to repeat: retries only requests the server didn't run, as the kernel's // non-idempotent strategy does: a failure to connect, a 429, or a 503 with `Retry-After`. Failures on a // connected socket (ECONNRESET, EPIPE, timeouts, ...) are not retried: the server may already have the request. -export default class UnsentRequestRetryPolicy extends HttpRetryPolicy { +export default class NonIdempotentRequestRetryPolicy extends HttpRetryPolicy { public async invokeWithRetry(operation: RetryableOperation): Promise { try { return await super.invokeWithRetry(operation); diff --git a/lib/connection/connections/ThriftHttpConnection.ts b/lib/connection/connections/ThriftHttpConnection.ts index ea514e64..4907a63f 100644 --- a/lib/connection/connections/ThriftHttpConnection.ts +++ b/lib/connection/connections/ThriftHttpConnection.ts @@ -61,7 +61,7 @@ interface ThriftHttpConnectionOptions { transport?: TTransportType; protocol?: TProtocolConstructor; getRetryPolicy(): Promise>; - getUnsentRequestRetryPolicy?(): Promise>; + getNonIdempotentRequestRetryPolicy?(): Promise>; } // This type describes a shape of internals of Thrift client object. @@ -97,8 +97,8 @@ const retryableThriftMethods = new Set([ ]); // Not safe to repeat once the server may have run the request, so retried only if it was never sent or -// the server rejected it unrun (see UnsentRequestRetryPolicy) -const unsentRequestRetryableThriftMethods = new Set(['ExecuteStatement']); +// the server rejected it unrun (see NonIdempotentRequestRetryPolicy) +const nonIdempotentRetryableThriftMethods = new Set(['ExecuteStatement']); export default class ThriftHttpConnection extends EventEmitter { private readonly url: string; @@ -132,10 +132,10 @@ export default class ThriftHttpConnection extends EventEmitter { } if ( thriftMethodName && - unsentRequestRetryableThriftMethods.has(thriftMethodName) && - this.options.getUnsentRequestRetryPolicy + nonIdempotentRetryableThriftMethods.has(thriftMethodName) && + this.options.getNonIdempotentRequestRetryPolicy ) { - return this.options.getUnsentRequestRetryPolicy(); + return this.options.getNonIdempotentRequestRetryPolicy(); } // Don't retry everything that is not explicitly allowed to retry return new NullRetryPolicy(); diff --git a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts b/tests/unit/connection/connections/NonIdempotentRequestRetryPolicy.test.ts similarity index 94% rename from tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts rename to tests/unit/connection/connections/NonIdempotentRequestRetryPolicy.test.ts index d543c8c6..f353a47e 100644 --- a/tests/unit/connection/connections/UnsentRequestRetryPolicy.test.ts +++ b/tests/unit/connection/connections/NonIdempotentRequestRetryPolicy.test.ts @@ -1,7 +1,7 @@ import { expect, AssertionError } from 'chai'; import sinon from 'sinon'; import { Request, Response, FetchError } from 'node-fetch'; -import UnsentRequestRetryPolicy from '../../../../lib/connection/connections/UnsentRequestRetryPolicy'; +import NonIdempotentRequestRetryPolicy from '../../../../lib/connection/connections/NonIdempotentRequestRetryPolicy'; import RetryError, { RetryErrorCode } from '../../../../lib/errors/RetryError'; import ClientContextStub from '../../.stubs/ClientContextStub'; @@ -17,7 +17,9 @@ function okResponse() { } function createPolicy(retryMaxAttempts = 5) { - return new UnsentRequestRetryPolicy(new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts })); + return new NonIdempotentRequestRetryPolicy( + new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts }), + ); } async function expectRejection(promise: Promise): Promise { @@ -32,7 +34,7 @@ async function expectRejection(promise: Promise): Promise { return expect.fail('It should throw an error'); } -describe('UnsentRequestRetryPolicy', () => { +describe('NonIdempotentRequestRetryPolicy', () => { // Failures to connect (the kernel's `is_connect`): the request never left the client const connectErrors: Array<[string, Error]> = [ ['ECONNREFUSED', systemError('ECONNREFUSED', 'connect ECONNREFUSED 10.0.0.1:443')], @@ -127,7 +129,7 @@ describe('UnsentRequestRetryPolicy', () => { } it('waits the Retry-After delay before retrying a 503, with the default config', async () => { - const policy = new UnsentRequestRetryPolicy(new ClientContextStub()); + const policy = new NonIdempotentRequestRetryPolicy(new ClientContextStub()); expect(await policy.shouldRetry(statusResponse(503, { 'Retry-After': '5' }))).to.deep.equal({ shouldRetry: true, @@ -155,7 +157,7 @@ describe('UnsentRequestRetryPolicy', () => { it('rethrows the original error once the retry timeout is exceeded', async () => { const error = systemError('ECONNREFUSED'); - const policy = new UnsentRequestRetryPolicy( + const policy = new NonIdempotentRequestRetryPolicy( new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 100, retriesTimeout: 0 }), ); const operation = sinon.stub().rejects(error); From 7ae6980d59e759f3afb9901d046a4f81a7b50bd5 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 04:45:17 +0000 Subject: [PATCH 09/15] fix: keep the original error once retries this PR adds run out OpenSession, ExecuteStatement on a 429 or a 503 with Retry-After, and idempotent RPCs on statuses main didn't retry (408, 501, other 4xx) failed with THTTPException, which carries statusCode, the response and the server's detail, or, for OpenSession, with the network error. Once their new retries ran out they failed with a HiveDriverError that has none of these. Throw the last attempt's error instead, as the ExecuteStatement connect-failure retry already does. Retries that idempotent RPCs made before (429, 5xx except 501, network errors) keep failing with the HiveDriverError. BaseCommand no longer needs to name the network error code, which only mattered for OpenSession, so it is back as on main. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- .../connections/ThriftHttpConnection.ts | 35 ++++++-- lib/hive/Commands/BaseCommand.ts | 11 +-- .../ThriftHttpConnectionRetry.test.ts | 81 ++++++++++++++++++- tests/unit/hive/commands/BaseCommand.test.ts | 27 +------ 4 files changed, 113 insertions(+), 41 deletions(-) diff --git a/lib/connection/connections/ThriftHttpConnection.ts b/lib/connection/connections/ThriftHttpConnection.ts index 4907a63f..f7a2dfdf 100644 --- a/lib/connection/connections/ThriftHttpConnection.ts +++ b/lib/connection/connections/ThriftHttpConnection.ts @@ -100,6 +100,26 @@ const retryableThriftMethods = new Set([ // the server rejected it unrun (see NonIdempotentRequestRetryPolicy) const nonIdempotentRetryableThriftMethods = new Set(['ExecuteStatement']); +// Once retries run out, fail with the last attempt's error, as without retries: THTTPException (status and server +// detail) or the network error. Idempotent RPCs other than OpenSession have always retried a 429, a 5xx other than +// 501 and a network error other than a request timeout; those keep the RetryError (a HiveDriverError in BaseCommand). +function getRetryExhaustedError(thriftMethod: string | undefined, error: RetryError): unknown { + const { payload } = error; + const response = (payload as { response?: unknown } | undefined)?.response; + const status = response instanceof Response ? response.status : undefined; + const alwaysRetried = + thriftMethod !== undefined && + thriftMethod !== 'OpenSession' && + retryableThriftMethods.has(thriftMethod) && + (status === undefined + ? !(payload instanceof FetchError && payload.type === 'request-timeout') + : status === 429 || (status >= 500 && status !== 501)); + if (alwaysRetried) { + return error; + } + return response instanceof Response ? new THTTPException(response) : payload ?? error; +} + export default class ThriftHttpConnection extends EventEmitter { private readonly url: string; @@ -168,9 +188,13 @@ export default class ThriftHttpConnection extends EventEmitter { // transient network error — the body Buffer is captured via closure so the // post-retry caller still gets it. let responseBuffer: Buffer | undefined; + let thriftMethodName: string | undefined; this.getThriftMethodName(data) - .then((thriftMethod) => this.getRetryPolicy(thriftMethod)) + .then((thriftMethod) => { + thriftMethodName = thriftMethod; + return this.getRetryPolicy(thriftMethod); + }) .then((retryPolicy) => { const makeRequest = async () => { responseBuffer = undefined; @@ -202,13 +226,8 @@ export default class ThriftHttpConnection extends EventEmitter { )(buffer); }) .catch((error) => { - // A request that timed out on every attempt still fails as a timeout - if ( - error instanceof RetryError && - error.payload instanceof FetchError && - error.payload.type === 'request-timeout' - ) { - error = error.payload; + if (error instanceof RetryError) { + error = getRetryExhaustedError(thriftMethodName, error); } if (error instanceof FetchError) { if (error.type === 'request-timeout') { diff --git a/lib/hive/Commands/BaseCommand.ts b/lib/hive/Commands/BaseCommand.ts index 0db2b029..22ac66fb 100644 --- a/lib/hive/Commands/BaseCommand.ts +++ b/lib/hive/Commands/BaseCommand.ts @@ -27,15 +27,16 @@ export default abstract class BaseCommand { ) { statusCode = error.payload.response.status; } - // After network errors, name the last one's code (e.g. ECONNREFUSED) instead of a status - const code = (error.payload as { code?: unknown } | null | undefined)?.code; - const cause = statusCode ?? (typeof code === 'string' && code ? code : 'Error'); switch (error.errorCode) { case RetryErrorCode.AttemptsExceeded: - throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Max retry count exceeded.`); + throw new HiveDriverError( + `Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Max retry count exceeded.`, + ); case RetryErrorCode.TimeoutExceeded: - throw new HiveDriverError(`Hive driver: ${cause} when connecting to resource. Retry timeout exceeded.`); + throw new HiveDriverError( + `Hive driver: ${statusCode ?? 'Error'} when connecting to resource. Retry timeout exceeded.`, + ); // no default } } diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 4f4eb8f5..5e81e4b0 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -5,6 +5,7 @@ import { FetchError } from 'node-fetch'; import HttpConnection from '../../../../lib/connection/connections/HttpConnection'; import ThriftHttpConnection, { THTTPException } from '../../../../lib/connection/connections/ThriftHttpConnection'; import { ClientConfig } from '../../../../lib/contracts/IClientContext'; +import RetryError, { RetryErrorCode } from '../../../../lib/errors/RetryError'; import ClientContextStub from '../../.stubs/ClientContextStub'; // Header-only Thrift binary messages: enough for method-name detection and for the fake `recv_*` below. @@ -27,9 +28,12 @@ type ServerAction = | 'stall' | 'status-408' | 'status-429' + | 'status-501' | 'status-503' | 'status-503-retry-after'; +const ERROR_DETAIL = 'server error detail'; + interface TestServer { port: number; requests: number; @@ -50,10 +54,10 @@ async function startServer(actions: ServerAction[]): Promise { } else if (action === 'stall') { // Never respond; the client's request timeout ends it } else if (action === 'status-503-retry-after') { - res.writeHead(503, { 'Retry-After': '1' }); + res.writeHead(503, { 'Retry-After': '1', 'x-thriftserver-error-message': ERROR_DETAIL }); res.end(); } else if (action.startsWith('status-')) { - res.writeHead(Number(action.slice('status-'.length))); + res.writeHead(Number(action.slice('status-'.length)), { 'x-thriftserver-error-message': ERROR_DETAIL }); res.end(); } else { const body = Buffer.concat(chunks); @@ -346,4 +350,77 @@ describe('ThriftHttpConnection retries', () => { expect(result).to.equal('GetOperationStatus result'); expect(server.requests).to.equal(2); }); + + describe('once retries run out', () => { + function expectHttpError(error: any, status: number) { + expect(error).to.be.instanceOf(THTTPException); + expect(error.statusCode).to.equal(status); + expect(error.message).to.equal(`Received a response with a bad HTTP status code: ${status}: ${ERROR_DETAIL}`); + } + + // Requests that weren't retried before fail as they did then + for (const [method, action, status] of [ + ['OpenSession', 'status-503', 503], + ['OpenSession', 'status-429', 429], + ['OpenSession', 'status-501', 501], + ['GetOperationStatus', 'status-408', 408], + ['GetOperationStatus', 'status-501', 501], + ['ExecuteStatement', 'status-429', 429], + ['ExecuteStatement', 'status-503-retry-after', 503], + ] as const) { + it(`fails ${method} with the last HTTP error (${action})`, async () => { + const { server, connection } = await setup([action, action, action], 0); + + const error = await expectRejection(send(connection, method, 1)); + + expectHttpError(error, status); + expect(server.requests).to.equal(3); + }); + } + + it('fails OpenSession with the connection error', async () => { + const { server, agent, connection } = await setup([], Infinity); + + const error = await expectRejection(send(connection, 'OpenSession', 1)); + + expect(error).to.be.instanceOf(FetchError); + expect(error.code).to.equal('ECONNREFUSED'); + expect(agent.connectAttempts).to.equal(3); + expect(server.requests).to.equal(0); + }); + + it('fails OpenSession with the network error when the connection closes after the request', async () => { + const action = 'close-after-request'; + const { server, connection } = await setup([action, action, action], 0); + + const error = await expectRejection(send(connection, 'OpenSession', 1)); + + expect(error).to.be.instanceOf(FetchError); + expect(error.code).to.equal('ECONNRESET'); + expect(server.requests).to.equal(3); + }); + + // Retried before, so they keep failing with the RetryError + it('keeps the RetryError for an idempotent RPC on a 503', async () => { + const { server, connection } = await setup(['status-503', 'status-503', 'status-503'], 0); + + const error = await expectRejection(send(connection, 'GetOperationStatus', 1)); + + expect(error).to.be.instanceOf(RetryError); + expect(error.errorCode).to.equal(RetryErrorCode.AttemptsExceeded); + expect(error.payload.response.status).to.equal(503); + expect(server.requests).to.equal(3); + }); + + it('keeps the RetryError for an idempotent RPC whose connection is refused', async () => { + const { agent, connection } = await setup([], Infinity); + + const error = await expectRejection(send(connection, 'GetOperationStatus', 1)); + + expect(error).to.be.instanceOf(RetryError); + expect(error.errorCode).to.equal(RetryErrorCode.AttemptsExceeded); + expect(error.payload.code).to.equal('ECONNREFUSED'); + expect(agent.connectAttempts).to.equal(3); + }); + }); }); diff --git a/tests/unit/hive/commands/BaseCommand.test.ts b/tests/unit/hive/commands/BaseCommand.test.ts index df372a05..b1514775 100644 --- a/tests/unit/hive/commands/BaseCommand.test.ts +++ b/tests/unit/hive/commands/BaseCommand.test.ts @@ -1,6 +1,6 @@ import { expect, AssertionError } from 'chai'; import sinon from 'sinon'; -import { FetchError, Request, Response } from 'node-fetch'; +import { Request, Response } from 'node-fetch'; import { Thrift } from 'thrift'; import HiveDriverError from '../../../../lib/errors/HiveDriverError'; import BaseCommand from '../../../../lib/hive/Commands/BaseCommand'; @@ -211,31 +211,6 @@ describe('BaseCommand', () => { }); }); - it('should name the network error code once retries run out', async () => { - const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); - const command = new CustomCommand( - new ThriftClientStub(context, async () => { - throw new FetchError('request to http://localhost/ failed, reason: connect ECONNREFUSED', 'system', { - code: 'ECONNREFUSED', - } as unknown as NodeJS.ErrnoException); - }), - context, - ); - - try { - await command.execute({}); - expect.fail('It should throw an error'); - } catch (error) { - if (error instanceof AssertionError || !(error instanceof Error)) { - throw error; - } - expect(error).to.be.instanceof(HiveDriverError); - expect(error.message).to.equal( - 'Hive driver: ECONNREFUSED when connecting to resource. Max retry count exceeded.', - ); - } - }); - it(`should re-throw unrecognized HTTP errors`, async () => { const context = new ClientContextStub(); From 2a3aac75148be11c031d6ecae8d135118de210b1 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 04:47:51 +0000 Subject: [PATCH 10/15] fix: don't retry certificate failures Node reports as UNSPECIFIED Node names 28 OpenSSL certificate verification errors and reports every other one with the code UNSPECIFIED (ncrypto's X509Pointer::ErrorCode), for example a CA without keyCertSign, an unhandled critical extension or a SHA-1 signature. The certificate list missed it, so OpenSession retried such a failure for the whole default budget (about 30 s), and CloudFetch downloads retried it too. Treat it as a certificate failure, which the kernel never retries either. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- lib/connection/connections/HttpRetryPolicy.ts | 5 +- .../connections/HttpRetryPolicy.test.ts | 8 +- .../ThriftHttpConnectionRetry.test.ts | 78 +++++++++++++++++++ .../result/CloudFetchResultHandler.test.ts | 30 ++++--- 4 files changed, 108 insertions(+), 13 deletions(-) diff --git a/lib/connection/connections/HttpRetryPolicy.ts b/lib/connection/connections/HttpRetryPolicy.ts index b7b2283b..d1b35925 100644 --- a/lib/connection/connections/HttpRetryPolicy.ts +++ b/lib/connection/connections/HttpRetryPolicy.ts @@ -39,7 +39,9 @@ const RETRYABLE_NETWORK_ERROR_CODES = new Set([ 'EAI_AGAIN', ]); -// The client's certificate verification (OpenSSL's X509 codes as Node names them, and its hostname check) +// The client's certificate verification (OpenSSL's X509 codes as Node names them, and its hostname check). +// Node reports every X509 code it doesn't name as `UNSPECIFIED` (ncrypto's X509Pointer::ErrorCode), e.g. a CA +// without keyCertSign or a SHA-1 signature; the kernel doesn't retry any certificate verification failure either. const CERTIFICATE_ERROR_CODES = [ 'CERT_CHAIN_TOO_LONG', 'CERT_HAS_EXPIRED', @@ -63,6 +65,7 @@ const CERTIFICATE_ERROR_CODES = [ 'UNABLE_TO_GET_ISSUER_CERT', 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY', 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', + 'UNSPECIFIED', ]; // Which statuses and network errors are retried matches the kernel (databricks-sql-kernel src/client/retry.rs diff --git a/tests/unit/connection/connections/HttpRetryPolicy.test.ts b/tests/unit/connection/connections/HttpRetryPolicy.test.ts index 1bbf678d..258f48c8 100644 --- a/tests/unit/connection/connections/HttpRetryPolicy.test.ts +++ b/tests/unit/connection/connections/HttpRetryPolicy.test.ts @@ -476,6 +476,8 @@ describe('HttpRetryPolicy', () => { 'SELF_SIGNED_CERT_IN_CHAIN', 'CERT_HAS_EXPIRED', 'ERR_TLS_CERT_ALTNAME_INVALID', + // Node's code for the X509 errors it doesn't name + 'UNSPECIFIED', 'ENOTFOUND', ]) { it(`throws ${code} at once on an RPC, without retrying`, () => expectThrownAtOnce(systemError(code))); @@ -498,8 +500,10 @@ describe('HttpRetryPolicy', () => { // As the kernel's CloudFetch chunk retry: everything but certificate failures it('retries ENOTFOUND on a CloudFetch download', () => expectRetried(systemError('ENOTFOUND'), 'cloudFetch')); - it('throws a certificate failure at once on a CloudFetch download', () => - expectThrownAtOnce(systemError('CERT_HAS_EXPIRED'), 'cloudFetch')); + for (const code of ['CERT_HAS_EXPIRED', 'UNSPECIFIED']) { + it(`throws a certificate failure (${code}) at once on a CloudFetch download`, () => + expectThrownAtOnce(systemError(code), 'cloudFetch')); + } }); it('should cancel a pending backoff sleep when the signal is aborted', async () => { diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index 5e81e4b0..f062fcb5 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -1,5 +1,10 @@ +import { execFileSync } from 'child_process'; +import fs from 'fs'; import http from 'http'; +import https from 'https'; import net, { AddressInfo } from 'net'; +import os from 'os'; +import path from 'path'; import { expect } from 'chai'; import { FetchError } from 'node-fetch'; import HttpConnection from '../../../../lib/connection/connections/HttpConnection'; @@ -424,3 +429,76 @@ describe('ThriftHttpConnection retries', () => { }); }); }); + +describe('ThriftHttpConnection with a certificate that fails verification', () => { + // A trusted self-signed `localhost` certificate with an unknown critical extension, generated per run so no + // private key is committed. OpenSSL rejects it as "unhandled critical extension", which Node reports as + // `UNSPECIFIED` because it has no name for that X509 error. + let cert = ''; + let key = ''; + + before(function () { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'dbsql-cert-')); + try { + execFileSync( + 'openssl', + [ + 'req', + '-x509', + '-newkey', + 'ec', + '-pkeyopt', + 'ec_paramgen_curve:prime256v1', + '-nodes', + '-days', + '1', + '-subj', + '/CN=localhost', + '-addext', + 'subjectAltName=DNS:localhost,IP:127.0.0.1', + '-addext', + '1.3.6.1.4.1.55555.1=critical,ASN1:UTF8String:x', + '-keyout', + path.join(dir, 'key.pem'), + '-out', + path.join(dir, 'cert.pem'), + ], + { stdio: 'ignore' }, + ); + cert = fs.readFileSync(path.join(dir, 'cert.pem'), 'utf8'); + key = fs.readFileSync(path.join(dir, 'key.pem'), 'utf8'); + } catch { + this.skip(); // openssl is not available + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + it('fails OpenSession at once with the certificate error Node reports as UNSPECIFIED', async () => { + let connections = 0; + const server = https.createServer({ cert, key }, (req, res) => res.end()); + server.on('connection', () => { + connections += 1; + }); + await new Promise((resolve) => { + server.listen(0, '127.0.0.1', resolve); + }); + const context = new ClientContextStub({ retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }); + const { port } = server.address() as AddressInfo; + const httpConnection = new HttpConnection({ host: 'localhost', port, path: '/', https: true, ca: cert }, context); + try { + const connection = await httpConnection.getThriftConnection(); + + const error = await expectRejection(send(connection, 'OpenSession', 1)); + + expect(error).to.be.instanceOf(FetchError); + expect(error.code).to.equal('UNSPECIFIED'); + expect(error.message).to.contain('unhandled critical extension'); + expect(connections).to.equal(1); + } finally { + (await httpConnection.getAgent())?.destroy(); + server.closeAllConnections(); + server.close(); + } + }); +}); diff --git a/tests/unit/result/CloudFetchResultHandler.test.ts b/tests/unit/result/CloudFetchResultHandler.test.ts index 2ded0970..a1dd524e 100644 --- a/tests/unit/result/CloudFetchResultHandler.test.ts +++ b/tests/unit/result/CloudFetchResultHandler.test.ts @@ -804,18 +804,28 @@ describe('CloudFetchResultHandler', () => { } as unknown as http.AgentOptions); } - it('should not leak the presigned URL from a network error that is not retried', async () => { - // A certificate failure is thrown without retrying - const agent = failingAgent('CERT_HAS_EXPIRED'); - const handler = createHandler('http://storage.example.invalid/chunk?sig=SECRET_SIGNATURE', {}, agent); + // A certificate failure is thrown without retrying (`UNSPECIFIED`: an X509 error Node doesn't name) + for (const code of ['CERT_HAS_EXPIRED', 'UNSPECIFIED']) { + it(`should not retry or leak the presigned URL from a certificate failure (${code})`, async () => { + let lookups = 0; + const agent = failingAgent(code, () => { + lookups += 1; + }); + const handler = createHandler( + 'http://storage.example.invalid/chunk?sig=SECRET_SIGNATURE', + { retryDelayMin: 1, retryDelayMax: 2, retryMaxAttempts: 3 }, + agent, + ); - const { error } = await fetchWithin(handler, 1500); - agent.destroy(); + const { error } = await fetchWithin(handler, 1500); + agent.destroy(); - expect(error.code).to.be.equal('CERT_HAS_EXPIRED'); - expect(error.message).to.be.equal('CloudFetch result download failed: network error CERT_HAS_EXPIRED'); - expect(error.stack).to.not.contain('SECRET_SIGNATURE'); - }); + expect(lookups).to.be.equal(1); + expect(error.code).to.be.equal(code); + expect(error.message).to.be.equal(`CloudFetch result download failed: network error ${code}`); + expect(error.stack).to.not.contain('SECRET_SIGNATURE'); + }); + } it('should retry a download whose host does not resolve, as the kernel', async () => { let lookups = 0; From dd04e5ffc9befe49f447a17edea00aff91675dde Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 04:49:15 +0000 Subject: [PATCH 11/15] docs: shorten the retry CHANGELOG entry and merge the ExecuteStatement one Rewrite the #605 entry as short sentences on what is now retried, what no longer is, Retry-After, the feature-flag cap and the error once retries run out, which is now the original error for calls that weren't retried before. Merge #458's ExecuteStatement connect-failure entry, which #605 broadens to any failure to connect and narrows to drop ENOTFOUND. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b993b2aa..2e93928e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the HTTP statuses and network errors the kernel backend retries, with the exceptions listed in #605; backoff and the `retryMaxAttempts` / `retriesTimeout` budget stay as before. `OpenSession` is retried like other idempotent requests. Idempotent requests and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously 429 and 5xx except 501), transient network errors such as a refused or reset connection, and, for Thrift requests, a response that doesn't arrive within `socketTimeout`, when `retriesTimeout` leaves time for another attempt (not with the defaults, 15 minutes each); a request that times out still fails with `Request timed out`. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, on HTTP 503 with a `Retry-After` header, and on any failure to connect (previously only `ECONNREFUSED` and `EAI_AGAIN`). Server certificate verification failures are no longer retried, and neither is a host name that doesn't resolve (`ENOTFOUND`) on Thrift requests and the feature-flag fetch; CloudFetch downloads still retry it, as the kernel does. The original error is thrown at once (for CloudFetch as `CloudFetch result download failed: network error `, without the presigned URL). A retry now waits as long as the server's `Retry-After` header asks (in seconds, within `retryDelayMin` and `retryDelayMax`; previously the value had no effect below 1000 seconds), and a `Retry-After` wait that would end after `retriesTimeout` is not started. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total, retries included, and falls back to defaults. **Behavior change:** when the budget runs out, these calls fail with `HiveDriverError` (`Hive driver: when connecting to resource. Max retry count exceeded.` or `... Retry timeout exceeded.`) instead of the `THTTPException` or network error, so `statusCode` and the server's error detail are no longer on the error. +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried. Idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously only 429 and 5xx other than 501), and idempotent calls also retry a request that times out (`socketTimeout`) if `retriesTimeout` leaves time for it, which the defaults don't. `ExecuteStatement` is retried only when the server didn't run it, so a statement can't run twice: on HTTP 429, on HTTP 503 with `Retry-After`, and on any failure to connect. Certificate verification failures are no longer retried (a CloudFetch download then fails with `CloudFetch result download failed: network error `, without the presigned URL), and neither is `ENOTFOUND`, except on CloudFetch downloads. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and isn't started if it would end after `retriesTimeout`. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total and falls back to defaults. Once retries run out, a call that wasn't retried before fails as it did then, with `THTTPException` (`statusCode` and the server's detail) or the network error; idempotent calls still fail with `HiveDriverError` on the 429, 5xx (except 501) and network errors they already retried, and CloudFetch downloads with `RetryError`. (#458, #605) - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. @@ -11,7 +11,6 @@ - Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574) - Thrift HTTP errors (`THTTPException`) now append server detail from the error headers (at most 500 characters). Kernel-backend errors now carry a non-enumerable `cause` summarizing the kernel error code and message (also at most 500 characters). Reyden fallback preserves KP001 after any existing kernel cause. (#467) - Add `Symbol.asyncDispose` to clients, sessions, and operations, delegating to `close()`. Use `asyncDisposable()` with interface-typed results in TypeScript; existing interfaces and factory return types remain unchanged. (#331) -- Thrift `ExecuteStatement` requests are now retried when the connection fails before the request is sent (`ECONNREFUSED`, `ENOTFOUND`, `EAI_AGAIN`), within the existing `retryMaxAttempts` / `retriesTimeout` budget; once retries run out, the original connection error is thrown. Failures after the request may have reached the server are still not retried, so a statement can't run twice. (#458) - Add the `https` connection option: set `https: false` to connect to `host` over plain HTTP (port 80 by default), for example through a trusted local reverse proxy. HTTPS remains the default, and plain HTTP should only be used on a trusted local or private link. (#503) - Add the `rowMode` option to `executeStatement()`: `rowMode: 'array'` returns each row as an array of values in column order, so columns with the same name (e.g. `SELECT carat AS a, color AS a`) are no longer lost; the default `'object'` rows are unchanged, and the driver now logs a warning when a result has duplicate column names in object mode. (#118) - CloudFetch retry-exhaustion errors now identify the download and the last failure: the `RetryError` message reads `CloudFetch result download failed: (last error: )`. `` is `HTTP : ` (or `HTTP ` when the status text is empty) for HTTP failures, and `network error `, `network error ()` (e.g. `aborted`, when there is no `code`) or a bare `network error` for network failures (the raw network-error message, which can contain the presigned URL, is not copied). If no cause can be determined, the ` (last error: …)` suffix is omitted. The `RetryError` instance, `errorCode` and `payload` (including the node-fetch `Request` / `Response`) are unchanged. (#471) From 9ac4ae021fe058e940977d095dcd9b6aaa4552d6 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 04:55:09 +0000 Subject: [PATCH 12/15] docs: say that certificate and ENOTFOUND failures now throw FetchError Idempotent RPCs used to retry these and then fail with HiveDriverError; now they fail at once with the node-fetch FetchError. Say so, and limit the HiveDriverError sentence to the network errors still retried. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e93928e..3b636a2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried. Idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously only 429 and 5xx other than 501), and idempotent calls also retry a request that times out (`socketTimeout`) if `retriesTimeout` leaves time for it, which the defaults don't. `ExecuteStatement` is retried only when the server didn't run it, so a statement can't run twice: on HTTP 429, on HTTP 503 with `Retry-After`, and on any failure to connect. Certificate verification failures are no longer retried (a CloudFetch download then fails with `CloudFetch result download failed: network error `, without the presigned URL), and neither is `ENOTFOUND`, except on CloudFetch downloads. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and isn't started if it would end after `retriesTimeout`. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total and falls back to defaults. Once retries run out, a call that wasn't retried before fails as it did then, with `THTTPException` (`statusCode` and the server's detail) or the network error; idempotent calls still fail with `HiveDriverError` on the 429, 5xx (except 501) and network errors they already retried, and CloudFetch downloads with `RetryError`. (#458, #605) +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried. Idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously only 429 and 5xx other than 501), and idempotent calls also retry a request that times out (`socketTimeout`) if `retriesTimeout` leaves time for it, which the defaults don't. `ExecuteStatement` is retried only when the server didn't run it, so a statement can't run twice: on HTTP 429, on HTTP 503 with `Retry-After`, and on any failure to connect. Certificate verification failures are no longer retried, and neither is `ENOTFOUND`, except on CloudFetch downloads. They now fail at once with the node-fetch `FetchError` (where idempotent calls used to fail with `HiveDriverError` once retries ran out), or for a CloudFetch download with `CloudFetch result download failed: network error `, without the presigned URL. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and isn't started if it would end after `retriesTimeout`. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total and falls back to defaults. Once retries run out, a call that wasn't retried before fails as it did then, with `THTTPException` (`statusCode` and the server's detail) or the network error; idempotent calls still fail with `HiveDriverError` on the 429, 5xx (except 501) and network errors they retried before and still retry, and CloudFetch downloads with `RetryError`. (#458, #605) - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. From 90e2eb2a8a4a15dea9303ed4c36283ae9d15f37f Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 05:24:05 +0000 Subject: [PATCH 13/15] fix: keep main's RetryError for an idempotent RPC on a status below 100 main retried a status below 100 on idempotent RPCs, as it did a 429 and a 5xx other than 501, and ended in HiveDriverError. Include it in the statuses whose retry exhaustion keeps the RetryError, so that row doesn't change either. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- .../connections/ThriftHttpConnection.ts | 7 +++--- .../ThriftHttpConnectionRetry.test.ts | 25 +++++++++++++------ 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/lib/connection/connections/ThriftHttpConnection.ts b/lib/connection/connections/ThriftHttpConnection.ts index f7a2dfdf..52bbb4f1 100644 --- a/lib/connection/connections/ThriftHttpConnection.ts +++ b/lib/connection/connections/ThriftHttpConnection.ts @@ -101,8 +101,9 @@ const retryableThriftMethods = new Set([ const nonIdempotentRetryableThriftMethods = new Set(['ExecuteStatement']); // Once retries run out, fail with the last attempt's error, as without retries: THTTPException (status and server -// detail) or the network error. Idempotent RPCs other than OpenSession have always retried a 429, a 5xx other than -// 501 and a network error other than a request timeout; those keep the RetryError (a HiveDriverError in BaseCommand). +// detail) or the network error. Idempotent RPCs other than OpenSession have always retried a status below 100, a 429, +// a 5xx other than 501 and a network error other than a request timeout; those keep the RetryError (a HiveDriverError +// in BaseCommand). function getRetryExhaustedError(thriftMethod: string | undefined, error: RetryError): unknown { const { payload } = error; const response = (payload as { response?: unknown } | undefined)?.response; @@ -113,7 +114,7 @@ function getRetryExhaustedError(thriftMethod: string | undefined, error: RetryEr retryableThriftMethods.has(thriftMethod) && (status === undefined ? !(payload instanceof FetchError && payload.type === 'request-timeout') - : status === 429 || (status >= 500 && status !== 501)); + : status < 100 || status === 429 || (status >= 500 && status !== 501)); if (alwaysRetried) { return error; } diff --git a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts index f062fcb5..4fe0b696 100644 --- a/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts +++ b/tests/unit/connection/connections/ThriftHttpConnectionRetry.test.ts @@ -31,6 +31,7 @@ type ServerAction = | 'reply' | 'close-after-request' | 'stall' + | 'status-099' | 'status-408' | 'status-429' | 'status-501' @@ -58,6 +59,9 @@ async function startServer(actions: ServerAction[]): Promise { req.socket.destroy(); } else if (action === 'stall') { // Never respond; the client's request timeout ends it + } else if (action === 'status-099') { + // ServerResponse rejects a status below 100, so write the status line itself + req.socket.end('HTTP/1.1 099 X\r\nContent-Length: 0\r\n\r\n'); } else if (action === 'status-503-retry-after') { res.writeHead(503, { 'Retry-After': '1', 'x-thriftserver-error-message': ERROR_DETAIL }); res.end(); @@ -406,16 +410,21 @@ describe('ThriftHttpConnection retries', () => { }); // Retried before, so they keep failing with the RetryError - it('keeps the RetryError for an idempotent RPC on a 503', async () => { - const { server, connection } = await setup(['status-503', 'status-503', 'status-503'], 0); + for (const [action, status] of [ + ['status-503', 503], + ['status-099', 99], + ] as const) { + it(`keeps the RetryError for an idempotent RPC on a ${status}`, async () => { + const { server, connection } = await setup([action, action, action], 0); - const error = await expectRejection(send(connection, 'GetOperationStatus', 1)); + const error = await expectRejection(send(connection, 'GetOperationStatus', 1)); - expect(error).to.be.instanceOf(RetryError); - expect(error.errorCode).to.equal(RetryErrorCode.AttemptsExceeded); - expect(error.payload.response.status).to.equal(503); - expect(server.requests).to.equal(3); - }); + expect(error).to.be.instanceOf(RetryError); + expect(error.errorCode).to.equal(RetryErrorCode.AttemptsExceeded); + expect(error.payload.response.status).to.equal(status); + expect(server.requests).to.equal(3); + }); + } it('keeps the RetryError for an idempotent RPC whose connection is refused', async () => { const { agent, connection } = await setup([], Infinity); From ec099fd2091fd90db2c4b63b46150bdd6cc24605 Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 05:24:40 +0000 Subject: [PATCH 14/15] docs: simplify the retry CHANGELOG entry Six short sentences: what is now retried, ExecuteStatement, what is no longer retried, Retry-After and the feature-flag cap, and the error once retries run out. Drop the previously-was asides, the CloudFetch message format (#471 documents it) and the retriesTimeout guard on Retry-After. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b636a2c..fa95319b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried. Idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416 (previously only 429 and 5xx other than 501), and idempotent calls also retry a request that times out (`socketTimeout`) if `retriesTimeout` leaves time for it, which the defaults don't. `ExecuteStatement` is retried only when the server didn't run it, so a statement can't run twice: on HTTP 429, on HTTP 503 with `Retry-After`, and on any failure to connect. Certificate verification failures are no longer retried, and neither is `ENOTFOUND`, except on CloudFetch downloads. They now fail at once with the node-fetch `FetchError` (where idempotent calls used to fail with `HiveDriverError` once retries ran out), or for a CloudFetch download with `CloudFetch result download failed: network error `, without the presigned URL. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and isn't started if it would end after `retriesTimeout`. The feature-flag fetch that `connect()` waits for gives up after 10 seconds in total and falls back to defaults. Once retries run out, a call that wasn't retried before fails as it did then, with `THTTPException` (`statusCode` and the server's detail) or the network error; idempotent calls still fail with `HiveDriverError` on the 429, 5xx (except 501) and network errors they retried before and still retry, and CloudFetch downloads with `RetryError`. (#458, #605) +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried, idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416, and idempotent calls also retry a request timeout when `retriesTimeout` leaves time. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, HTTP 503 with `Retry-After`, or a failure to connect. Certificate verification failures, and `ENOTFOUND` except on CloudFetch downloads, are no longer retried and fail at once with node-fetch's `FetchError`. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and the feature-flag fetch that `connect()` waits for gives up after 10 seconds and uses the defaults. Once retries run out, `OpenSession`, `ExecuteStatement`, request timeouts and newly retried statuses fail with the same error as before (such as `THTTPException` or the network error); other calls fail with `HiveDriverError`, or `RetryError` for CloudFetch downloads. (#458, #605) - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results. From 271d2d504186c02de87acc49f6feada48e14c48f Mon Sep 17 00:00:00 2001 From: Vu Anh Phung Date: Tue, 17 Nov 2026 06:37:25 +0000 Subject: [PATCH 15/15] docs: say CloudFetch downloads fail with RetryError once retries run out The last sentence said newly retried statuses keep their old error, which isn't true for CloudFetch: a 501 there used to be a plain Error and now ends as RetryError. Name CloudFetch first so the claim covers only Thrift calls. Co-authored-by: Isaac Signed-off-by: Vu Anh Phung --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa95319b..c76492f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried, idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416, and idempotent calls also retry a request timeout when `retriesTimeout` leaves time. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, HTTP 503 with `Retry-After`, or a failure to connect. Certificate verification failures, and `ENOTFOUND` except on CloudFetch downloads, are no longer retried and fail at once with node-fetch's `FetchError`. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and the feature-flag fetch that `connect()` waits for gives up after 10 seconds and uses the defaults. Once retries run out, `OpenSession`, `ExecuteStatement`, request timeouts and newly retried statuses fail with the same error as before (such as `THTTPException` or the network error); other calls fail with `HiveDriverError`, or `RetryError` for CloudFetch downloads. (#458, #605) +- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried, idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416, and idempotent calls also retry a request timeout when `retriesTimeout` leaves time. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, HTTP 503 with `Retry-After`, or a failure to connect. Certificate verification failures, and `ENOTFOUND` except on CloudFetch downloads, are no longer retried and fail at once with node-fetch's `FetchError`. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and the feature-flag fetch that `connect()` waits for gives up after 10 seconds and uses the defaults. Once retries run out, CloudFetch downloads fail with `RetryError`; `OpenSession`, `ExecuteStatement`, request timeouts and newly retried statuses fail with the same error as before (such as `THTTPException` or the network error), and other calls with `HiveDriverError`. (#458, #605) - The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611) - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) - Reduce result chunking and `fetchAll()` overhead for large query results.