Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a9a6052
fix: retry Thrift OpenSession and a 503-with-Retry-After ExecuteState…
vuanhphung Nov 16, 2026
c6b3606
fix: wait as long as Retry-After asks, except for the feature-flag fetch
vuanhphung Nov 16, 2026
ab623a9
fix: retry the same statuses and network errors as the kernel backend
vuanhphung Nov 16, 2026
b3b240d
fix: keep "Request timed out" when every retried attempt times out
vuanhphung Nov 16, 2026
111ee19
test: pin "Request timed out" when the first timeout uses up the budget
vuanhphung Nov 16, 2026
be73bde
fix: keep the feature-flag fetch's retries, without stretching connect()
vuanhphung Nov 16, 2026
26abeb4
fix: cap the feature-flag fetch's total time, and retry a multi-addre…
vuanhphung Nov 16, 2026
fa5e6e4
refactor: rename UnsentRequestRetryPolicy to NonIdempotentRequestRetr…
vuanhphung Nov 17, 2026
7ae6980
fix: keep the original error once retries this PR adds run out
vuanhphung Nov 17, 2026
2a3aac7
fix: don't retry certificate failures Node reports as UNSPECIFIED
vuanhphung Nov 17, 2026
dd04e5f
docs: shorten the retry CHANGELOG entry and merge the ExecuteStatemen…
vuanhphung Nov 17, 2026
9ac4ae0
docs: say that certificate and ENOTFOUND failures now throw FetchError
vuanhphung Nov 17, 2026
90e2eb2
fix: keep main's RetryError for an idempotent RPC on a status below 100
vuanhphung Nov 17, 2026
ec099fd
docs: simplify the retry CHANGELOG entry
vuanhphung Nov 17, 2026
271d2d5
docs: say CloudFetch downloads fail with RetryError once retries run out
vuanhphung Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased

- Thrift requests now retry the same HTTP statuses and network errors as the kernel backend, within the existing backoff and `retryMaxAttempts` / `retriesTimeout` budget. `OpenSession` is now retried, idempotent calls and CloudFetch downloads retry every status except 400, 401, 403–405 and 409–416, and idempotent calls also retry a request timeout when `retriesTimeout` leaves time. `ExecuteStatement` is retried only when the server didn't run it: on HTTP 429, HTTP 503 with `Retry-After`, or a failure to connect. Certificate verification failures, and `ENOTFOUND` except on CloudFetch downloads, are no longer retried and fail at once with node-fetch's `FetchError`. A retry waits as long as `Retry-After` asks, within `retryDelayMin` and `retryDelayMax`, and the feature-flag fetch that `connect()` waits for gives up after 10 seconds and uses the defaults. Once retries run out, CloudFetch downloads fail with `RetryError`; `OpenSession`, `ExecuteStatement`, request timeouts and newly retried statuses fail with the same error as before (such as `THTTPException` or the network error), and other calls with `HiveDriverError`. (#458, #605)
- The Thrift token federation exchange (`enableTokenFederation`) now goes through the connection's agent, so `proxy`, `checkServerCertificate`, `customCaCert` and `clientCert`/`clientKey` apply to it. Previously it connected directly with Node's default agent, so behind a required proxy the exchange failed and the driver fell back to the original token. (#611)
- Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608)
- Reduce result chunking and `fetchAll()` overhead for large query results.
Expand All @@ -10,7 +11,6 @@
- Honor the client `useCloudFetch: false` default on the kernel backend so complete results can be fetched without cloud-storage downloads. (#574)
- Thrift HTTP errors (`THTTPException`) now append server detail from the error headers (at most 500 characters). Kernel-backend errors now carry a non-enumerable `cause` summarizing the kernel error code and message (also at most 500 characters). Reyden fallback preserves KP001 after any existing kernel cause. (#467)
- Add `Symbol.asyncDispose` to clients, sessions, and operations, delegating to `close()`. Use `asyncDisposable()` with interface-typed results in TypeScript; existing interfaces and factory return types remain unchanged. (#331)
- Thrift `ExecuteStatement` requests are now retried when the connection fails before the request is sent (`ECONNREFUSED`, `ENOTFOUND`, `EAI_AGAIN`), within the existing `retryMaxAttempts` / `retriesTimeout` budget; once retries run out, the original connection error is thrown. Failures after the request may have reached the server are still not retried, so a statement can't run twice. (#458)
- Add the `https` connection option: set `https: false` to connect to `host` over plain HTTP (port 80 by default), for example through a trusted local reverse proxy. HTTPS remains the default, and plain HTTP should only be used on a trusted local or private link. (#503)
- Add the `rowMode` option to `executeStatement()`: `rowMode: 'array'` returns each row as an array of values in column order, so columns with the same name (e.g. `SELECT carat AS a, color AS a`) are no longer lost; the default `'object'` rows are unchanged, and the driver now logs a warning when a result has duplicate column names in object mode. (#118)
- CloudFetch retry-exhaustion errors now identify the download and the last failure: the `RetryError` message reads `CloudFetch result download failed: <original message> (last error: <cause>)`. `<cause>` is `HTTP <status>: <statusText>` (or `HTTP <status>` when the status text is empty) for HTTP failures, and `network error <code>`, `network error (<type>)` (e.g. `aborted`, when there is no `code`) or a bare `network error` for network failures (the raw network-error message, which can contain the presigned URL, is not copied). If no cause can be determined, the ` (last error: …)` suffix is omitted. The `RetryError` instance, `errorCode` and `payload` (including the node-fetch `Request` / `Response`) are unchanged. (#471)
Expand Down
9 changes: 7 additions & 2 deletions lib/FeatureFlagCache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ export default class FeatureFlagCache {

private readonly MAX_CACHE_DURATION_S = 3600;

// The flags fetch's total time, retries included: the 10 s a single attempt could already take
private readonly FETCH_DEADLINE_MS = 10 * 1000;

constructor(private context: IClientContext, private authProvider?: IAuthentication) {
this.userAgent = buildUserAgentString(this.context.getConfig().userAgentEntry);
}
Expand Down Expand Up @@ -236,11 +239,13 @@ export default class FeatureFlagCache {
const connectionProvider = await this.context.getConnectionProvider();
const agent = await connectionProvider.getAgent(url);
const retryPolicy = await connectionProvider.getRetryPolicy();
const requestConfig: RequestInit = { agent, ...init };
// connect() waits for the flags, so the attempts, the waits between them and reading the body share one deadline
const signal = AbortSignal.timeout(this.FETCH_DEADLINE_MS);
const requestConfig: RequestInit = { agent, ...init, signal: signal as unknown as RequestInit['signal'] };
const result = await retryPolicy.invokeWithRetry(() => {
const request = new Request(url, requestConfig);
return fetch(request).then((response) => ({ request, response }));
});
}, signal);
return result.response;
}

Expand Down
10 changes: 5 additions & 5 deletions lib/connection/connections/HttpConnection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ import { LogLevel } from '../../contracts/IDBSQLLogger';
import HiveDriverError from '../../errors/HiveDriverError';

import ThriftHttpConnection from './ThriftHttpConnection';
import IRetryPolicy from '../contracts/IRetryPolicy';
import IRetryPolicy, { RetryRequestClass } from '../contracts/IRetryPolicy';
import HttpRetryPolicy from './HttpRetryPolicy';
import UnsentRequestRetryPolicy from './UnsentRequestRetryPolicy';
import NonIdempotentRequestRetryPolicy from './NonIdempotentRequestRetryPolicy';

// Set by the driver, auth provider or Thrift transport. node-fetch merges
// same-named headers case-insensitively into one value, so a caller copy
Expand Down Expand Up @@ -214,7 +214,7 @@ export default class HttpConnection implements IConnectionProvider {
transport: thrift.TBufferedTransport,
protocol: thrift.TBinaryProtocol,
getRetryPolicy: () => this.getRetryPolicy(),
getUnsentRequestRetryPolicy: async () => new UnsentRequestRetryPolicy(this.context),
getNonIdempotentRequestRetryPolicy: async () => new NonIdempotentRequestRetryPolicy(this.context),
},
{
agent,
Expand All @@ -227,7 +227,7 @@ export default class HttpConnection implements IConnectionProvider {
return this.connection;
}

public async getRetryPolicy(): Promise<IRetryPolicy<HttpTransactionDetails>> {
return new HttpRetryPolicy(this.context);
public async getRetryPolicy(requestClass?: RetryRequestClass): Promise<IRetryPolicy<HttpTransactionDetails>> {
return new HttpRetryPolicy(this.context, requestClass);
}
}
127 changes: 99 additions & 28 deletions lib/connection/connections/HttpRetryPolicy.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import IRetryPolicy, { ShouldRetryResult, RetryableOperation } from '../contracts/IRetryPolicy';
import IRetryPolicy, { ShouldRetryResult, RetryableOperation, RetryRequestClass } from '../contracts/IRetryPolicy';
import { HttpTransactionDetails } from '../contracts/IConnectionProvider';
import IClientContext from '../../contracts/IClientContext';
import RetryError, { RetryErrorCode } from '../../errors/RetryError';
Expand Down Expand Up @@ -36,10 +36,81 @@ const RETRYABLE_NETWORK_ERROR_CODES = new Set([
'EHOSTUNREACH',
'ENETUNREACH',
'EPIPE',
'ENOTFOUND',
'EAI_AGAIN',
]);

// The client's certificate verification (OpenSSL's X509 codes as Node names them, and its hostname check).
// Node reports every X509 code it doesn't name as `UNSPECIFIED` (ncrypto's X509Pointer::ErrorCode), e.g. a CA
// without keyCertSign or a SHA-1 signature; the kernel doesn't retry any certificate verification failure either.
const CERTIFICATE_ERROR_CODES = [
'CERT_CHAIN_TOO_LONG',
'CERT_HAS_EXPIRED',
'CERT_NOT_YET_VALID',
'CERT_REJECTED',
'CERT_REVOKED',
'CERT_SIGNATURE_FAILURE',
'CERT_UNTRUSTED',
'DEPTH_ZERO_SELF_SIGNED_CERT',
'ERROR_IN_CERT_NOT_AFTER_FIELD',
'ERROR_IN_CERT_NOT_BEFORE_FIELD',
'ERR_TLS_CERT_ALTNAME_FORMAT',
'ERR_TLS_CERT_ALTNAME_INVALID',
'HOSTNAME_MISMATCH',
'INVALID_CA',
'INVALID_PURPOSE',
'PATH_LENGTH_EXCEEDED',
'SELF_SIGNED_CERT_IN_CHAIN',
'UNABLE_TO_DECODE_ISSUER_PUBLIC_KEY',
'UNABLE_TO_DECRYPT_CERT_SIGNATURE',
'UNABLE_TO_GET_ISSUER_CERT',
'UNABLE_TO_GET_ISSUER_CERT_LOCALLY',
'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
'UNSPECIFIED',
];

// Which statuses and network errors are retried matches the kernel (databricks-sql-kernel src/client/retry.rs
// and the CloudFetch chunk retry); backoff and budget stay Node's.
// Network errors retrying can't fix, thrown at once: certificate failures, and for RPCs a host name that doesn't
// resolve (`ENOTFOUND`; a temporary DNS failure is `EAI_AGAIN`). The kernel retries CloudFetch downloads on it.
const PERMANENT_NETWORK_ERROR_CODES: Record<RetryRequestClass, Set<string>> = {
rpc: new Set([...CERTIFICATE_ERROR_CODES, 'ENOTFOUND']),
cloudFetch: new Set(CERTIFICATE_ERROR_CODES),
};

// Idempotent requests retry every status except these (the kernel's IDEMPOTENT_NON_RETRYABLE); non-idempotent
// ones only a 429, or a 503 with `Retry-After`.
const IDEMPOTENT_NON_RETRYABLE_STATUSES = new Set([400, 401, 403, 404, 405, 409, 410, 411, 412, 413, 414, 415, 416]);

export function isRetryableStatus({ response }: HttpTransactionDetails, idempotent: boolean): boolean {
const { status } = response;
if (status >= 200 && status < 400) {
return false;
}
if (idempotent) {
return !IDEMPOTENT_NON_RETRYABLE_STATUSES.has(status);
}
return status === 429 || (status === 503 && response.headers.has('Retry-After'));
}

// A failure to connect (the kernel's `is_connect`), so the request never left the client and even a
// non-idempotent one can be retried. Node reports these as `connect <CODE> <address>`, as a socket closed
// before the TLS handshake finished, or, for a host with several addresses, as an AggregateError of the
// attempts, which has a code but no message (node-fetch's reason is then empty).
export function isConnectError(error: unknown): boolean {
const { code, message } = (error ?? {}) as { code?: unknown; message?: unknown };
if (typeof code !== 'string' || PERMANENT_NETWORK_ERROR_CODES.rpc.has(code)) {
return false;
}
return (
code === 'ECONNREFUSED' ||
code === 'EAI_AGAIN' ||
(typeof message === 'string' &&
(message.includes(`connect ${code}`) ||
message.includes('before secure TLS connection was established') ||
message.endsWith('reason: ')))
);
}

// Fallback message patterns for errors that don't carry an errno. node-fetch surfaces
// "socket hang up" as a generic FetchError, and "Premature close" when the response
// body stream closes before all data is received — both occur regularly when a
Expand All @@ -53,10 +124,13 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta

private attempt: number;

constructor(context: IClientContext) {
private readonly requestClass: RetryRequestClass;

constructor(context: IClientContext, requestClass: RetryRequestClass = 'rpc') {
this.context = context;
this.startTime = Date.now();
this.attempt = 0;
this.requestClass = requestClass;
}

public async shouldRetry(details: HttpTransactionDetails): Promise<ShouldRetryResult> {
Expand Down Expand Up @@ -135,13 +209,17 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta
throw new RetryError(RetryErrorCode.AttemptsExceeded, details);
}

// If possible, use `Retry-After` header as a floor for a backoff algorithm
// Wait as long as `Retry-After` asks (up to retryDelayMax), but don't start a wait that would end after
// retriesTimeout, as the kernel does. Without the header, back off exponentially.
const retryAfterHeader = this.getRetryAfterHeader(details, clientConfig.retryDelayMin);
const retryAfter = this.getBackoffDelay(
this.attempt,
retryAfterHeader ?? clientConfig.retryDelayMin,
clientConfig.retryDelayMax,
);
if (retryAfterHeader !== undefined) {
const retryAfter = Math.min(retryAfterHeader, clientConfig.retryDelayMax);
if (Date.now() - this.startTime + retryAfter > clientConfig.retriesTimeout) {
throw new RetryError(RetryErrorCode.TimeoutExceeded, details);
}
return { shouldRetry: true, retryAfter };
}
const retryAfter = this.getBackoffDelay(this.attempt, clientConfig.retryDelayMin, clientConfig.retryDelayMax);

return { shouldRetry: true, retryAfter };
}
Expand Down Expand Up @@ -172,11 +250,14 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta
}
const candidate = error as { code?: string; type?: string; message?: string };

// node-fetch FetchError surfaces low-level network failures with `type: 'system'`
// and a body-stream timeout with `type: 'body-timeout'`. Both should be retried;
// `request-timeout` is converted to a Thrift TApplicationException upstream so
// we don't need to retry it here.
if (candidate.type === 'system' || candidate.type === 'body-timeout') {
if (typeof candidate.code === 'string' && PERMANENT_NETWORK_ERROR_CODES[this.requestClass].has(candidate.code)) {
return false;
}

// node-fetch FetchError surfaces low-level network failures with `type: 'system'`, no response
// within the request timeout as `request-timeout` (retried like the kernel's `is_timeout`), and a
// body-stream timeout as `body-timeout`.
if (candidate.type === 'system' || candidate.type === 'request-timeout' || candidate.type === 'body-timeout') {
Comment thread
vuanhphung marked this conversation as resolved.
Resolved
return true;
}

Expand All @@ -191,31 +272,21 @@ export default class HttpRetryPolicy implements IRetryPolicy<HttpTransactionDeta
return false;
}

protected isRetryable({ response }: HttpTransactionDetails): boolean {
const statusCode = response.status;

const result =
// Retry on all codes below 100
statusCode < 100 ||
// ...and on `429 Too Many Requests`
statusCode === 429 ||
// ...and on all `5xx` codes except for `501 Not Implemented`
(statusCode >= 500 && statusCode !== 501);

return result;
protected isRetryable(details: HttpTransactionDetails): boolean {
return isRetryableStatus(details, true);
}

protected getRetryAfterHeader({ response }: HttpTransactionDetails, delayMin: number): number | undefined {
// `Retry-After` header may contain a date after which to retry, or delay seconds. We support only delay seconds.
// Value from `Retry-After` header is used when:
// 1. it's available and is non-empty
// 2. it could be parsed as a number, and is greater than zero
// 3. additionally, we clamp it to not be smaller than minimal retry delay
// 3. additionally, we clamp it to not be smaller than minimal retry delay (in milliseconds)
const header = response.headers.get('Retry-After') || '';
if (header !== '') {
const value = Number(header);
if (Number.isFinite(value) && value > 0) {
return Math.max(delayMin, value);
return Math.max(delayMin, value * 1000);
}
}
return undefined;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
import HttpRetryPolicy from './HttpRetryPolicy';
import HttpRetryPolicy, { isConnectError, isRetryableStatus } from './HttpRetryPolicy';
import { HttpTransactionDetails } from '../contracts/IConnectionProvider';
import { RetryableOperation } from '../contracts/IRetryPolicy';
import RetryError from '../../errors/RetryError';

// Raised before a connection exists, so no request bytes were written. Failures on a connected
// socket (ECONNRESET, EPIPE, timeouts, ...) are excluded: the server may already have the request.
const UNSENT_REQUEST_ERROR_CODES = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']);

// For operations that are not safe to repeat: retries only requests that never left the client.
export default class UnsentRequestRetryPolicy extends HttpRetryPolicy {
// For operations that are not safe to repeat: retries only requests the server didn't run, as the kernel's
// non-idempotent strategy does: a failure to connect, a 429, or a 503 with `Retry-After`. Failures on a
// connected socket (ECONNRESET, EPIPE, timeouts, ...) are not retried: the server may already have the request.
export default class NonIdempotentRequestRetryPolicy extends HttpRetryPolicy {
public async invokeWithRetry(operation: RetryableOperation<HttpTransactionDetails>): Promise<HttpTransactionDetails> {
try {
return await super.invokeWithRetry(operation);
Expand All @@ -21,12 +19,11 @@ export default class UnsentRequestRetryPolicy extends HttpRetryPolicy {
}
}

protected isRetryable(): boolean {
return false;
protected isRetryable(details: HttpTransactionDetails): boolean {
return isRetryableStatus(details, false);
}

protected isRetryableNetworkError(error: unknown): boolean {
const code = (error as { code?: unknown } | undefined)?.code;
return typeof code === 'string' && UNSENT_REQUEST_ERROR_CODES.has(code);
return isConnectError(error);
}
}
Loading
Loading