Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased

- Restore `docs/readme.md` (getting started) and `docs/troubleshooting.md`, rewritten for the current driver, so the links from the Databricks documentation work again. (#274)
- CloudFetch retry-exhaustion errors now identify the download and the last failure: the `RetryError` message reads `CloudFetch result download failed: <original message> (last error: <cause>)`. `<cause>` is `HTTP <status>: <statusText>` (or `HTTP <status>` when the status text is empty) for HTTP failures, and `network error <code>`, `network error (<type>)` (e.g. `aborted`, when there is no `code`) or a bare `network error` for network failures (the raw network-error message, which can contain the presigned URL, is not copied). If no cause can be determined, the ` (last error: …)` suffix is omitted. The `RetryError` instance, `errorCode` and `payload` (including the node-fetch `Request` / `Response`) are unchanged. (#471)
- Report untyped NULL columns as STRING in kernel result metadata, matching Thrift. (#470)
- Reject non-scalar query parameter values at bind time with a `ParameterError` on both the Thrift and kernel backends, instead of stringifying them (which could leave a Thrift operation stuck before FINISHED). Newly rejected on the Thrift backend: arrays, plain objects, boxed primitives (e.g. `new String('x')`, `new Number(1)`), functions, symbols, and a `DBSQLParameter` that wraps another `DBSQLParameter`. Primitives, `Date`, `Int64`, and `DBSQLParameter` are still accepted.
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ client
});
```

## Documentation

- [Getting started](docs/readme.md): connecting, authentication, parameters, and reading results.
- [Troubleshooting](docs/troubleshooting.md): logging, common errors, proxies, TLS, and timeouts.

## Configuration

See [CONNECTION_PARAMETERS.md](CONNECTION_PARAMETERS.md) for every connection,
Expand Down
97 changes: 97 additions & 0 deletions docs/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Getting started

The Databricks SQL Driver for Node.js (`@databricks/sql`) runs SQL on Databricks SQL warehouses and clusters. This page is a short tour; the [README](../README.md) covers installation and [CONNECTION_PARAMETERS.md](../CONNECTION_PARAMETERS.md) lists every option. If something goes wrong, see [Troubleshooting](troubleshooting.md).

## Requirements and installation

The driver needs Node.js 22.23.4+, 24.22.1+, or 26.12.1+ (see `engines` in [package.json](../package.json)).

```bash
npm i @databricks/sql
```

## Connect and run a query

```javascript
const { DBSQLClient } = require('@databricks/sql');

async function main() {
const client = new DBSQLClient();
await client.connect({
host: process.env.DATABRICKS_HOST, // e.g. 'dbc-a1b2345c-d6e7.cloud.databricks.com', no 'https://'
path: process.env.DATABRICKS_HTTP_PATH, // e.g. '/sql/1.0/warehouses/a1b234c567d8e9fa'
token: process.env.DATABRICKS_TOKEN,
});

const session = await client.openSession();
const operation = await session.executeStatement('SELECT :greeting AS greeting', {
namedParameters: { greeting: 'Hello, World!' },
});
const rows = await operation.fetchAll(); // [{ greeting: 'Hello, World!' }]
await operation.close();

console.table(rows);

await session.close();
await client.close();
}

main().catch((error) => {
console.error(error);
process.exitCode = 1;
});
```

Copy `host` and `path` from the **Connection details** tab of your SQL warehouse or cluster.

## Authentication

A personal access token in `token` is the default (`authType: 'access-token'`). For OAuth, set `authType: 'databricks-oauth'`:

- With `oauthClientId` and `oauthClientSecret`, the driver uses machine-to-machine OAuth with a service principal.
- Without a secret, the driver uses user-to-machine OAuth: it opens a browser and listens on `localhost:8030` for the redirect.

Token-based modes (`'static-token'`, `'external-token'`, `'token-provider'`, and token federation) are shown in [examples/tokenFederation](../examples/tokenFederation/README.md). The [authentication table](../CONNECTION_PARAMETERS.md#authentication) lists every auth option and which backend supports it.

## Sessions

`client.openSession()` accepts `initialCatalog`, `initialSchema`, `configuration` (session configuration such as `STATEMENT_TIMEOUT`), and `queryTags`. One client can open many sessions. Besides `executeStatement`, a session has metadata methods: `getCatalogs`, `getSchemas`, `getTables`, `getTableTypes`, `getColumns`, `getFunctions`, `getPrimaryKeys`, `getCrossReference`, `getTypeInfo`, and `getInfo`. See [examples/session.js](../examples/session.js) and [examples/session_params.js](../examples/session_params.js).

## Query parameters

Pass values with `namedParameters` (`:name` markers) or `ordinalParameters` (`?` markers), but not both in one statement. The driver infers each parameter's SQL type from its JavaScript value. To set the type explicitly, wrap the value:

```javascript
const { DBSQLParameter, DBSQLParameterType } = require('@databricks/sql');

await session.executeStatement('SELECT * FROM orders WHERE order_date >= ?', {
ordinalParameters: [new DBSQLParameter({ type: DBSQLParameterType.DATE, value: '2024-01-31' })],
});
```

## Reading results

Each statement returns an operation. Rows are plain objects keyed by column name.

- `fetchAll()` returns every row.
- `fetchChunk({ maxRows })` with `hasMoreRows()` reads the result in pages.
- `iterateRows()` / `iterateChunks()` are async iterators, and `toNodeStream()` returns a Node.js `Readable`.
- `getResultMetadata()` returns the result schema.
- `cancel()` stops a running statement ([examples/cancel_operation.js](../examples/cancel_operation.js)).

By default, DECIMAL and BIGINT values are converted to JavaScript `number`, which can lose precision. Set `preserveBigNumericPrecision: true` in `connect()` to get exact strings for DECIMAL and `bigint` for BIGINT. [examples/data_types.js](../examples/data_types.js) shows how other SQL types are returned.

## Closing

Close operations, sessions, and the client when you're done. `session.close()` closes that session's open operations, and `client.close()` closes the client's open sessions. Long-running applications should `await client.close()` at shutdown so buffered [telemetry](TELEMETRY.md) is flushed.

## Backends

By default the driver uses the Thrift protocol. Passing `useKernel: true` to `connect()` opts into an experimental backend built on the Databricks SQL kernel (a native module installed through the driver's optional dependencies). This option is internal and unstable: it isn't in the published TypeScript types and may change or be removed without notice. Some options behave differently on each backend; [CONNECTION_PARAMETERS.md](../CONNECTION_PARAMETERS.md) has a Thrift and a Kernel column for every option.

## More resources

- [Troubleshooting](troubleshooting.md): logging, errors, proxies, TLS, and timeouts.
- [examples](../examples): runnable scripts, including [logging](../examples/logging.js) and [query tags](../examples/query_tags.js).
- [CHANGELOG.md](../CHANGELOG.md): release notes.
- [Databricks documentation for the Node.js driver](https://docs.databricks.com/en/dev-tools/nodejs-sql-driver.html).
116 changes: 116 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Troubleshooting

This page covers common problems with the Databricks SQL Driver for Node.js. [Getting started](readme.md) is the overview, and [CONNECTION_PARAMETERS.md](../CONNECTION_PARAMETERS.md) is the full option reference. "Kernel" below means the experimental backend you get with `useKernel: true`; everything else applies to the default Thrift backend.

## Turn on debug logging

Start here. Pass a `DBSQLLogger` to the client:

```javascript
const { DBSQLClient, DBSQLLogger, LogLevel } = require('@databricks/sql');

const logger = new DBSQLLogger({ level: LogLevel.debug, filepath: 'databricks-sql.log' });
const client = new DBSQLClient({ logger });
```

- The levels are `error`, `warn`, `info` (the default), and `debug`. Without a logger, the driver logs JSON lines to the console at `info`. To quiet it, pass `new DBSQLLogger({ level: LogLevel.error })` as the client's logger.
- `filepath` also writes the log to that file. Point it at a writable file path, not a directory.
- `logger.setLevel(LogLevel.debug)` changes the level at runtime ([examples/logging.js](../examples/logging.js)).
- A custom logger is any object with a `log(level, message)` method, so you can route driver logs into your application's logger.
- On Kernel, native kernel logs go to the same logger, prefixed with `[kernel <target>]`, at the logger's level. The kernel log sink is shared by the whole process, so with several clients the most recently connected client's logger receives them.

Debug logs include host names and session and statement IDs. Review them before sharing.

## Reading errors

The driver's error classes aren't exported, so check error properties (`name`, `message`, `statusCode`, `sqlState`, `errorCode`) rather than using `instanceof`. These are the errors you'll typically see:

| Situation | Thrift | Kernel |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Invalid or expired token | `Received a response with a bad HTTP status code: 401` (or `403`), with `statusCode` set | `name: 'AuthenticationError'`, a message such as `HTTP 403: ... Invalid access token`, and `kernelMetadata.httpStatus` |
| Wrong `path` | `Received a response with a bad HTTP status code: 404` | `HTTP 404: The warehouse <id> was not found.` |
| Unknown host or network failure | `FetchError` with a `code` such as `ENOTFOUND`, `ECONNREFUSED`, or `ECONNRESET` | `HTTP request failed after N attempts: error sending request for url (...)` |
| Untrusted TLS certificate | `FetchError` with a `code` such as `SELF_SIGNED_CERT_IN_CHAIN`, `UNABLE_TO_GET_ISSUER_CERT_LOCALLY`, or `UNABLE_TO_VERIFY_LEAF_SIGNATURE` | The same message as a network failure; see [TLS and certificates](#tls-and-certificates) |
| SQL error from the server | The server's message, `errorCode: 'ERROR'`, and `sqlState` (for example `42P01` for a missing table)¹ | Same as Thrift |
| Statement timeout | `Statement has timed out after N seconds.` with `sqlState` `57KD0`¹ | Same as Thrift |
| Retries exhausted | `Hive driver: <status> when connecting to resource. Max retry count exceeded.` (or `Retry timeout exceeded.`) | Kernel's own error message |

¹ In 3.0.0 and earlier, Thrift errors expose the SQLSTATE only as `error.response.sqlState`.

The client is an `EventEmitter`. Thrift transport errors that aren't tied to a pending call are logged and emitted as `error` events, so register `client.on('error', ...)` in long-running applications. Older versions of this page recommended reconnecting on the client's `close` event. That's no longer needed: the driver uses independent HTTPS requests, so there is no persistent connection to re-establish and the transport doesn't emit `close`.

## Connection settings

- `host` is the bare host name from **Connection details**, without `https://` or a path.
- `path` is the HTTP path, for example `/sql/1.0/warehouses/<id>`. A wrong path returns HTTP 404.
- Thrift reads the port from `port` (default `443`). Kernel ignores `port`; put a non-default port in `host` instead (`host: 'example.com:8443'`).

## Authentication

- **Personal access tokens:** check that the token hasn't expired or been revoked, and that it belongs to the workspace in `host`.
- **OAuth user-to-machine** (`authType: 'databricks-oauth'` without `oauthClientSecret`) opens a browser and waits for the redirect on `localhost:8030`. If that port is taken, Thrift fails with `Failed to start server: all ports are in use`. On servers and in CI, use machine-to-machine OAuth (`oauthClientId` plus `oauthClientSecret`) instead.
- **Azure:** Thrift uses Microsoft Entra ID OAuth by default. Set `useDatabricksOAuthInAzure: true` to use Databricks OAuth instead. Thrift also throws `OAuth is not supported for <host>` when `host` isn't a recognized Databricks domain, for example a custom DNS alias.
- **Kernel** supports `'access-token'`, `'static-token'`, and `'databricks-oauth'`. It rejects `'custom'`, `'token-provider'`, and `'external-token'` with `kernel backend: unsupported auth mode`. See [authentication](../CONNECTION_PARAMETERS.md#authentication) for the per-backend details.

## Proxies

Configure the proxy explicitly with the `proxy` option:

```javascript
await client.connect({
host,
path,
token,
proxy: {
protocol: 'http',
host: 'proxy.example.com',
port: 8080,
auth: { username: 'user', password: 'pass' }, // optional
},
});
```

- Thrift accepts `http`, `https`, `socks`, `socks4`, `socks4a`, `socks5`, and `socks5h`. Kernel supports only HTTP and HTTPS proxies.
- Thrift ignores the `HTTPS_PROXY` and `HTTP_PROXY` environment variables. Kernel currently picks up `HTTPS_PROXY` when `proxy` isn't set.
- If the proxy inspects TLS traffic, also follow the next section.

## TLS and certificates

Both backends verify the server certificate by default. Certificate errors usually mean a TLS-inspecting proxy or a private CA sits between you and Databricks.

- Pass the CA certificate (PEM, as a string or `Buffer`) that issued the proxy's or server's certificate as `customCaCert`. Both backends add it to their default trust roots.
- Thrift also honors the `NODE_EXTRA_CA_CERTS` environment variable. When `customCaCert` is set, Thrift builds its trust store from Node's bundled roots, `NODE_EXTRA_CA_CERTS`, and `customCaCert`, so roots that Node would otherwise load from the operating system (`--use-system-ca`) aren't included.
- Kernel has its own TLS stack and doesn't read `NODE_EXTRA_CA_CERTS` or `--use-system-ca`. Use `customCaCert`. Because Kernel reports certificate failures as `error sending request for url`, try the same connection on Thrift (without `useKernel`) to see the certificate error code.
- For mutual TLS, pass `clientCert` and `clientKey` together. The driver rejects one without the other. PKCS#8 keys (`BEGIN PRIVATE KEY`) are recommended for Kernel.
- `checkServerCertificate: false` turns verification off completely and exposes your token to man-in-the-middle attacks. Use it only for local debugging, never in production.

## Timeouts and retries

- **Long-running queries:** on SQL warehouses, limit run time with `STATEMENT_TIMEOUT`, in seconds. Set it per session with `openSession({ configuration: { STATEMENT_TIMEOUT: '600' } })`, or run `SET STATEMENT_TIMEOUT = 600`. Both backends honor it. The `queryTimeout` statement option (seconds) applies only to clusters on Thrift. To stop a query from the client, call `operation.cancel()`.
- **`socketTimeout`** (milliseconds) limits how long the driver waits on a single HTTP request. It defaults to 15 minutes on Thrift and 120 seconds on Kernel. On Thrift a timeout surfaces as `Request timed out`; `0` disables the timeout on Thrift, while Kernel ignores `0` and keeps its default.
- **Retries** are controlled by `retryMaxAttempts` (total attempts, default `5`), `retriesTimeout` (default 15 minutes), `retryDelayMin` (default 1 second), and `retryDelayMax` (default 60 seconds); all times are in milliseconds. Thrift retries HTTP 429, 5xx (except 501), and transient network errors with exponential backoff, but only for requests that are safe to repeat: status polling, metadata calls, closing operations and sessions, and CloudFetch downloads. Opening a session and executing a statement aren't retried. Kernel runs its own retry loop using the same settings.

## Kernel native module

Kernel loads a native module from the platform-specific optional dependency `@databricks/databricks-sql-kernel-<platform>`. Published builds cover Linux x64 and arm64 (glibc and musl), macOS x64 and arm64, and Windows x64 and arm64.

- `kernel native binding not installed for platform ...`: the optional dependency is missing. Don't install with `--omit=optional` (or `--no-optional`), and run `npm install` on the target OS and architecture. For example, don't copy `node_modules` from a macOS host into a Linux container.
- `kernel native binding present but failed to dlopen ...`: usually a glibc/musl or CPU architecture mismatch. Reinstall on the target platform.
- Without `useKernel`, Thrift still switches to Kernel automatically when a warehouse rejects Thrift sessions with SQLSTATE `KP001`. That fallback also needs the native module.

## Results

If DECIMAL or BIGINT values lose precision, set `preserveBigNumericPrecision: true` in `connect()`. DECIMAL values then come back as strings and BIGINT values as `bigint`.

## Telemetry

Set `telemetryEnabled: false` in `connect()`, or set the environment variable `DATABRICKS_TELEMETRY_DISABLED=1`, to turn telemetry off. See [TELEMETRY.md](TELEMETRY.md) for details and its [troubleshooting section](TELEMETRY.md#troubleshooting).

## Reporting a problem

[Open an issue](https://github.com/databricks/databricks-sql-nodejs/issues) and include:

- the driver version (`npm ls @databricks/sql`), the Node.js version, and the OS
- whether `useKernel` is set
- a minimal script that reproduces the problem
- a debug log with tokens and other secrets removed
Loading