From e99c7084377f754486e69b5effce82ebb1c7e75f Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Thu, 1 Oct 2026 21:48:46 +0100 Subject: [PATCH 1/3] docs(qwp): specify refreshing bearer-token providers for Entra ID MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a cross-language specification for QWP clients that authenticate with rotating bearer tokens (Microsoft Entra ID managed identities and service principals), plus the Java findings and implementation plan. - design/qwp-token-provider-spec.md (v0.3, decisions resolved): the token-source contract, a proactively refreshing shared token cache, when clients fetch tokens, failure policy by phase (one retry after 401), connect-string keys (token_provider, azure_resource, azure_client_id), connection health, an optional authentication- outage deadline, redaction rules and conformance tests C1-C23. - design/entra-id-qwp-auth.md: findings for the Java client with code references, the design rationale, and the four-step Java plan (§10). Security-relevant server findings are reported privately and are not part of this change. --- design/entra-id-qwp-auth.md | 556 +++++++++++++++++++++++++++++ design/qwp-token-provider-spec.md | 575 ++++++++++++++++++++++++++++++ 2 files changed, 1131 insertions(+) create mode 100644 design/entra-id-qwp-auth.md create mode 100644 design/qwp-token-provider-spec.md diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md new file mode 100644 index 000000000..03985fce8 --- /dev/null +++ b/design/entra-id-qwp-auth.md @@ -0,0 +1,556 @@ +# Entra ID app-only auth over QWP: findings and proposed design + +Status: **draft for review**. No production code has been written. This file is not committed. + +**Read these first:** + +- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.3, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. +- **Java plan:** §10. +- **Line references** are as of `c9968f24` and may drift. + +Code inspected: + +- **Client:** `java-questdb-client` `main` @ `c9968f24`. +- **Server:** QuestDB Enterprise at the time of writing (private source, so this doc describes server behaviour only), and the open-source `questdb` core @ `c06b4c8267`. +- **Rust core** (what the Python client wraps): `c-questdb-client` `main` @ `5ea51a47`. +- **Spec:** `documentation` @ `7c35c02`, files `qwp-ingress-websocket.md` and `qwp-egress-websocket.md`. The spec pins client `67bb5e4` (2026-05-13). + +Abbreviations for client files, all under `core/src/main/java/io/questdb/client/`: + +| Abbrev. | File | +|---|---| +| `S` | `Sender.java` | +| `QWS` | `cutlass/qwp/client/QwpWebSocketSender.java` | +| `CWSL` | `cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java` | +| `BD` | `cutlass/qwp/client/sf/cursor/BackgroundDrainer.java` | +| `QQC` | `cutlass/qwp/client/QwpQueryClient.java` | +| `WSC` | `cutlass/http/client/WebSocketClient.java` | + +## TL;DR + +1. **The token-provider hook already exists and is wired into every QWP upgrade path.** + - The hook is `HttpTokenProvider`, added in #52. + - With a provider, Java pulls a fresh token on every connect round: the initial connect, reconnect, failover, orphan drainers, pool recovery, egress connect and egress failover. + - Only a static `token=` / `httpToken()` is captured once. + - So the work is not plumbing. What's missing is: + - an expiry-aware, shared, proactively refreshing cache; + - classifying provider failures at startup; + - a "token rejected" signal from the client back to the provider; + - a way to select a provider from a connect string; + - spec updates and Rust/Python parity. +2. **The brief's premise ("401 is terminal") no longer matches Java, but it does match the Rust core.** + - Since #66 and #52, a Java foreground sender that has connected once retries 401/403 forever while store-and-forward (SF) buffers. + - 401/403 is terminal only during initialization, for orphan drainers (with a bounded ride-out for rotating credentials), and on egress. + - The Rust core treats `AuthError` as terminal on reconnect and sends a static header. That is the Python customer's likely client, and it has exactly the failure described in the brief. + - The public spec is stale on both points. +3. **The server works for app-only Entra tokens, but only in local-JWT mode.** Required settings: + - `acl.oidc.groups.encoded.in.token=true`, `acl.oidc.groups.claim=roles`, `acl.oidc.sub.claim=oid`; + - QuestDB app registration set to issue v2 tokens. + + The default UserInfo mode cannot work for app-only tokens. A QuestDB 401 also covers "IdP or JWKS unreachable", so it is not proof that the token is bad. +4. **Recommendation:** do (a) now (core cache and SPI) plus (c), a thin optional `questdb-client-azure` module. That module adapts `TokenCredential` and registers a connect-string provider through `ServiceLoader`. Defer (b), the built-in IMDS provider. + +## 1. Current auth configuration + +**Ingress (`Sender`)** + +- **Schema and TLS:** + - `ws::` or `wss::` (`S:3540-3549`); `wss` enables TLS. + - TLS keys: `tls_verify=on|unsafe_off`, `tls_roots` (PEM or JKS), `tls_roots_password`. All are rejected on `ws::` (`S:4245-4250`). + - Builder equivalents: `enableTls()` (`S:2124`), and `advancedTls().customTrustStore(..)` / `.disableCertificateValidation()` (`S:4572-4624`). +- **Auth keys:** + - Keys are `username`/`password` (aliases `user`/`pass`) and `token`, registered in `impl/ConfigSchema.java:49-59`. + - Unknown keys are rejected (`impl/ConfigView.java:74-86`). + - The keys are applied in `fromConfigWebSocket` (`S:4029-4036`). + - Cross-key rules in `validateWsConfig` (`S:4227-4255`): Basic auth needs both halves, and `token` is exclusive with Basic. +- **Builder:** + - `httpToken` (`S:2284`), `httpUsernamePassword` (`S:2372`), `httpTokenProvider` (`S:2343`); all three are mutually exclusive. + - `auth_timeout_ms` bounds the upgrade (default 15 s). `connect_timeout` bounds the TCP connect. +- **Where the header is built:** `buildWebSocketAuthHeader` (`S:3427-3460`). + - Basic and static Bearer values go through `QwpWebSocketSender.fixedAuthHeader`, which tags them as constant. + - A provider becomes a lambda that calls `getToken()`, snapshots it, runs `HttpTokenProvider.validateToken`, and returns `"Bearer " + token`. + - That lambda is passed as `Supplier` to `QwpWebSocketSender.connectWithCredentialSupplier` (`S:1468`, `S:1694`). +- **On the wire:** + - `WSC.upgrade(path, timeout, authorizationHeader)` writes `Authorization: ` (`WSC:640-722`, header at `:700-704`). + - The ingress path is `/write/v4` (`QWS:148`). The client never uses `/api/v4/write`. + +**Egress (`QwpQueryClient`)** + +- Same keys, read in `fromConfig` (`QQC:375-490`, auth at `:417-419` and `:485-486`). +- Programmatic API: `withBasicAuth`, `withBearerToken`, `withBearerTokenProvider` (`QQC:1127`, `:1147`, `:1177`; mutually exclusive), plus `withTls()` / `withTrustStore()` / `withInsecureTls()`. +- The header is built in `resolveAuthorizationHeader()` (`QQC:1933-1960`). +- The upgrade to `/read/v1` happens in `runUpgradeWithTimeout` (`QQC:1971-1996`). + +**`QuestDB` facade** + +- `QuestDBBuilder.httpTokenProvider` (`QuestDBBuilder.java:330`) rejects a provider combined with `token`, `username` or `password` in the config (`:190-194`). There is also `QuestDB.connect(cs, provider)` (`QuestDB.java:105`). +- One provider instance is shared by every pooled sender (`SenderPool.java:540-544`, `:2046-2051`) and every pooled query client (`QueryClientPool.java:599-600`). + +## 2. Captured once, or re-read per upgrade? + +| Path | Code | Credential pull | +|---|---|---| +| Static `token=` / `httpToken` / Basic | `S:3431-3439`, `FixedAuthHeader` `QWS:5328` | **Captured once.** An expired token is presented forever. | +| Provider: ingress initial connect (OFF/SYNC) | `ensureConnected` `QWS:3999-4055` → `buildAndConnect` → `connectWalk` | Once per round, before the endpoint walk (`QWS:3305-3351`), on the thread calling `build()`. | +| ASYNC initial connect, and every reconnect | `CWSL.connectLoop` (`:1731`) → `reconnectFactory.reconnect(cancellation)` | Once per round, on the cursor I/O thread. | +| Failover to another host | Inside the same round (`QWS:3352-3394`) | **The same token is reused for every endpoint in the round**; the next round re-pulls. This is deliberate: a token is cluster-wide. | +| SF replay after reconnect | Replay starts on the new socket after the upgrade | No extra auth; covered by the reconnect pull. | +| Orphan drainers (SF recovery of other slots) | Background `ReconnectSupplier`, `QWS:2860`; same supplier | Once per round, on a drainer-pool thread. | +| `SenderPool` startup-recovery delegates | `SenderPool.java:2079-2081` | OFF-mode build with the provider. | +| Egress connect | `QQC:754-786` | Once per `connect()` walk. | +| Egress mid-query failover | `QQC:1576-1683` → `reconnectViaTracker` `:1865-1931` | Once per failover reconnect. | + +These behaviours are already pinned by tests: + +- `WebSocketTokenProviderTest`: `testProviderRequeriedOnEveryReconnect`, `testThrowingProviderResolvedOncePerConnectRound`, and others. +- `QwpQueryClientTokenProviderTest.testProviderTokenReResolvedOnFailoverReconnect`. +- `SenderPoolSfTokenProviderTest`. + +## 3. Failure classification + +**Where failures are classified** + +- `QwpUpgradeFailures.classify` (`:41-57`): + - 421 with `X-QuestDB-Role` → role reject (transient); + - 401 or 403 → `QwpAuthFailedException`; + - anything else passes through as `WebSocketUpgradeException` or `HttpClientException`. +- A provider that throws is wrapped as `QwpCredentialUnavailableException` (`QWS:3334-3344`). +- The policy applied to each class is decided per phase: + +| Phase | 401/403 | Other non-421 upgrade reject (404/426/5xx) | Provider threw | +|---|---|---|---| +| OFF initial connect (the default) | thrown from `build()` | thrown: latched as `terminalUpgradeError` (`QWS:3442-3447`), thrown at round end (`:3593-3596`) | the provider's own exception is thrown (`QWS:4040-4054`) | +| SYNC initial connect (`initial_connect_retry=on`, or any `reconnect_*` key set) | terminal, no retry (`CWSL:1041-1052`) | terminal, no retry | **fails fast** (`CWSL:1053-1065`); pinned by `testThrowingProviderFailsFastInSyncInitialConnect` | +| ASYNC, before the first connect | terminal: goes to the error inbox and is rethrown on `close()` (`CWSL:1825-1867` via `endpointPolicyFailureIsTerminal`, `:2111`) | terminal | retried forever (`CWSL:1928-1963`) | +| Foreground sender after its first connect | **retried forever** with capped backoff; a RETRIABLE `SECURITY_ERROR` is dispatched per attempt (`CWSL:1876-1886`) | retried forever | retried forever | +| Orphan drainer | constant credential: slot quarantined (`.failed`). Rotating credential: ride-out of ≥6 attempts **and** ≥ min(`reconnect_max_duration`, 5 min), capped at 256 attempts (`BD:97-160`, `:481-556`) | quarantined | retried forever | +| Egress connect / failover | thrown from `connect()` (`QQC:785`); during failover, `onError("auth failure during failover reconnect")` (`:1665-1676`) | next endpoint; error if every endpoint fails | `LineSenderException`; during failover, `onError("failover reconnect failed")` | + +**Where the brief, the spec and the code disagree** + +- **Spec vs Java.** The spec (`qwp-ingress-websocket.md:1055-1063`) says "401/403 terminal at any host" and "all other upgrade errors transient (404, 426, 503…)". It was accurate at `67bb5e4`: the reconnect loop issued `HALT` on 401/403 (verified). #66 (`37d4b0ab`) and #52 (`0b9b5766`) changed that to the table above. Also, non-421 4xx/5xx is **terminal** during initialization, contrary to the spec. For example, a single node answering 503 fails a SYNC `build()` immediately. +- **Rust core (Python).** + - The header is a static `auth_header: Option` (`questdb-rs/src/ingress.rs:497`, `:599`, `:664`). + - `reconnect_error_is_terminal` treats `AuthError` as terminal on reconnect (`questdb-rs/src/ingress/sender/qwp_ws_driver.rs:2596-2603`). + - So the brief's failure mode is real there. +- **Egress pool (suspected bug; not yet reproduced).** After a failed failover reconnect, the client is left with `connected=false` (`QQC:1623`). The worker still returns to the pool, and every later `execute()` throws `"QwpQueryClient not connected"` (`QQC:1581`). `reapIdle` never removes a worker while the pool is at `query_pool_min` (`QueryClientPool.java:500`). A 401 during failover is one way in. This needs a red test before anything else. + +## 4. Threading and the initial-connect budget + +**Which thread connects** + +- **OFF/SYNC initial connect:** the thread calling `Sender.build()` or `QuestDB.build()` (pool pre-warm). Pool growth runs on the acquiring thread; recovery delegates run on the housekeeper thread. None of these paths has a `ConnectCancellation`. +- **ASYNC initial connect and every reconnect:** the sender's cursor I/O thread. +- **Orphan drainers:** `BackgroundDrainerPool` threads. +- **Egress:** the caller of `connect()`; failover runs on the thread executing `execute()` (in the pool, the `QueryWorker` dispatch thread). + +**What can block within one attempt** + +- **`getToken()`:** this is caller code, and nothing in the client bounds it. `close()` interrupts it, but only on I/O-thread and drainer paths (`CWSL` `ConnectCancellation.cancel`, near `:3764-3790`). +- **DNS:** bounded only by the OS. +- **TCP connect:** bounded by `connect_timeout`. The foreground default is 0, which means the OS SYN timeout of 60-130 s. Background connects default to 15 s (`QWS:3139-3141`). +- **TLS handshake:** bounded by `connect_timeout`, or by the request timeout when that is unset. +- **Upgrade:** bounded by `auth_timeout_ms`. +- Producers are not blocked by any of this until SF fills and append backpressure kicks in. + +**How the budget interacts with a slow token fetch** + +- `connectWithRetry` checks `reconnect_max_duration_millis` only *between* attempts (`CWSL:1029`). +- A slow `getToken()` therefore stretches `build()` past the budget: an attempt is never cut short. +- A throwing `getToken()` ends SYNC immediately. +- ASYNC and reconnect have no budget (Invariant B). +- **Entra consequence:** + - The first `DefaultAzureCredential` call runs unbounded on the `build()` thread. That call includes chain probing, an IMDS cold start, and the SDK's own retries on 429. + - An IMDS 429 that escapes the SDK fails OFF and SYNC startup. + +## 5. Constraints from repo conventions + +- **Dependencies:** + - The core's only runtime dependency is `slf4j-api` (annotations are `provided`). + - Java 8 is the API floor, and the code must also compile on JDK 11+. + - So no `azure-core` or `azure-identity` in core. `ServiceLoader`, `CompletableFuture` and `ScheduledExecutorService` are all Java 8. +- **Allocation:** + - Zero-GC is mandatory on per-row producer calls and the steady-state I/O loop (`.pi/skills/review-pr/SKILL.md:37`, `:583-590`). + - Connect paths already allocate: a `WebSocketClient` per attempt, and `"Bearer " + token`. + - A cache hit only needs to be cheap: no I/O and no locks. +- **Config flow:** + - Ingress: string → `ConfigString` → `ConfigView` (strict `ConfigSchema` registry) → `fromConfigWebSocket` setters → `build()` → `connectWithCredentialSupplier(..., Supplier, ...)`. + - Egress: `QwpQueryClient.fromConfig` → `with*` setters. + - The facade validates both sides (`QuestDBBuilder.java:184-199`). + - New keys must go into `ConfigSchema` and are shared vocabulary with every client. +- **Test harnesses:** + - `TestWebSocketServer`: captures `Authorization`, supports `setRejectWithStatus` and `setRejectWithRole` and drop handlers, and caps requests at 8 KB. + - `MockOidcServer`: a raw-socket HTTP mock that can return JSON, chunked, dropped or dribbled responses. + - `assertMemoryLeak`, and `HandOffCharSequence` for TOCTOU tests. + - Scripted `ReconnectFactory` stubs, as in `BackgroundDrainerMidDrainAuthRejectTest`. +- **API compatibility:** `ExportedApiCompatibilityTest` pins exported signatures. Adding is fine; retyping is not. + +## 6. Problems with ~2 KB rotating bearer tokens + +**Size limits** + +- **Client buffers:** fine. The upgrade request buffer starts at ≥64 KiB and grows (`WSC:181-186`, `WebSocketSendBuffer.java:504-515`). The response buffer is ≥64 KiB. +- **Server header buffer:** + - Production default is 64,448 B (`PropServerConfiguration.java:1213`; the code says `32 * 2014`, a harmless typo for 1024). + - `DefaultHttpContextConfiguration`, used by embedded and test servers, allows only 4,096 B. A 2 KB token fits; a role-heavy token may not. + - `TestWebSocketServer` drops any request over 8,192 B. +- **Connect string:** no length limit. A `;` must be doubled, and control characters are rejected (`ConfStringParser.java:164-183`). JWTs are safe. + +**Rotation** + +- One token serves the whole round. +- With the foreground `connect_timeout=0`, a round across several black-holed hosts can run for minutes, so a near-expiry token can expire mid-round. +- The cache's freshness margin and the 401 retry (§7.7) cover this. + +**Where a token could leak** + +- **Already safe:** + - `validateToken` never echoes the token (`HttpTokenProvider.java:70-80`). + - `WebSocketUpgradeException` carries only the status line (`WSC:1346-1357`). + - `QwpAuthFailedException` carries status, host and port only. + - The header is never logged. +- **Watch:** + - `QwpCredentialUnavailableException` uses the *provider's* exception message as its own. That message flows into `SenderError` (`CWSL:1954`) and into logs (`CWSL:1957`, `BD:720`), so provider error text must be token-free. + - `HttpClient.Request.toString()` dumps the raw request, headers and body included (`HttpClient.java:580-586`). An HTTP-based token source must never log or wrap a `Request`: a client-credentials body carries `client_secret`. + - Test-only config snapshots expose the token (`S:4330`, `QQC:972`). + - The upgrade bytes stay in the native send buffer after the handshake until frames overwrite them. This is low risk. +- **Server-side:** server findings that touch security are reported privately (see `SECURITY.md`). + +## 7. Design + +### 7.1 Shape + +``` +TokenSource fetchToken() -> ExpiringToken(token, expiresAt[, refreshAt]) + ▲ Azure TokenCredential adapter (module), future IMDS, anything else +RefreshingTokenProvider shared cache: background refresh, single-flight, jitter, backoff + ▲ implements HttpTokenProvider +existing plumbing Sender / QwpQueryClient / QuestDB pools: call sites unchanged +``` + +Keep `HttpTokenProvider` as the single integration point; it is already in every path in §2. Add an expiry-aware source SPI behind it. The cache belongs in **core**: it is generic (OIDC client credentials, Vault, …), and it is the part every client must mirror. + +### 7.2 API (core, Java 8) + +```java +// io.questdb.client.cutlass.auth (exported) +public final class ExpiringToken { // name avoids clashing with azure-core AccessToken + public ExpiringToken(String token, long expiresAtEpochMillis); // validates printable ASCII + public ExpiringToken(String token, long expiresAtEpochMillis, long refreshAtEpochMillis); // 0 = derive + // toString(): "ExpiringToken{, expiresAt=...}" +} + +@FunctionalInterface +public interface TokenSource { + // Only ever called on the provider's refresher thread, one call at a time. May block. + // Throw TokenUnavailableException to classify; any other RuntimeException counts as retryable. + ExpiringToken fetchToken(); +} + +public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable { + public static Builder builder(TokenSource source); // margins, coldWaitMillis, backoff, clock/scheduler seams + public CharSequence getToken(); // warm: volatile read, no I/O; cold: bounded wait + public void onTokenRejected(CharSequence token, int httpStatus); // forced refresh, rate-limited + public boolean awaitFirstToken(long timeoutMillis); // optional startup readiness gate + public void close(); // stops the refresher thread +} + +// io.questdb.client (exported) +public class TokenUnavailableException extends LineSenderException { + public boolean isRetryable(); // IdP unreachable / timeout / 429 / 5xx -> true + public long getRetryAfterMillis(); // -1 if none +} + +// HttpTokenProvider (existing): one default method added; still a @FunctionalInterface +default void onTokenRejected(CharSequence token, int httpStatus) { } +``` + +Under option (a), the Azure adapter is user code (or a README snippet): + +```java +TokenCredential cred = new DefaultAzureCredentialBuilder().build(); +TokenRequestContext ctx = new TokenRequestContext().addScopes("api:///.default"); +RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> { + com.azure.core.credential.AccessToken t = cred.getTokenSync(ctx); + return new ExpiringToken(t.getToken(), t.getExpiresAt().toInstant().toEpochMilli()); +}).build(); +QuestDB db = QuestDB.builder().fromConfig("wss::addr=qdb1:9000,qdb2:9000;").httpTokenProvider(tokens).build(); +``` + +### 7.3 Connect-string surface + +``` +wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;azure_client_id=; +``` + +- **New keys,** reserved in the spec for all clients and registered as `COMMON` in `ConfigSchema`: + - `token_provider=`; + - `azure_resource`, from which the scope `/.default` is derived; + - optional `azure_client_id`, for a user-assigned managed identity or a workload-identity client ID. +- **Safe to log:** none of these values is secret, so they can live in `QDB_CLIENT_CONF`. +- **Exclusivity:** mutually exclusive with `token`, `username`, `password`, and a programmatic `httpTokenProvider`. +- **Resolution:** + - Core defines a small `TokenProviderFactory` SPI, looked up by name through `ServiceLoader` (with a `uses` clause in `module-info`). + - Core ships no Azure code. `token_provider=azure` without the module fails at parse time: `"token_provider=azure requires questdb-client-azure on the class path"`. +- **Sharing:** + - A process-wide registry keyed by provider name plus canonical parameters hands out one ref-counted `RefreshingTokenProvider`. + - Every `Sender`, `QwpQueryClient` and `QuestDB` built from equivalent strings therefore shares one cache, so a process makes at most one IMDS/Entra call at a time. + - The last `close()` stops the refresher. + +### 7.4 Options + +| | (a) Hook and cache in core | (b) Built-in IMDS in core | (c) `questdb-client-azure` module | +|---|---|---|---| +| Runtime dependencies | none | none | `azure-core` (the user brings `azure-identity`) | +| Azure hosts covered | all, through the user's SDK | VM/VMSS IMDS only. Not App Service/Functions (`IDENTITY_ENDPOINT`), AKS workload identity (federated token exchange), or Arc | all, through `DefaultAzureCredential` | +| Service principal with client credentials | yes | no | yes | +| Connect-string-only deployments (e.g. Kafka connector) | no | yes | yes, through the SPI | +| Maintenance | small | we own an Azure protocol: retry rules for 404/410/429, `expires_on` formats, identity selectors | tracks the `azure-core` API; another artifact to build on JDK 8 and 11 | + +**Recommendation:** + +- Ship (a) and (c) together. +- Add (b) only if a zero-dependency, connect-string-only VM deployment appears. It would plug into the same cache as `token_provider=azure_imds`, with an endpoint override for tests. + +### 7.5 Where it plugs in (changes only) + +| Path | Change | +|---|---| +| All ingress and egress pulls | None to the call sites. A warm cache makes the pull a memory read. `buildWebSocketAuthHeader` returns a small named credential class (dynamic, with `onRejected`) instead of a lambda. | +| `connectWalk` 401 branch (`QWS:3427`) | If the credential is dynamic and the status is 401: call `onTokenRejected`, re-pull, and retry the **same endpoint once** if the token changed. No host-health penalty for the first 401. `AUTH_FAILED` fires only on the final failure. | +| SYNC `connectWithRetry` credential branch (`CWSL:1053`) | A retryable `TokenUnavailableException` becomes transient within the budget. Anything else still fails fast, so OIDC device flow behaviour is unchanged. | +| Foreground reconnect and orphan drainer | Nothing extra. Each attempt re-pulls, and `onTokenRejected` already fired in the walk, so the rotating-401 ride-out now actually receives new tokens. | +| Egress `connect()` / `reconnectViaTracker()` (`QQC:785`, `:1889`) | The same 401 retry-once. | +| Egress pool | Separate fix: do not return a worker whose failover reconnect failed. Red test first (§3). | +| `ConfigSchema`, `fromConfig` paths, `QuestDBBuilder` | Register the keys; resolve them through the registry; add mutual-exclusion checks. | + +Keep "one pull per round" rather than one per endpoint. It is deliberate (`QWS:3310-3320`), it keeps provider failures cluster-wide, and the cache margin plus the 401 retry cover long rounds. + +### 7.6 Caching and refresh + +- **Single-flight.** One daemon refresher thread per provider is the only caller of `fetchToken()`, so each process makes at most one concurrent call per source. This is what keeps it within the IMDS limits of 5 concurrent requests and 20 requests per second. +- **Schedule.** Computed at fetch time: + - `refreshAt = fetchedAt + (expiresAt - fetchedAt) × U(0.45, 0.55)`, clamped to the range `[fetchedAt + 30 s, expiresAt - 5 min]`. + - A source hint wins if it is earlier (newer azure-core and MSAL expose a refresh-at time). + - A service-principal token (60-90 min) is refreshed about every 30-45 min. + - For a managed-identity token (~24 h), IMDS may keep returning the platform-cached token. The schedule then halves the remaining time on each fetch, which is a handful of calls per token lifetime. +- **Hand-out.** + - A token with at least 60 s left is returned from memory: no I/O, no lock. + - With no usable token, `getToken()` asks for a fetch and waits up to `coldWait` (default 30 s). The wait is interruptible, so `close()`'s interrupt still works. If it times out, it throws a retryable `TokenUnavailableException`. + - An expired token is never handed out. +- **Failures.** + - Retry with exponential backoff and full jitter, from 0.5 s up to a 60 s cap, honouring `retryAfterMillis`. + - Keep serving the current token while it is above the hand-out floor. + - Log at most one WARN per minute, without the token. Expose `lastFailure()`. +- **`onTokenRejected`.** + - Applies only if the rejected token is still the current one, and at most once per 30 s. + - It triggers an immediate fetch; the walk waits up to 5 s for a *different* token. + - If the result is the same token (the managed-identity platform cache), it is accepted, and the walk falls back to the phase policy. + - Don't oversell this: for managed identity, proactive refresh with a margin is the real fix. The forced refresh mainly helps client credentials and clock skew. +- **Startup.** The provider starts its first fetch at construction. `awaitFirstToken` lets an application gate `build()` on it. `lazy_connect`/ASYNC never blocks on it. +- **Clocks.** Wall clock for expiry, `nanoTime` for scheduling; validity is re-checked at hand-out. + +### 7.7 Error classification and the 401 policy + +- **Provider failures:** + - **Transient** (retryable): unreachable, timeout, 429, 5xx, IMDS 404/410. + - **Permanent:** identity not found, malformed configuration. + - **Behaviour by phase:** + - OFF fails fast after `coldWait`, which already absorbs short IMDS 429 storms. + - SYNC retries transient failures within the budget and fails fast on permanent ones. + - ASYNC, the established foreground sender and the drainers retry forever, as today. +- **401 with a refreshable credential:** one immediate retry with a *different* token, in every phase. After that, the existing phase policy applies. +- **403:** no forced refresh. 403 is authorization: an app-role or alias change, and for managed identity a new token may not appear for up to 24 h. The existing phase policy applies. +- **Spec impact (needs a cross-client decision):** + 1. Replace "401/403 terminal at any host" with the phase table in §3. Recommendation: adopt Java's policy. A QuestDB 401 can be an IdP outage, and SF promises Invariant B. The other option is to revert Java, which would recreate the brief's problem for every SF user. + 2. Add a "credential sources" section: + - pull per upgrade round; + - reuse within a round; + - proactive refresh, a SHOULD; + - never log tokens, a MUST; + - printable-ASCII validation, a MUST. + 3. Add the 401 rule: a client SHOULD invalidate and retry the same endpoint once if a different credential is obtained. The retry counts toward neither backoff nor host health. + 4. Define the retryable/permanent split for credential-acquisition failures, including the SYNC change. + 5. Reserve `token_provider`, `azure_resource` and `azure_client_id`. + 6. Separately: fix the "all other upgrade errors are transient" text against the code, or fix the code (§3). + +### 7.8 Redaction + +- **Rules for the new code:** + - **`toString()`:** never prints token text. This applies to `ExpiringToken`, the provider and registry entries; they show length, expiry, and at most an 8-hex-digit SHA-256 prefix for correlating rotations. + - **Response bodies:** a source never logs or embeds one, because they contain `access_token`. On a parse failure it reports the field or shape only. + - **Error text from the IdP or SDK:** passed through `DisplaySafe` filtering and capped at about 256 chars. + - **Exceptions:** never carry an HTTP `Request` as message or cause. +- **Optional hardening:** + - Zero the upgrade bytes in the send buffer after sending. It is cheap: under 4 KB. + - Refuse `token_provider=*` on `ws::` (cleartext bearer), and WARN once for a programmatic provider on `ws::`. To decide in §9. + +### 7.9 Language-neutral contract (for Python and Rust) + +- **Source:** returns `(token, expires_at)`, plus an optional `refresh_at`. + - Python: `lambda: cred.get_token(scope)` already returns `AccessToken(token, expires_on)`. + - Rust: `Fn() -> Result<(String, SystemTime), TokenError>`. + - C: a callback with out-parameters. +- **Cache:** the same semantics and defaults in every client, with one cache per source instance and the same registry for connect-string providers. +- **Python:** keeping the cache in the Rust core means the Python callable runs only on the refresher thread, rarely. That keeps GIL hand-offs off the I/O threads. +- **Rust core work:** + - build the header per upgrade attempt instead of `auth_header: String`; + - add the callback and classification; + - decide the reconnect `AuthError` policy (see §7.7, item 1). + +### 7.10 Test plan + +**1. Unit tests for `RefreshingTokenProvider`.** Use a fake clock and scheduler and a scripted source. Cover: +- the schedule: the halving formula, the clamps, jitter bounds, hint precedence, and managed-identity convergence when the source keeps returning the same token; +- single-flight: 64 threads cold-starting at once produce one fetch; +- backoff and `Retry-After`; +- serving the current token until the floor; +- cold failure: a retryable exception is thrown within `coldWait`; +- an interrupt during the wait: the exception is thrown and the interrupt flag is preserved; +- `onTokenRejected`: rate limit, same-token result, and ignoring a stale token; +- that `close()` leaves no thread behind. + +**2. Stub server that rejects expired tokens.** Extend `TestWebSocketServer` with `setAuthorizationValidator(Function)`. Tokens look like `T.`, and the server shares the test's fake clock, returning 401 once a token has expired. + +- **Proactive refresh on reconnect:** advance the clock past T1's expiry and drop the socket. The reconnect must carry T2 with **no** 401 seen. +- **Stalled refresher:** stall the refresher so the reconnect still carries T1. Expect exactly one 401, then an immediate retry with T2 (no backoff gap), and the batch lands. +- **Initial-connect matrix:** OFF, SYNC and ASYNC × a cold provider failure that is retryable or permanent. SYNC must retry the retryable case within the budget. `testThrowingProviderFailsFastInSyncInitialConnect` must stay green. + +**3. Failover and SF replay across a rotation.** Use two validating servers, A and B, with `addr=A,B` and `sf_dir` set. + +- **Failover:** connect to A with T1, rotate, then kill A. B must receive T2, and the unacked frames are replayed exactly once (assert rows and FSNs at B). Add a variant where the refresher is stalled, so there is one 401, then the retry. +- **SF replay:** the server rejects T1 with 401 until the rotation while the producer keeps writing. Every row must arrive, with no `.failed` sentinel, no `DATA_LOSS`, and no terminal. +- **Orphan drainer:** restart with an orphan slot and `drain_orphans=on`. The drainer drains with the rotated token. + +**4. Facade and egress.** +- A `QuestDB` handle with 4 senders and 2 query clients sharing one provider. Force a reconnect storm: `fetchToken` runs once, and every upgrade carries the same fresh token. +- Egress: failover in the middle of a query with a rotated token, and the 401 retry-once on `connect()`. +- A red test for the dead pooled egress worker (§3). + +**5. Connect string.** Cover: +- parse and validation errors; +- mutual exclusion; +- the "module missing" message; +- registry sharing across `fromConfig` instances, and ref-count release; +- rejection on `ws::`, if adopted. + +**6. Redaction.** Use a sentinel token and capture every logger with a logback `ListAppender`. Run the 401, provider-failure and malformed-response scenarios, then assert the sentinel appears nowhere in logs, exception messages, `SenderError` messages or `toString()` output. + +**7. Module (c).** Test against a fake `TokenCredential`, with no network: the expiry/refresh-at mapping, and mapping exceptions to retryable or permanent. Add a manual smoke run on an Azure VM with a managed identity; this is not for CI. + +**8. Fake IMDS (only if we do (b)).** Build it on `MockOidcServer`. Assert `Metadata: true`, `api-version`, `resource` and `client_id`. Script 200, 400, 404, 410, 429 with `Retry-After`, 500, malformed JSON, and both `expires_on` formats. Error messages must never echo the response body. + +**9. General.** Run everything under `assertMemoryLeak`. Tests run on JDK 8; also compile on JDK 25. + +**10. Server e2e (Enterprise, separate PR).** Extend the Enterprise OIDC tests with locally signed tokens shaped like Entra app-only v1 and v2 tokens. Assert: +- v2 with `sub.claim=oid` and `groups.claim=roles` is accepted; +- v1 with `aud=api://…` is rejected under the default audience; +- no roles gives 401; +- an expired token gives 401, and a rotated token then succeeds. + +## 8. Server side (QuestDB Enterprise) + +This section describes server behaviour only. Enterprise source is private, and security-relevant server findings are reported privately (see `SECURITY.md`). + +**Local JWKS validation or UserInfo?** Both exist. The switch is `acl.oidc.groups.encoded.in.token`; its default, `false`, means UserInfo. + +- **UserInfo cannot work for app-only tokens.** Entra's UserInfo is a Microsoft Graph endpoint: it accepts only Graph-audience user tokens. +- **With `true`, the token is validated locally.** The server looks up the token's `kid` in the identity provider's signing keys (JWKS), then: + - verifies the signature; + - checks `aud` against one exact string, `acl.oidc.audience`, which defaults to `acl.oidc.client.id`; + - checks `exp`, with 60 s of leeway; + - requires a non-empty `sub`. +- **Key rollover:** an unknown `kid` makes the server reload its keys. +- **Cache:** verified tokens are cached for 30 s (`acl.oidc.cache.ttl`). + +**Can `acl.oidc.sub.claim` point at `oid` or `azp`?** Yes. + +- Any top-level claim name works. +- `oid` exists in both v1 and v2 tokens; `azp` exists only in v2 (v1 has `appid`). **Use `oid`.** +- The server also requires a **non-empty groups claim**; without one it returns 401. +- Caveat: there is a single `sub.claim` for every login. A deployment that also has human users (UPN) and apps (no UPN) needs `oid`. + +**Does `acl.oidc.groups.claim=roles` work with `EXTERNAL ALIAS`?** Yes. + +- The claim can be an array or a single string. +- Its values are matched against `EXTERNAL ALIAS`. Role values containing dots need quotes: `CREATE GROUP ingest WITH EXTERNAL ALIAS 'QuestDB.Ingest'`. +- The service principal or managed identity must hold at least one app role. +- Authorization failures come back as 403 at the upgrade. Established connections keep their old authorization. + +**How are v1 and v2 tokens handled?** + +- **The practical pitfall is `aud`.** A v1 token's `aud` is the resource string as requested (`api://`); a v2 token's `aud` is the client-ID GUID. +- **Fix:** set `requestedAccessTokenVersion: 2` (manifest `accessTokenAcceptedVersion`) on the QuestDB app registration. Every client (IMDS `resource=` or an SDK `.default` scope) then gets `aud=`, which matches the default. +- **To verify against the tenant:** that the `/discovery/keys` and `/discovery/v2.0/keys` endpoints serve the same signing keys. + +**Working configuration** + +``` +acl.oidc.enabled=true +acl.oidc.configuration.url=https://login.microsoftonline.com//v2.0/.well-known/openid-configuration +acl.oidc.client.id= # also the audience +acl.oidc.groups.encoded.in.token=true # local JWKS validation; required for app-only +acl.oidc.groups.claim=roles +acl.oidc.sub.claim=oid +``` + +**Server follow-ups** + +- **Client-facing:** spec Appendix C proposes two changes: + - send a `WWW-Authenticate: Bearer` challenge on 401; + - return 503, not 401, when the server cannot verify tokens. +- **Also useful:** a fallback list for `sub.claim`, and 403 instead of 401 when the roles claim is missing. +- **Security hardening:** tracked privately. + +**What the client assumes about the server** + +- Auth happens only at the upgrade. True: no re-check mid-stream. +- A token is valid on every node. True if the OIDC configuration is identical across the cluster. +- 401/403 mean credential problems. **Partly false:** JWKS and UserInfo outages also return 401. +- 421 with `X-QuestDB-Role` means a role reject. + +## 9. Decisions + +All resolved. See the spec's §12 (D1–D9). + +## 10. Java implementation plan + +Build in this order. Each step is one PR and must pass the conformance tests listed for it (spec §10). + +**Constraints for every step** (see `CLAUDE.md`): + +- Code must build on Java 8 and also compile on JDK 11+. +- Zero-GC applies to per-row producer calls and the steady-state I/O loop, not to connect paths. +- Tests run under `assertMemoryLeak`. +- `ExportedApiCompatibilityTest` must stay green: add API, never retype it. + +1. **Token cache.** + - Implements spec §4–§5, with the types named in spec Appendix B. + - Self-contained: no connect-path changes. + - Tests C1–C8, plus C20 for the cache. +2. **Client integration.** Implements spec §6 and §8.1–§8.3. Tests C9–C16 and C23. + - Add the default method `HttpTokenProvider.onTokenRejected`. + - `S.buildWebSocketAuthHeader` (`S:3427`) returns a named dynamic-credential class with `onRejected`, instead of a lambda. + - Retry the same endpoint once on 401, in three places, and fire `AUTH_FAILED` only on the final outcome: + - the `QWS.connectWalk` 401 branch (`QWS:3427`); + - `QQC.connect` (`:785`); + - `QQC.reconnectViaTracker` (`:1889`). + - Classify failures at SYNC startup in `CWSL.connectWithRetry` (`:1053`), per D6 and D8. + - Test harness: + - add an authorization validator to `TestWebSocketServer` that picks the response status from the header; + - add an expiring-token scheme driven by a fake clock. +3. **Connect string, registry and Azure module.** Implements spec §7. Tests C17–C19. + - Register the keys in `ConfigSchema`. + - Parse and validate the keys in: + - `S.fromConfigWebSocket` and `validateWsConfig` (`S:4018`, `:4227`); + - `QQC.fromConfig` and `validateConfig` (`:375`); + - the exclusivity check in `QuestDBBuilder.build()` (`:190-194`). + - Add the `uses` clause to `module-info.java`. + - Release the lease when a sender or query client closes. + - Add the new `azure/` reactor module. +4. **Health accessor and deadline.** Implements spec §8.4 and §8.5. Tests C21–C22. + +**Separate tickets, outside this feature:** + +- **Dead pooled egress worker** (§3). Start with a test that reproduces it. +- **Treat a 503 at the upgrade as transient in every phase** (§3, spec Appendix C). This must land before any server change to return 503. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md new file mode 100644 index 000000000..9acd0e197 --- /dev/null +++ b/design/qwp-token-provider-spec.md @@ -0,0 +1,575 @@ +# QWP dynamic bearer credentials: cross-language specification + +**Status:** v0.3, ready for implementation. All decisions are resolved (§12). + +**Changes:** + +- **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). +- **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. + +- **Implementations:** Java is the reference. Other clients implement this document; that includes the Rust core used by Python, Go, .NET and Node. +- **Freezing:** the spec freezes when the Java implementation merges. Any later change needs a version bump and a note to client maintainers. +- **Rationale and Java code locations:** `design/entra-id-qwp-auth.md`. Its §10 is the Java implementation plan. + +**Conventions:** + +- **MUST, SHOULD, MAY** are used as in RFC 2119. +- **[Dn]** marks a rule that follows from a decision recorded in §12. +- Durations are defaults that implementations MAY expose as options. They SHOULD be identical across clients. + +## 1. Scope + +**In scope:** + +- obtaining, caching and refreshing bearer tokens for the QWP WebSocket upgrade, for both ingress (`/write/v4`) and egress (`/read/v1`); +- when clients obtain tokens, and how token-related failures are handled; +- connect-string keys that select a token provider, and the `azure` provider. + +**Out of scope:** + +- the protocol after the upgrade; +- ILP over HTTP and PGWire. A provider defined here MAY also serve them; +- server behaviour. Appendix A describes it and Appendix C proposes changes; both are informative only; +- interactive sign-in. The OIDC device flow is covered by `design/oidc-token-persistence.md`. + +## 2. Terms + +| Term | Meaning | +|---|---| +| Credential | The value of the `Authorization` header on the upgrade. It is either **static** (fixed at configuration: Basic auth, or a `token=` value) or **dynamic** (obtained from a token provider). | +| Token source | A function, supplied by the application or an integration, that obtains a new token from an identity platform (§4). | +| Token provider | The client-side component that connection code asks for the current token. The **refreshing provider** (§5) is a token provider that wraps a token source. | +| Upgrade attempt | One HTTP upgrade request sent to one endpoint. | +| Connect round | One walk over the configured endpoints. It ends at the first successful upgrade, or when every endpoint has failed. | +| Initialization | The phase of a sender before its first successful upgrade. | +| Established | The phase of a sender after its first successful upgrade. | +| Orphan drain | The background replay of a store-and-forward slot left behind by another sender. | +| Egress operation | One query execution, including any failover reconnects it triggers. | + +## 3. Wire format + +This section restates current behaviour; nothing here changes. + +- **Where the credential goes:** it is sent only on the upgrade request, as `Authorization: Bearer `. An established connection is never re-authenticated. +- **Token values:** a token is opaque. + - A client MUST reject an empty or blank token. + - A client MUST reject a token that contains any character outside printable ASCII (0x20–0x7E). + - A client MUST NOT trim or otherwise alter a token. +- **Validating mutable strings:** where the language allows a token to change after it is returned, the client MUST validate a snapshot of it and send that same snapshot. +- **Prefix:** token sources and providers return the token without the `Bearer ` prefix. + +## 4. Token source contract + +`fetch() -> TokenResult | TokenError` + +### TokenResult + +| Field | Required | Meaning | +|---|---|---| +| `token` | yes | The token. | +| `expires_at` | yes | An absolute instant. | +| `refresh_at` | no | The earliest instant the platform suggests refreshing. | + +Rules for `expires_at`: + +- If the source receives a relative lifetime (such as `expires_in`), it SHOULD compute `expires_at` from its own local time of receipt. This makes the cache robust to clock skew. +- If the source cannot determine the expiry, it MUST supply a conservative value. + +### TokenError + +| Field | Meaning | +|---|---| +| `retryable` | Whether retrying can succeed. See classification below. | +| `retry_after` | Optional duration to wait before retrying. | +| `message` | Description of the failure. It MUST NOT contain a token or a raw response body (§9). | + +Classification: + +- **Retryable:** network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. +- **Permanent:** configuration that is missing or wrong. Examples: no credential configured, identity not found, invalid client. +- **When unsure:** treat the failure as retryable. +- An error that carries no classification MUST be treated as retryable. + +### Rules + +- **No concurrency:** a provider never calls `fetch()` concurrently with itself. +- **Where it runs:** implementations SHOULD run `fetch()` on a background context owned by the provider, never on a connection thread. +- **Blocking:** `fetch()` MAY block, but it SHOULD bound its own network operations. 30 s per attempt is RECOMMENDED. +- **Bad results:** the provider MUST treat a result as a retryable error in either of these cases: + - the token fails the checks in §3; + - `expires_at` is not later than the time the result was received. + +## 5. Refreshing provider + +### 5.1 Parameters + +| Name | Default | Meaning | +|---|---|---| +| `refresh_margin` | 5 min | Upper bound on how close to expiry a scheduled refresh may run. | +| `min_refresh_interval` | 30 s | Lower bound on the time between a fetch and the next scheduled refresh. | +| `handout_floor` | 60 s | A token is **usable** only while `now < expires_at - handout_floor`. | +| `cold_wait` | 30 s | The longest a caller waits when no usable token is held. | +| `backoff_initial` | 0.5 s | First retry delay after a failed fetch. | +| `backoff_max` | 60 s | Largest retry delay after failed fetches. | +| `retry_after_max` | 5 min | Cap applied to a source's `retry_after`. | +| `forced_min_interval` | 30 s | Minimum spacing between forced refreshes (§5.5). | +| `forced_wait` | 5 s | The longest `on_rejected` waits for a forced refresh. | +| `registry_linger` | 60 s | How long a connect-string provider stays alive after its last lease is released (§7.4). | + +### 5.2 Refresh schedule + +After a successful fetch received at time `f`, with expiry `e`: + +``` +L = e - f +m = min(refresh_margin, L / 2) +p = min(min_refresh_interval, L / 2) +r = f + L * U(0.45, 0.55) // jittered half-life; U is uniform +r = min(r, e - m) +r = max(r, f + p) +if refresh_at is present: + r = min(r, max(refresh_at, f + p)) +``` + +- **Proactive refresh:** the provider MUST start a fetch at `r` without waiting for a caller. +- **Effect:** while the source is healthy, even an idle client always holds a usable token. + +*Informative examples:* + +- A 60-minute service-principal token is refreshed at about 27–33 minutes. +- A 24-hour managed-identity token is refreshed at about 11–13 hours. +- If the platform keeps returning the same token, each refresh happens at about half of the remaining lifetime. So a token is fetched only a handful of times before it expires. + +### 5.3 Handing out a token: `get_token()` + +1. **Usable token held.** Return it at once, with no I/O and no waiting. If `now >= r` and no fetch is running or scheduled, start one in the background. +2. **No usable token held.** Proceed as follows: + - Start a fetch now, unless one is already running or the provider is in failure backoff (§5.4). + - If the earliest permitted fetch is later than `now + cold_wait`, fail immediately. + - Otherwise wait until one of these happens: a fetch yields a usable token, `cold_wait` elapses, or the caller is cancelled. + - On failure, raise a **token-unavailable** error. It takes the classification of the most recent failed fetch. If no fetch has failed (the wait timed out, or the caller was cancelled), it is retryable. +3. **Never hand out an unusable token.** The provider MUST NOT do so. +4. **Cancellation.** If the calling operation is cancelled (for example, because the client is closing): + - the wait MUST end promptly with a retryable error; + - the cancellation signal MUST be preserved for the caller. In Java, that is the thread's interrupt flag. +5. **Concurrency.** `get_token()` MUST be safe to call concurrently from any number of threads. Any number of concurrent callers without a usable token MUST cause at most one fetch. + +### 5.4 Failures + +**Backoff after the n-th consecutive failed fetch:** + +``` +base = min(backoff_max, backoff_initial * 2^(n-1)) +delay = base / 2 + U(0, base / 2) +if retry_after is present: + delay = max(delay, min(retry_after, retry_after_max)) +``` + +A successful fetch resets `n`. + +**While fetches are failing:** + +- **Keep retrying** while the provider is open, whatever the classification. An operator can fix a "permanent" error without a restart, for example by assigning a role. +- **Keep serving** the current token for as long as it remains usable. +- **Log** each failure as a warning, at most once per minute per provider. The log contains only the classification and the sanitized message. + +### 5.5 Forced refresh: `on_rejected(token, status)` + +Connection code calls this when an upgrade that presented `token` was rejected (§8.2). + +**Ignore the call** in any of these cases: + +- `status` is not 401 **[D2]**; +- `token` is no longer the provider's current token (it has already rotated); +- a forced refresh ran less than `forced_min_interval` ago. + +**Otherwise:** + +- Start a fetch now. Join a fetch that is already running, and respect any failure backoff. +- Wait up to `forced_wait` for the fetch to finish, then return. +- A forced fetch that returns the same token counts as a normal success. + +*Informative:* platforms that cache tokens, such as Azure managed identity, usually return the same token again. A forced refresh mainly helps client-credential flows and clock skew; proactive refresh (§5.2) remains the main defence. + +### 5.6 Lifecycle + +- **Prefetch.** The provider SHOULD start its first fetch as soon as it is created. +- **Readiness.** It SHOULD offer `await_ready(timeout) -> bool`, which returns true once a usable token is held. Applications can use it to gate startup. +- **Close.** + - `close()` stops scheduled fetches and wakes all waiters. + - After `close()`, `get_token()` MUST fail with a permanent error. + - A client MUST NOT close a provider that the application supplied. +- **Clocks.** + - Expiry comparisons use wall-clock time. + - Delays and waits use a monotonic clock. + - Usability is re-evaluated on every call. + +### 5.7 Observability + +The provider SHOULD expose: + +- the current token's expiry; +- the time of the last successful fetch; +- the last failure: its classification, sanitized message, and time. + +None of these MAY reveal the token itself. + +## 6. When the client obtains the token + +- **At the start of every connect round.** A client using a dynamic credential MUST get it from the provider: + - on the initial connect; + - on every reconnect; + - on every failover reconnect; + - on every orphan-drain or recovery connect; + - on every egress connect and egress failover reconnect. +- **Reuse.** The client MAY present that credential on every upgrade attempt within the round. It MUST NOT reuse it in a later round. +- **Provider failure.** If the provider fails, the round MUST end without contacting any endpoint. The failure is classified as **credential-unavailable** (§8). +- **Cancellation.** Getting the credential MUST be cancellable when the client shuts down. +- **Established connections.** Sending data on an established connection MUST NOT query the provider. + +## 7. Connect-string keys + +### 7.1 Keys [D3] + +| Key | Values | Required | Notes | +|---|---|---|---| +| `token_provider` | `azure`. The name `azure_imds` is reserved **[D5]**. | no | Selects a provider. | +| `azure_resource` | `api://` or `` | when `token_provider=azure` | The application ID URI or the client ID of the QuestDB app registration. A trailing `/.default` MUST be stripped. The requested scope is `/.default`. | +| `azure_client_id` | a GUID | no | The client ID of a user-assigned managed identity or of a workload identity. | + +### 7.2 Validation + +A client MUST reject the configuration, with an error that names the offending key, if any of these holds: + +- `token_provider` is combined with `token`, `username`, `password`, or a provider supplied by the application; +- `token_provider` is empty, unknown, or not supported by this client. The error MUST list the supported values. A client MUST NOT silently connect without credentials; +- a provider-specific key is present but the selected provider does not accept it. For example, `azure_resource` without `token_provider=azure`; +- a provider key that is required is missing; +- `token_provider` is used with any schema other than `wss::`: + - plain `ws::` is rejected because the token would cross the network in cleartext **[D4]**; + - non-QWP schemas, such as ILP's `http::`, are out of scope **[D9]**. + +**Ingress and egress.** Both MUST accept these keys, so a single string can configure both. + +**Validating without connecting.** Validation that does not connect, such as a facade checking its configuration at build time, MUST NOT fetch a token. + +### 7.3 Secrets + +- **No secrets in the string.** The keys defined here are non-secret, so a connect string that uses them stays safe to log and to place in `QDB_CLIENT_CONF`. +- **Where secrets live.** Secrets, such as a service-principal secret or certificate, come from the platform's standard configuration. They never come from connect-string keys. + +### 7.4 Sharing + +- **One provider per configuration.** A process MUST NOT run more than one refreshing provider for the same `(token_provider, parameters)`. +- **Registry.** Clients resolve connect-string providers through a registry that is shared by the whole process. Its key is the provider name plus the normalized parameter values. +- **Leases.** + - Every client instance built from a connect string acquires a lease and releases it on close. That covers each sender, each query client and each pooled connection. + - When the last lease is released, the provider is closed after `registry_linger`, unless a new lease is acquired first. +- **Application-supplied providers.** Sharing is the application's responsibility: use one provider instance per identity per process. + +### 7.5 The `azure` provider + +- **How tokens are obtained:** through the language's Azure Identity default credential chain, for the scope `/.default`. Depending on the environment, the chain finds: + - an environment service principal, + - a workload identity, + - a managed identity, + - developer tools. +- **`azure_client_id`:** when set, it selects both the managed identity and the workload identity. +- **Expiry:** `expires_at`, and `refresh_at` when the library exposes it, come from the library's token result. +- **Classification of library errors:** + - **Permanent:** "no credential available in the chain", and authentication failures that Entra reports (such as an invalid client or an application that does not exist). + - **Retryable:** network errors, timeouts, throttling, 5xx responses, and anything else. +- **Unsupported platforms:** a client whose platform has no Azure Identity library MAY leave `azure` unsupported, and must then reject it as described in §7.2. + +*Informative bindings:* + +- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. +- **Python:** `azure-identity` as an optional extra. +- **Rust and C:** not required. + +## 8. Failure handling + +### 8.1 Classes + +| Class | Trigger | +|---|---| +| credential-unavailable (retryable or permanent) | The provider failed while the client was getting the credential (§6). | +| auth-rejected 401 | The upgrade response was `401`. | +| auth-rejected 403 | The upgrade response was `403`. | + +This specification does not change how other upgrade failures are handled: role rejects (`421`), transport errors, version mismatches, and other 4xx and 5xx responses. + +**Exceptions from application-supplied providers [D8].** + +- An exception from an application-supplied provider that is not a token-unavailable error counts as credential-unavailable, **permanent**. Startup therefore fails fast, as it does today for an OIDC device-flow provider that is not signed in. +- Only a token-unavailable error marked retryable is retried at SYNC startup. + +### 8.2 One retry after a 401 + +When an upgrade attempt that presented a dynamic credential receives `401`, the client: + +1. MUST call `on_rejected(presented token, 401)`; +2. MUST get the credential from the provider again; +3. if the new credential differs from the one presented, MUST retry the same endpoint once, immediately and without backoff. That retry: + - MUST NOT count as an endpoint failure for health tracking or backoff; + - is an ordinary upgrade attempt for any outcome other than `401`; +4. if the credential is unchanged, or the retry also receives `401`, applies the phase policy in §8.3. + +Further rules: + +- **Limit:** at most one such retry per connect round. +- **Events:** a client that emits connection events SHOULD report the authentication failure only for the round's final outcome. +- **No forced refresh on 403:** a `403` MUST NOT trigger a forced refresh **[D2]**. +- **Static credentials:** they never get this retry. +- **`WWW-Authenticate` challenge:** if the `401` carries a `WWW-Authenticate: Bearer` challenge with an `error` parameter, the client SHOULD take steps 1–3 only when `error` is `invalid_token` (RFC 6750 §3.1). Without such a challenge, the status code alone decides. Current QuestDB servers send no challenge; Appendix C proposes adding one. + +### 8.3 Policy by phase + +| Phase | credential-unavailable, retryable | credential-unavailable, permanent | 401 / 403 | +|---|---|---|---| +| Initialization, `initial_connect_retry=off` | fail startup | fail startup | fail startup | +| Initialization, `on` / `sync` | retry within `reconnect_max_duration_millis` **[D6]** | fail startup | fail startup | +| Initialization, `async` | retry indefinitely | retry indefinitely | fail, and report through the error handler | +| Established | retry indefinitely | retry indefinitely | retry indefinitely **[D1]** | +| Orphan drain, dynamic credential | retry indefinitely | retry indefinitely | ride out, then quarantine (rule below) | +| Orphan drain, static credential | n/a | n/a | quarantine on the first rejection | +| Egress operation | fail the operation | fail the operation | fail the operation | + +**Retry indefinitely** + +- **Meaning:** retry with capped exponential backoff for as long as the client is open, unless the optional deadline in §8.5 is configured. Store-and-forward MUST NOT drop a live producer. +- **Reporting:** each retried failure MUST be reported to the application's error channel, not only to logs. The report is a retriable, security-category error that names the failure class. The failure SHOULD also be visible in the connection health (§8.4). + +**Ride out, then quarantine** + +The slot is quarantined when either of these holds: + +- at least 6 rejections, **and** the rejections have persisted for at least `min(reconnect_max_duration_millis, 5 min)`; +- 256 rejections within one episode. + +An episode ends only when the drain makes acknowledgement progress. A failure of another class in between restarts the persistence clock but not the count. + +**Egress recovery** + +- After a failed connect or failover reconnect, the client MUST do one of two things: reconnect on its next operation, or be replaced by its pool. It MUST NOT remain permanently unusable. +- The error reported for the failed operation MUST name the failure class. + +### 8.4 Connection health + +Retrying indefinitely is only safe if an outage is easy to see. An application that registers no error handler must still be able to find out that its sender has not reached the server for an hour. Kafka shows the failure mode: when credentials expire, a consumer keeps failing authentication in the background while the application sees nothing (KAFKA-10840, Appendix D). + +Each sender and query client SHOULD expose a snapshot of its connection health, with at least: + +| Field | Meaning | +|---|---| +| `state` | `connecting` (no successful upgrade yet), `connected`, `reconnecting` (the connection was lost and is being re-established), `failed` (terminal) or `closed`. | +| `last_connected_at` | Time of the most recent successful upgrade, or none. | +| `outage_since` | Start of the current period without a connection, or none while connected. | +| `failed_rounds` | Number of failed connect rounds since `outage_since`. | +| `last_failure` | The most recent failed connect round: its class (`credential_unavailable`, `auth_rejected` with the status code, `role_rejected`, `transport` or `other`), a sanitized message, and the time. It is kept after the connection recovers. | + +Rules: + +- **Cheap and safe to poll.** Reading the snapshot MUST NOT wait on I/O or on connection threads, and MUST be safe from any thread, so that it can back a health endpoint. +- **No secrets.** The snapshot MUST NOT contain a credential (§9). +- **Pools and facades** SHOULD expose an aggregate: the number of connections in each `state`, the oldest `outage_since`, and the most recent `last_failure`. +- **Query clients** use the same fields. They connect on demand, so for them `reconnecting` means the last connect or failover reconnect failed and the next operation will try again. + +### 8.5 Optional authentication-outage deadline [D7] + +Some applications prefer a hard failure to an authentication outage that lasts indefinitely, much as Kafka's `delivery.timeout.ms` bounds how long a record may wait. A client MAY offer this. If it does, it MUST use this key and these semantics. + +| Key | Values | Default | +|---|---|---| +| `auth_failure_max_duration_millis` | a positive duration in milliseconds | not set (no deadline) | + +- **Which failures count.** Only *authentication-class* failures: credential-unavailable, and auth-rejected (`401` or `403`). +- **Where it applies.** Ingress senders, in the phases where §8.3 retries an authentication-class failure indefinitely: `async` initialization and established. It does not apply to orphan drains, which have their own rule, or to egress, which already fails the operation. +- **The clock.** It starts at the first authentication-class failure of the current outage. Only a successful upgrade resets it. +- **When it fires.** The sender becomes terminal when a connect round fails with an authentication-class failure and the clock has reached the configured duration. + - Rounds that fail for other reasons neither reset the clock nor fire the deadline. + - So a network outage that follows a single `401` does not fire it. + - And a cluster that alternates between `401` and network errors cannot postpone it forever. +- **What happens then.** + - The error names the failure class and the elapsed time. It is reported as terminal through the error handler and raised from later producer calls. + - Unacknowledged data stays in on-disk store-and-forward storage, for a later sender or an orphan drain. In memory-only mode it is lost when the sender closes. + +## 9. Security + +- **Tokens.** A client MUST NOT write a token into logs, error messages, events, or the string representation of any object. A representation MAY include the token's length, its expiry, and a fingerprint of at most the first 8 hex digits of the token's SHA-256. +- **Response bodies.** A token source MUST NOT log or embed a token-endpoint response body. For a malformed response, it reports only the shape of the problem, such as a missing or invalid field. +- **Error text from libraries and endpoints.** Before such text enters client errors or logs, the client SHOULD: + - remove control, bidirectional and other non-displayable characters; + - truncate it to 256 characters. +- **Raw requests.** Objects that serialize raw HTTP requests MUST NOT be attached to errors or logs, because requests may carry secrets. +- **Cleartext.** A client SHOULD warn once when a provider supplied by the application is used over a non-TLS schema. + +## 10. Conformance tests + +Every client that implements this specification SHOULD pass these scenarios. They use: + +- a fake clock; +- a scripted token source; +- a stub server that answers `401` to an expired token. Tokens can be shaped `T.` and checked against the shared fake clock. + +| ID | Scenario | Expected result | +|---|---|---| +| C1 | Refresh schedule | Refresh times follow §5.2 for `L` = 24 h, 60 min and 5 min, with and without `refresh_at`. | +| C2 | Same token returned | Fetches converge as described in §5.2, and the token stays usable. | +| C3 | Cold burst | 64 concurrent callers with no usable token cause exactly one fetch. | +| C4 | Cold failure | A retryable error is raised within `cold_wait`. A permanent classification is passed through. | +| C5 | Backoff | Delays follow §5.4. `retry_after` is honoured and capped. | +| C6 | Hand-out floor | A token inside `handout_floor` is never handed out. | +| C7 | Cancellation | A cancelled wait returns promptly, and the cancellation signal is preserved. | +| C8 | Forced refresh | Rate limited. A stale token is ignored. A same-token result is accepted. | +| C9 | Reconnect after expiry | The reconnect carries the refreshed token, and no `401` is observed. | +| C10 | Stale token presented | Exactly one `401`, then an immediate retry with a new token: no backoff, no health penalty. | +| C11 | Persistent `401` | Each phase follows §8.3. | +| C12 | Initialization matrix | `off`, `sync` and `async`, each with a retryable and a permanent source failure, behave as in §8.3. | +| C13 | Provider outage while established | Retried indefinitely and reported. Recovers when the source recovers. | +| C14 | Failover across a rotation | The second endpoint receives the new token. Unacknowledged data is replayed exactly once. | +| C15 | Store-and-forward across a rotation | All rows are delivered. No quarantine and no data-loss report. | +| C16 | Orphan drain across a rotation | The slot drains with the new token. | +| C17 | Shared cache | N senders and query clients built from one string use one provider, with one fetch per refresh. | +| C18 | Connect-string validation | Every rule in §7.2 is enforced, and validation does not fetch a token. | +| C19 | Registry lifecycle | Lease release, linger, and re-acquiring a lease during the linger period. | +| C20 | Redaction | A sentinel token never appears in logs, errors, events, string representations or health snapshots, across C4, C10, C11 and a malformed source response. | +| C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | +| C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | +| C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | + +## 11. Relation to existing specifications + +- **QWP ingress and egress specs** (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`): + - their statement that "authentication errors are terminal at any host" is replaced by §8 **[D1]**; + - their authentication sections should point to §6. +- **Connect-string reference:** it should reserve the keys in §7.1, and `auth_failure_max_duration_millis` (§8.5). +- **OIDC device-flow token store** (`design/oidc-token-persistence.md`): unaffected. An `OidcDeviceAuth` instance remains a valid provider supplied by the application. + +## 12. Decisions + +All decisions below are resolved. Changing one after the Java implementation merges needs a version bump. + +| ID | Question | Decision | +|---|---|---| +| D1 | How to treat 401/403 once a sender is established | Retry indefinitely, report it (§8.3), and expose it in the connection health (§8.4). This is Java's current behaviour and matches common practice (Appendix D). The Rust core must change to match. | +| D2 | Whether a 403 also triggers a forced refresh | No. Only a 401 does, as in RFC 6750 and the clients in Appendix D. | +| D3 | Key names | `token_provider`, `azure_resource`, `azure_client_id`. (`client_id` is already used by egress.) | +| D4 | Whether to reject `token_provider` on `ws::` | Yes. | +| D5 | A zero-dependency `azure_imds` provider | The name is reserved; it is implemented only on demand. | +| D6 | Whether SYNC initialization retries retryable provider failures | Yes. | +| D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | +| D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | +| D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | + +**Still to verify in a real Entra tenant.** These checks don't block implementation, but they should be done before the spec freezes: + +- Managed-identity tokens from IMDS carry the client-ID GUID as `aud` once the app registration is set to v2 tokens. +- The v1 and v2 signing-key endpoints serve the same keys. +- Whether IMDS returns the same token when asked again. This affects only the convergence note in §5.2. +- How long `DefaultAzureCredential` takes on a cold start. This sizes `cold_wait`. + +## Appendix A. QuestDB Enterprise and Entra configuration (informative) + +Describes QuestDB Enterprise behaviour at the time of writing. + +**Server properties:** + +``` +acl.oidc.enabled=true +acl.oidc.configuration.url=https://login.microsoftonline.com//v2.0/.well-known/openid-configuration +acl.oidc.client.id= # also the expected audience +acl.oidc.groups.encoded.in.token=true # local JWKS validation; required for app-only tokens +acl.oidc.groups.claim=roles +acl.oidc.sub.claim=oid +``` + +**Entra setup:** + +- On the QuestDB app registration: + - expose an application ID URI such as `api://`; + - define app roles whose allowed member type is Applications; + - set `requestedAccessTokenVersion` to 2 (`accessTokenAcceptedVersion` in the manifest). Every token then carries the app's client ID as `aud`. +- Assign at least one app role to each managed identity or service principal. + +**QuestDB setup:** + +```sql +CREATE GROUP ingest WITH EXTERNAL ALIAS 'QuestDB.Ingest'; +GRANT HTTP TO ingest; +GRANT INSERT ON trades TO ingest; +``` + +**Server behaviours clients should expect:** + +- **401 does not always mean a bad token.** `401` is also returned when the server cannot fetch its signing keys (JWKS). +- **One audience.** `aud` must match a single exact string. v1 tokens carry the requested resource string as `aud`; v2 tokens carry the client ID. +- **Expiry.** `exp` is enforced with 60 s of leeway. +- **Roles are required.** A token without the roles claim is rejected with `401`. +- **Cache.** A verified token is cached for 30 s. +- **Role changes apply only to new connections,** and only once a new token carries them. Managed-identity tokens can live for about 24 h. + +## Appendix B. Java binding (decided) + +New types live in `io.questdb.client.cutlass.auth` unless noted. + +| Spec concept | Java | +|---|---| +| TokenResult | `ExpiringToken(String token, long expiresAtEpochMillis)`, plus an overload that adds `long refreshAtEpochMillis` (0 means none) | +| Token source | `TokenSource`, a functional interface: `ExpiringToken fetchToken()` | +| Refreshing provider | `RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable`, created with `RefreshingTokenProvider.builder(TokenSource)`. Methods: `getToken()`, `onTokenRejected(CharSequence, int)`, `awaitReady(long)`, `close()`. The builder exposes the §5.1 parameters, plus clock and scheduler seams for tests. | +| token-unavailable error | `TokenUnavailableException extends LineSenderException`, with `isRetryable()` and `getRetryAfterMillis()` (-1 means none) | +| `on_rejected` | `onTokenRejected(CharSequence token, int httpStatus)`, a default no-op method on the existing `io.questdb.client.HttpTokenProvider` | +| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`) | +| Registry | `TokenProviderRegistry`: process-wide, hands out ref-counted leases | +| `azure` provider | The new reactor module `azure/`, artifact `io.questdb:questdb-client-azure`, package `io.questdb.client.azure`, class `AzureTokenProviderFactory`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | +| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as an aggregate by `QuestDB.health()`. | +| Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | + +**Python notes:** + +- A token source maps to a callable that returns `azure.core.credentials.AccessToken`, which carries `token` and `expires_on` (in epoch seconds). +- Keep the provider in the Rust core, so that the callable runs only on the provider's refresh thread. +- A provider MUST NOT be used across `fork()`. + +## Appendix C. Recommended server changes (informative) + +These changes would let clients refresh only when a new token can help, and treat server-side outages as transient. + +| Situation | Today | Recommended | +|---|---|---| +| The token is invalid: bad signature, expired, wrong audience, or malformed | `401`, no challenge | `401` with `WWW-Authenticate: Bearer error="invalid_token"`. An `error_description` is fine, but never the token. | +| The token is valid but grants nothing: no roles claim, or no mapped group | `401` without a roles claim; `403` without a mapped group | `403` with `WWW-Authenticate: Bearer error="insufficient_scope"` | +| The server cannot verify tokens: it cannot fetch its signing keys (JWKS) or reach UserInfo | `401` | `503`, optionally with `Retry-After` | + +**Sequencing: land the client fix first.** + +- Java currently treats any non-421 rejection of the upgrade as terminal during initialization. +- An orphan drain quarantines its slot on such a rejection immediately (`design/entra-id-qwp-auth.md` §3). +- If the server switched to `503` first, a signing-key outage would therefore quarantine orphan slots that today ride out a `401`. +- The client should treat a `503` at the upgrade as transient in every phase, as the public QWP spec already says. + +Other server-side follow-ups, including security hardening, are tracked privately with the QuestDB Enterprise team (see `SECURITY.md`). + +## Appendix D. Precedents for D1 and D2 (informative) + +| Source | Behaviour | Bearing on this spec | +|---|---|---| +| RFC 6750 §3.1 | `invalid_token` goes with `401`, and "the client MAY request a new access token and retry". `insufficient_scope` goes with `403`. | Refresh on `401` only (D2); the challenge rule in §8.2. | +| MongoDB driver auth spec (MONGODB-OIDC) | On an authentication failure with a cached token: clear that token from the cache (atomically, and only if it is unchanged), fetch a new one, and retry once. Other errors go to the user. At least 100 ms between calls to the token callback. | The closest match to §5.5 and §8.2. | +| Google auth library, `HttpCredentialsAdapter` | Refreshes and retries on a `401`, or on a Bearer `invalid_token` challenge. A plain `403` does not trigger it. | D2. | +| Kubernetes client-go, exec credential plugin | On a `401`, refreshes "unless they were rotated already", then returns the `401`. The next request uses the new credentials. | The stale-token guard in §5.5. | +| Azure SDK, `BearerTokenAuthenticationPolicy` | Retries only on a `401` that carries a Continuous Access Evaluation claims challenge. A plain `401` goes back to the caller. | A stricter variant of D2. | +| Kafka: KIP-152 and KAFKA-6516 | Authentication failures are non-retriable at the API. The client is not closed, though, and keeps reconnecting in the background; a request to stop that was closed Won't Fix. | D1. | +| Kafka: KAFKA-10840 (open) | When credentials expire, a consumer keeps failing authentication in the background and the application cannot see it. A proposed fix notes that Kafka Connect tasks report RUNNING meanwhile. | Why §8.4 exists. | + +Links: + +- RFC 6750: https://www.rfc-editor.org/rfc/rfc6750#section-3.1 +- MongoDB auth spec: https://github.com/mongodb/specifications/blob/master/source/auth/auth.md +- Google `HttpCredentialsAdapter`: https://github.com/googleapis/google-auth-library-java/blob/main/oauth2_http/java/com/google/auth/http/HttpCredentialsAdapter.java +- client-go exec plugin: https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/exec/exec.go +- Azure `BearerTokenAuthenticationPolicy`: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/core/azure-core/src/main/java/com/azure/core/http/policy/BearerTokenAuthenticationPolicy.java +- KIP-152: https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures +- KAFKA-6516: https://issues.apache.org/jira/browse/KAFKA-6516 +- KAFKA-10840: https://issues.apache.org/jira/browse/KAFKA-10840 (proposed fix: https://github.com/apache/kafka/pull/16418) From c9f2b96a45cfc41978ba5701d51ad58bf1cccdc9 Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Fri, 2 Oct 2026 11:03:20 +0100 Subject: [PATCH 2/3] feat(qwp): refreshing bearer-token providers and Microsoft Entra ID auth Implement the dynamic-credential specification (design/qwp-token-provider-spec.md, v0.3) and the four-step Java plan in design/entra-id-qwp-auth.md, section 10. - Token cache: RefreshingTokenProvider, ExpiringToken, TokenSource and TokenUnavailableException. Proactive jittered refresh at about half the token lifetime, a 60 s hand-out floor, single-flight cold waits bounded by cold_wait, jittered backoff honouring Retry-After, rate-limited forced refresh, and token redaction everywhere. - Client integration: one immediate same-endpoint retry after a refreshable 401 (WWW-Authenticate aware; never for a 403 or a static credential) on every ingest connect path, orphan drains included, and on egress connect and failover. SYNC startup retries a retryable provider failure within its budget (D6); any other provider exception still fails fast (D8). Egress errors name the failure class, and a query client whose failover reconnect failed reconnects on its next execute() instead of staying unusable (the suspected dead pooled worker, reproduced by a test before the fix). - Connect string: token_provider, azure_resource and azure_client_id on both clients (wss:: only; exclusive with static credentials and application-supplied providers), resolved through a ServiceLoader SPI and a process-wide ref-counted registry with a 60 s linger. Validation never fetches a token. - New optional module azure/ (org.questdb:questdb-client-azure): token_provider=azure on DefaultAzureCredential with the spec's error classification. Java 8 floor, released together with the client. - Connection health: Sender.health(), QwpQueryClient.health() and an aggregate QuestDB.health(); an optional auth_failure_max_duration_millis deadline for authentication outages. - Tests for conformance scenarios C1-C23, plus a TLS mode for the test WebSocket server (its key is generated at test time). README and design docs updated; the spec's Appendix B now names the published artifact org.questdb:questdb-client-azure. --- .gitignore | 1 + README.md | 99 ++ azure/pom.xml | 307 +++++ .../azure/AzureTokenProviderFactory.java | 91 ++ .../client/azure/AzureTokenSource.java | 230 ++++ ...b.client.cutlass.auth.TokenProviderFactory | 1 + .../azure/test/AzureTokenSourceTest.java | 292 +++++ .../io/questdb/client/ConnectionHealth.java | 290 +++++ .../io/questdb/client/HttpTokenProvider.java | 31 + .../main/java/io/questdb/client/QuestDB.java | 12 + .../io/questdb/client/QuestDBBuilder.java | 4 + .../main/java/io/questdb/client/Sender.java | 837 ++++++++------ .../cutlass/auth/CredentialRedaction.java | 145 +++ .../client/cutlass/auth/ExpiringToken.java | 148 +++ .../cutlass/auth/RefreshingTokenProvider.java | 1027 +++++++++++++++++ .../cutlass/auth/TokenProviderFactory.java | 83 ++ .../cutlass/auth/TokenProviderRegistry.java | 302 +++++ .../cutlass/auth/TokenProviderSpec.java | 248 ++++ .../client/cutlass/auth/TokenSource.java | 63 + .../auth/TokenUnavailableException.java | 128 ++ .../client/cutlass/http/BearerChallenge.java | 158 +++ .../cutlass/http/client/WebSocketClient.java | 45 + .../qwp/client/QwpAuthFailedException.java | 58 +- .../client/QwpConnectionHealthTracker.java | 182 +++ .../QwpCredentialUnavailableException.java | 41 +- .../cutlass/qwp/client/QwpQueryClient.java | 212 +++- .../qwp/client/QwpUpgradeFailures.java | 3 +- .../qwp/client/QwpWebSocketSender.java | 206 +++- .../sf/cursor/CursorWebSocketSendLoop.java | 140 ++- .../io/questdb/client/impl/ConfigSchema.java | 9 + .../io/questdb/client/impl/PooledSender.java | 6 + .../questdb/client/impl/QueryClientPool.java | 16 + .../io/questdb/client/impl/QuestDBImpl.java | 9 + .../io/questdb/client/impl/SenderPool.java | 20 + core/src/main/java/module-info.java | 4 + .../auth/RefreshingTokenProviderTest.java | 841 ++++++++++++++ .../auth/TestTokenProviderFactory.java | 100 ++ .../cutlass/auth/TokenProviderConfigTest.java | 255 ++++ .../test/cutlass/auth/TokenTestKit.java | 269 +++++ .../cutlass/http/BearerChallengeTest.java | 94 ++ .../qwp/client/ConnectionHealthTest.java | 431 +++++++ .../QwpQueryClientDynamicCredentialTest.java | 349 ++++++ ...QwpWebSocketSenderJvmErrorCleanupTest.java | 5 +- .../qwp/client/TokenProviderSharingTest.java | 161 +++ .../WebSocketDynamicCredentialTest.java | 968 ++++++++++++++++ .../test/cutlass/qwp/websocket/TestTls.java | 112 ++ .../qwp/websocket/TestWebSocketServer.java | 109 +- .../impl/QwpQueryClientConfigHonoredTest.java | 14 + .../test/impl/WsSenderConfigHonoredTest.java | 15 + design/entra-id-qwp-auth.md | 52 +- design/qwp-token-provider-spec.md | 8 +- pom.xml | 1 + 52 files changed, 8799 insertions(+), 433 deletions(-) create mode 100644 azure/pom.xml create mode 100644 azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java create mode 100644 azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java create mode 100644 azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java create mode 100644 core/src/main/java/io/questdb/client/ConnectionHealth.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java diff --git a/.gitignore b/.gitignore index 872223992..edcddfd0b 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ core/target utils/target utils/dependency-reduced-pom.xml examples/target +azure/target compat/target core/node deploy*.bat diff --git a/README.md b/README.md index e945c7770..ebdd2e145 100644 --- a/README.md +++ b/README.md @@ -485,6 +485,101 @@ The token is stored as **plaintext JSON protected by file permissions** — `060 `FileTokenStore` is safe to share between processes that sign in as the same identity: each update is written atomically (so a concurrent reader never sees a half-written credential), and when the identity provider rotates the refresh token on each refresh, the read-refresh-write is serialized across processes with a lock file so they do not race each other into an unnecessary re-prompt. The lock file's staleness is judged by its modification time, so this coordination assumes the processes share a clock — a single machine, or machines with synchronized clocks; under significant clock skew (for example a store directory on NFS shared across hosts) a live lock can be mis-judged stale or a dead one never expire. `clearCache()` removes the persisted entry under the same lock, but across processes it is best-effort: a peer that still holds a live in-memory token may legitimately re-persist afterwards (it always forces a fresh sign-in for the calling process). +### Rotating Bearer Tokens (Microsoft Entra ID and Other Identity Platforms) + +Service identities - an Azure managed identity, a service principal, an AKS workload identity - authenticate with +bearer tokens that expire every hour or every day. Select a **token provider** and the client fetches tokens itself, +keeps them in a shared cache, and refreshes them in the background well before they expire, so long-lived senders and +query clients keep reconnecting with a valid token. + +**Microsoft Entra ID from a connect string.** Add the optional `questdb-client-azure` artifact (same version as +`questdb-client`); it brings Azure Identity and registers `token_provider=azure`: + +```xml + + org.questdb + questdb-client-azure + 1.0.0 + +``` + +```java +try (QuestDB db = QuestDB.connect( + "wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;")) { + // ... use db ... +} +``` + +- `azure_resource` (required) is the application ID URI (`api://`) or the client ID of the QuestDB app + registration; the client requests the scope `/.default`. +- `azure_client_id` (optional) selects a user-assigned managed identity or a workload identity. +- The credential comes from Azure Identity's `DefaultAzureCredential`: an environment service principal + (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` or a certificate), a workload identity, a managed + identity, or developer tools. No secret ever goes into the connect string, so it stays safe to log and to put in + `QDB_CLIENT_CONF`. +- `token_provider` requires `wss::`, and cannot be combined with `token`, `username`/`password` or an + application-supplied provider. +- Every sender, query client and pooled connection built from equivalent connect strings shares one provider per + process: one refresher thread, one call to Entra at a time. + +On the server, validate tokens locally (`acl.oidc.groups.encoded.in.token=true`, `acl.oidc.groups.claim=roles`, +`acl.oidc.sub.claim=oid`), set the QuestDB app registration to issue v2 tokens, and map its app roles to groups with +`CREATE GROUP ... WITH EXTERNAL ALIAS ''`. + +**Any other identity platform.** Wrap a `TokenSource` in a `RefreshingTokenProvider` and pass the provider to +`QuestDB.connect(config, provider)`, `httpTokenProvider(...)` or `withBearerTokenProvider(...)`. The application owns +the provider: close it after the clients that use it. + +```java +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; + +RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> { + MyToken t = myIdentityPlatform.requestToken(); // runs on the provider's own thread + return new ExpiringToken(t.value(), t.expiresAtEpochMillis()); +}).build(); +try (QuestDB db = QuestDB.connect("wss::addr=qdb1:9000;", tokens)) { + // ... use db ... +} finally { + tokens.close(); +} +``` + +A source reports a failure by throwing `TokenUnavailableException.retryable(...)` (network trouble, throttling, +5xx) or `TokenUnavailableException.permanent(...)` (missing or wrong configuration); anything else counts as +retryable. Never put a token or a raw response body in the message. The provider keeps retrying with backoff and +keeps serving the current token while it is still valid. + +How failures are handled: + +- A token is refreshed at about half its lifetime, so an idle client always holds a valid one. +- When the server rejects a token with `401`, the client asks the provider for a new one and retries the same server + once, immediately. +- At startup, `initial_connect_retry=off` (the default) fails if no token can be obtained; `on` keeps retrying a + retryable provider failure within `reconnect_max_duration_millis`; `async` retries in the background. +- Once connected, a store-and-forward sender rides out credential outages and `401`/`403` indefinitely, buffering + rows and reporting each failure to the error handler. Set `auth_failure_max_duration_millis` to make the sender + fail instead once such an outage lasts that long (unacknowledged rows stay on disk). + +### Connection Health + +A sender that rides out an outage keeps accepting rows, so check its connection health to see that it is not +reaching the server. Reading health never blocks and never contains a credential, so it can back a health endpoint. + +```java +ConnectionHealth.Aggregate health = db.health(); // every pooled sender and query client +if (health.count(ConnectionHealth.State.RECONNECTING) > 0) { + long since = health.getOldestOutageSinceEpochMillis(); + ConnectionHealth.Failure failure = health.getLastFailure(); // e.g. AUTH_REJECTED 401, CREDENTIAL_UNAVAILABLE + // ... +} +``` + +`Sender.health()` and `QwpQueryClient.health()` return the same information for a single client: its state +(`CONNECTING`, `CONNECTED`, `RECONNECTING`, `FAILED`, `CLOSED`), the last successful connect, the start of the current +outage, the number of failed connect rounds since, and the last failure with its class. + ### Explicit Timestamps ```java @@ -529,6 +624,10 @@ schema::key1=value1;key2=value2; | `tls_roots_password` | | Optional JKS/PKCS#12 password; omit when `tls_roots` is PEM | | `connect_timeout` | _(OS)_ | TCP connect + TLS handshake timeout, in milliseconds | | `auth_timeout_ms` | `15000` | Authentication/upgrade request timeout, in milliseconds | +| `token_provider` | | Refreshing bearer-token provider, `wss` only: `azure` (needs `questdb-client-azure`) | +| `azure_resource` | | `token_provider=azure`: application ID URI or client ID of the QuestDB app registration | +| `azure_client_id` | | `token_provider=azure`: client ID of a user-assigned managed identity or workload identity | +| `auth_failure_max_duration_millis` | _(none)_ | Ingest: fail the sender once a credential outage or `401`/`403` lasts this long | ### Pool keys (facade only) diff --git a/azure/pom.xml b/azure/pom.xml new file mode 100644 index 000000000..754ad21cb --- /dev/null +++ b/azure/pom.xml @@ -0,0 +1,307 @@ + + + + + 4.0.0 + + org.questdb + questdb-client-azure + 1.3.10-SNAPSHOT + jar + QuestDB client - Microsoft Entra ID token provider + token_provider=azure for the QuestDB Java client: Microsoft Entra ID bearer tokens through Azure Identity's DefaultAzureCredential + https://questdb.io/ + + + + Apache 2.0 + https://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + QuestDB Team + hello@questdb.io + + + + + https://github.com/questdb/java-questdb-client + scm:git:https://github.com/questdb/java-questdb-client.git + scm:git:https://github.com/questdb/java-questdb-client.git + HEAD + + + + UTF-8 + + 1.3.8 + 1.5.25 + + + + + + com.azure + azure-sdk-bom + ${azure-sdk-bom.version} + pom + import + + + + + + + org.questdb + questdb-client + ${project.version} + + + com.azure + azure-identity + + + + + junit + junit + 4.13.2 + test + + + ch.qos.logback + logback-classic + ${logback.version} + test + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.11.0 + + ${javac.compile.source} + ${javac.compile.target} + + + + org.apache.maven.plugins + maven-surefire-plugin + 3.5.3 + + + false + + + + org.apache.maven.plugins + maven-jar-plugin + 3.0.1 + + + + + io.questdb.client.azure + ${project.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + 2.8.2 + + true + + + + + + + + java11+ + + [11,) + + + + 11 + 11 + + + + java8 + + 1.8 + + + 1.8 + 1.8 + + 1.3.15 + + + + javadoc + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.5.0 + + + attach-javadocs + + jar + + + + + none + ${javac.compile.source} + false + + + + + + + release-artifacts + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.5.0 + + + attach-javadocs + + jar + + + + + none + ${javac.compile.source} + false + + + + org.apache.maven.plugins + maven-source-plugin + 3.0.1 + + + attach-sources + + jar + + + + + + org.apache.maven.plugins + maven-gpg-plugin + 3.2.7 + + + sign-artifacts + verify + + sign + + + + + gpg + + --pinentry-mode + loopback + + + + + + + + maven-central-publish + + + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.0.0-M3 + + + enforce-publish-from-jdk8 + + enforce + + + + + [1.8,1.9) + questdb-client-azure is published with questdb-client, from JDK 8. + + + + + + + + org.sonatype.central + central-publishing-maven-plugin + 0.9.0 + true + + central + false + validated + + + + + + + diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java new file mode 100644 index 000000000..950f1bbcb --- /dev/null +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java @@ -0,0 +1,91 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure; + +import com.azure.identity.DefaultAzureCredentialBuilder; +import io.questdb.client.cutlass.auth.TokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; + +import java.util.Map; + +/** + * The {@code azure} token provider (design/qwp-token-provider-spec.md, section 7.5): Microsoft Entra ID tokens + * from Azure Identity's {@code DefaultAzureCredential}. Discovered through {@link java.util.ServiceLoader}; put + * this artifact on the class path or module path and connect with + * 
+ * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;
+ *
+ *
    + *
  • {@code azure_resource} (required) - the application ID URI or client ID of the QuestDB app + * registration; the requested scope is {@code /.default}.
    • + *
  • {@code azure_client_id} (optional) - the client ID of a user-assigned managed identity or of a workload + * identity; it selects both.
    • + *
+ * Depending on the environment, the credential chain finds an environment service principal + * ({@code AZURE_CLIENT_ID}, {@code AZURE_TENANT_ID}, {@code AZURE_CLIENT_SECRET} or a certificate), a workload + * identity, a managed identity, or developer tools. Secrets come from that standard configuration, never from + * the connect string. + */ +public final class AzureTokenProviderFactory implements TokenProviderFactory { + + @Override + public TokenSource createSource(Map params) { + final String resource = params.get(TokenProviderSpec.KEY_AZURE_RESOURCE); + if (resource == null) { + throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); + } + final DefaultAzureCredentialBuilder builder = new DefaultAzureCredentialBuilder(); + final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); + if (clientId != null) { + builder.managedIdentityClientId(clientId); + builder.workloadIdentityClientId(clientId); + } + return new AzureTokenSource(builder.build(), resource); + } + + @Override + public String describe(Map params) { + final StringBuilder sb = new StringBuilder(TokenProviderSpec.AZURE) + .append("[resource=").append(params.get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); + if (clientId != null) { + sb.append(", client_id=").append(clientId); + } + return sb.append(']').toString(); + } + + @Override + public String name() { + return TokenProviderSpec.AZURE; + } + + @Override + public void validate(Map params) { + if (params.get(TokenProviderSpec.KEY_AZURE_RESOURCE) == null) { + throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); + } + } +} diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java new file mode 100644 index 000000000..b0c6e793d --- /dev/null +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -0,0 +1,230 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure; + +import com.azure.core.credential.AccessToken; +import com.azure.core.credential.TokenCredential; +import com.azure.core.credential.TokenRequestContext; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpResponse; +import com.azure.identity.CredentialUnavailableException; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; + +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.concurrent.TimeUnit; + +/** + * A {@link TokenSource} that obtains Microsoft Entra ID access tokens from an Azure Identity + * {@link TokenCredential} - by default a {@code DefaultAzureCredential} - for the scope + * {@code /.default} (design/qwp-token-provider-spec.md, section 7.5). + *

+ * Use it directly to put an Azure credential of your choice behind a + * {@link io.questdb.client.cutlass.auth.RefreshingTokenProvider}: + * 

{@code
+ * TokenCredential credential = new ClientCertificateCredentialBuilder()...build();
+ * RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(
+ *         new AzureTokenSource(credential, "api://")).build();
+ * }
+ * or let a connect string select {@code token_provider=azure}, which uses {@code DefaultAzureCredential}. + *

+ * Expiry and the refresh hint come from the library's {@link AccessToken}. Failures are classified as section + * 7.5 prescribes: + *

    + *
  • permanent - "no credential available in the chain" ({@link CredentialUnavailableException}), and + * authentication failures Entra reports: an HTTP 400 or 401 from the token endpoint, or a known AADSTS + * configuration error such as an invalid client or an application that does not exist;
    • + *
  • retryable - network errors, timeouts, throttling, 5xx responses, and anything else. A + * {@code Retry-After} in seconds is passed on.
    • + *
+ * Each attempt is bounded (30 s by default). Library exceptions never travel as a cause - their response + * objects reference the raw HTTP request, which can carry a client secret - only their sanitized message does. + */ +public final class AzureTokenSource implements TokenSource { + /** + * The bound on one token request. + */ + public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30); + private static final String DEFAULT_SCOPE_SUFFIX = "/.default"; + private static final int MAX_CAUSE_DEPTH = 16; + // Entra (AADSTS) errors that a retry cannot fix: the configuration is wrong, an operator must act. + private static final String[] PERMANENT_AADSTS = { + "AADSTS700016", // application not found in the directory + "AADSTS7000215", // invalid client secret + "AADSTS7000222", // client secret expired + "AADSTS700027", // client assertion signature invalid + "AADSTS70011", // invalid scope + "AADSTS500011", // resource principal not found: azure_resource is wrong + "AADSTS90002", // tenant not found + "AADSTS700213", // no matching federated identity record + "AADSTS70021", // no matching federated identity record + "AADSTS7000112", // application disabled + "AADSTS50049", // unknown or invalid instance + }; + private final TokenRequestContext context; + private final TokenCredential credential; + private final String scope; + private final Duration timeout; + + /** + * @param credential the Azure Identity credential + * @param resource the application ID URI ({@code api://}) or client ID of the QuestDB app + * registration; a trailing {@code /.default} is accepted + */ + public AzureTokenSource(TokenCredential credential, String resource) { + this(credential, resource, DEFAULT_TIMEOUT); + } + + /** + * @param credential the Azure Identity credential + * @param resource the application ID URI ({@code api://}) or client ID of the QuestDB app + * registration; a trailing {@code /.default} is accepted + * @param timeout the bound on one token request + */ + public AzureTokenSource(TokenCredential credential, String resource, Duration timeout) { + if (credential == null) { + throw new IllegalArgumentException("credential must not be null"); + } + if (resource == null || resource.isEmpty()) { + throw new IllegalArgumentException("resource must not be empty"); + } + if (timeout == null || timeout.isNegative() || timeout.isZero()) { + throw new IllegalArgumentException("timeout must be positive"); + } + this.credential = credential; + this.scope = resource.endsWith(DEFAULT_SCOPE_SUFFIX) ? resource : resource + DEFAULT_SCOPE_SUFFIX; + this.context = new TokenRequestContext().addScopes(scope); + this.timeout = timeout; + } + + /** + * Classifies a failure from Azure Identity per section 7.5. Never attaches the library exception. + */ + static TokenUnavailableException classify(Throwable failure, String scope) { + final String description = describe(failure); + Throwable t = failure; + for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof CredentialUnavailableException) { + return TokenUnavailableException.permanent( + "no credential available in the Azure Identity chain for " + scope + ": " + description); + } + if (t instanceof HttpResponseException) { + final HttpResponse response = ((HttpResponseException) t).getResponse(); + if (response != null) { + final int status = response.getStatusCode(); + if (status == 400 || status == 401) { + return TokenUnavailableException.permanent( + "Entra rejected the token request for " + scope + " with HTTP " + status + ": " + description); + } + return TokenUnavailableException.retryable( + "the token request for " + scope + " failed with HTTP " + status + ": " + description, + retryAfterMillis(response)); + } + } + final String message = t.getMessage(); + if (message != null) { + for (String code : PERMANENT_AADSTS) { + if (message.contains(code)) { + return TokenUnavailableException.permanent( + "Entra rejected the token request for " + scope + " (" + code + "): " + description); + } + } + } + } + return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + description); + } + + private static String describe(Throwable t) { + final String message = t.getMessage(); + return message == null ? t.getClass().getName() : t.getClass().getSimpleName() + ": " + message; + } + + private static long retryAfterMillis(HttpResponse response) { + final String value; + try { + value = response.getHeaderValue("Retry-After"); + } catch (RuntimeException e) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + if (value == null) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + try { + // delta-seconds only; an HTTP-date is ignored and the provider's own backoff applies + return TimeUnit.SECONDS.toMillis(Long.parseLong(value.trim())); + } catch (NumberFormatException e) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + } + + private static long refreshAtMillis(AccessToken token) { + try { + final OffsetDateTime refreshAt = token.getRefreshAt(); + return refreshAt == null ? ExpiringToken.NO_REFRESH_AT : refreshAt.toInstant().toEpochMilli(); + } catch (NoSuchMethodError e) { + // an azure-core older than the refresh hint + return ExpiringToken.NO_REFRESH_AT; + } + } + + @Override + public ExpiringToken fetchToken() { + final AccessToken token; + try { + token = credential.getToken(context).block(timeout); + } catch (RuntimeException e) { + throw classify(e, scope); + } + if (token == null) { + throw TokenUnavailableException.retryable("Azure Identity returned no token for " + scope); + } + final OffsetDateTime expiresAt = token.getExpiresAt(); + if (token.getToken() == null || expiresAt == null) { + // the shape of the problem only, never the response + throw TokenUnavailableException.retryable("Azure Identity returned a token for " + scope + + " without " + (token.getToken() == null ? "a value" : "an expiry")); + } + try { + return new ExpiringToken(token.getToken(), expiresAt.toInstant().toEpochMilli(), refreshAtMillis(token)); + } catch (IllegalArgumentException e) { + throw TokenUnavailableException.retryable("Azure Identity returned an unusable token for " + scope + + ": " + e.getMessage()); + } + } + + /** + * @return the requested scope, {@code /.default} + */ + public String getScope() { + return scope; + } + + @Override + public String toString() { + return "AzureTokenSource{scope=" + scope + ", credential=" + credential.getClass().getSimpleName() + '}'; + } +} diff --git a/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory b/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory new file mode 100644 index 000000000..97f279b6a --- /dev/null +++ b/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory @@ -0,0 +1 @@ +io.questdb.client.azure.AzureTokenProviderFactory diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java new file mode 100644 index 000000000..d8d6b470b --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -0,0 +1,292 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import com.azure.core.credential.AccessToken; +import com.azure.core.credential.TokenCredential; +import com.azure.core.credential.TokenRequestContext; +import com.azure.core.exception.ClientAuthenticationException; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.HttpResponse; +import com.azure.identity.AuthenticationRequiredException; +import com.azure.identity.CredentialUnavailableException; +import io.questdb.client.azure.AzureTokenProviderFactory; +import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.Assert; +import org.junit.Test; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.time.Duration; +import java.time.Instant; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +/** + * The {@code azure} provider (design/qwp-token-provider-spec.md, section 7.5) against fake Azure Identity + * credentials - no network: expiry and refresh-hint mapping, the requested scope, the classification of library + * errors, and discovery through {@link java.util.ServiceLoader}. + */ +public class AzureTokenSourceTest { + private static final OffsetDateTime EXPIRES = OffsetDateTime.of(2027, 1, 15, 12, 0, 0, 0, ZoneOffset.UTC); + private static final OffsetDateTime REFRESH = EXPIRES.minusMinutes(30); + + @Test + public void testAuthenticationFailuresEntraReportsArePermanent() { + assertPermanent(new ClientAuthenticationException("bad request", new FakeResponse(400, null)), "HTTP 400"); + assertPermanent(new ClientAuthenticationException("unauthorized", new FakeResponse(401, null)), "HTTP 401"); + // DefaultAzureCredential often wraps MSAL without a response: recognize the Entra error code + assertPermanent(new ClientAuthenticationException( + "AADSTS700016: Application with identifier 'x' was not found in the directory", null, (Object) null), + "AADSTS700016"); + assertPermanent(new RuntimeException("wrapper", new IllegalStateException( + "AADSTS7000215: Invalid client secret provided.")), "AADSTS7000215"); + } + + @Test + public void testConnectStringResolvesTheAzureFactory() { + // C18 with the module present: the azure keys parse on both clients and nothing is fetched + String cfg = "wss::addr=localhost:9000;token_provider=azure;azure_resource=api://qdb/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"; + TokenProviderSpec spec = TokenProviderSpec.parse(new ConfigView(ConfigString.parse(cfg)), true); + Assert.assertNotNull(spec); + Assert.assertTrue(spec.factory() instanceof AzureTokenProviderFactory); + Assert.assertEquals("api://qdb", spec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + Assert.assertEquals("azure[resource=api://qdb, client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee]", spec.describe()); + QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(cfg)), true); + try { + TokenProviderSpec.parse(new ConfigView(ConfigString.parse("wss::addr=localhost:9000;token_provider=azure;")), true); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("requires azure_resource")); + } + try { + TokenProviderSpec.parse(new ConfigView(ConfigString.parse("wss::addr=localhost:9000;token_provider=nope;")), true); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("supported values: [azure]")); + } + } + + @Test + public void testCredentialUnavailableIsPermanent() { + assertPermanent(new CredentialUnavailableException("EnvironmentCredential authentication unavailable"), + "no credential available"); + assertPermanent(new AuthenticationRequiredException("interactive sign-in required", new TokenRequestContext()), + "no credential available"); + assertPermanent(new RuntimeException("chain failed", new CredentialUnavailableException("none")), + "no credential available"); + } + + @Test + public void testExpiryAndRefreshHintAreMapped() { + AzureTokenSource source = new AzureTokenSource( + ctx -> Mono.just(new AccessToken("eyJ.token", EXPIRES, REFRESH)), "api://qdb"); + ExpiringToken token = source.fetchToken(); + Assert.assertEquals("eyJ.token", token.getToken()); + Assert.assertEquals(EXPIRES.toInstant().toEpochMilli(), token.getExpiresAtEpochMillis()); + Assert.assertEquals(REFRESH.toInstant().toEpochMilli(), token.getRefreshAtEpochMillis()); + + AzureTokenSource noHint = new AzureTokenSource(ctx -> Mono.just(new AccessToken("t", EXPIRES)), "api://qdb"); + Assert.assertFalse(noHint.fetchToken().hasRefreshAt()); + } + + @Test + public void testFactoryBuildsADefaultAzureCredentialSourceWithoutNetworkIo() { + AzureTokenProviderFactory factory = new AzureTokenProviderFactory(); + Map params = new HashMap<>(); + params.put(TokenProviderSpec.KEY_AZURE_RESOURCE, "api://qdb"); + params.put(TokenProviderSpec.KEY_AZURE_CLIENT_ID, "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"); + TokenSource source = factory.createSource(params); + Assert.assertTrue(source instanceof AzureTokenSource); + Assert.assertEquals("api://qdb/.default", ((AzureTokenSource) source).getScope()); + Assert.assertTrue(source.toString(), source.toString().contains("DefaultAzureCredential")); + } + + @Test + public void testFactoryIsDiscoveredThroughServiceLoader() { + Assert.assertTrue(TokenProviderRegistry.findFactory("azure") instanceof AzureTokenProviderFactory); + List supported = TokenProviderRegistry.supportedProviders(); + Assert.assertTrue(supported.toString(), supported.contains("azure")); + } + + @Test + public void testLibraryExceptionsAreNeverAttached() { + // a library exception's response references the raw HTTP request, which can carry a client secret + TokenUnavailableException e = fetchFailure(new HttpResponseException("throttled", new FakeResponse(429, "3"))); + Assert.assertNull(e.getCause()); + } + + @Test + public void testOtherFailuresAreRetryable() { + assertRetryable(new HttpResponseException("throttled", new FakeResponse(429, "7")), "HTTP 429", 7_000); + assertRetryable(new HttpResponseException("unavailable", new FakeResponse(503, null)), "HTTP 503", -1); + assertRetryable(new HttpResponseException("gone", new FakeResponse(410, "Wed, 21 Oct 2015 07:28:00 GMT")), "HTTP 410", -1); + assertRetryable(new IllegalStateException("connection reset"), "connection reset", -1); + assertRetryable(new ClientAuthenticationException("ManagedIdentityCredential: IMDS endpoint timed out", null, + (Object) null), "timed out", -1); + } + + @Test + public void testResultsWithoutAValueOrExpiryAreRetryable() { + TokenUnavailableException e = expectFailure(new AzureTokenSource( + ctx -> Mono.just(new AccessToken("t", null)), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("without an expiry")); + e = expectFailure(new AzureTokenSource(ctx -> Mono.empty(), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + e = expectFailure(new AzureTokenSource(ctx -> Mono.just(new AccessToken("bad\r\ntoken", EXPIRES)), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + Assert.assertFalse("the token must not be echoed", e.getMessage().contains("bad")); + } + + @Test + public void testScopeIsResourceDefault() { + AtomicReference> scopes = new AtomicReference<>(); + TokenCredential credential = ctx -> { + scopes.set(ctx.getScopes()); + return Mono.just(new AccessToken("t", EXPIRES)); + }; + new AzureTokenSource(credential, "api://qdb").fetchToken(); + Assert.assertEquals("[api://qdb/.default]", scopes.get().toString()); + new AzureTokenSource(credential, "11111111-2222-3333-4444-555555555555/.default").fetchToken(); + Assert.assertEquals("[11111111-2222-3333-4444-555555555555/.default]", scopes.get().toString()); + } + + @Test(timeout = 30_000) + public void testSlowCredentialIsBoundedAndRetryable() { + AzureTokenSource source = new AzureTokenSource(ctx -> Mono.never(), "api://qdb", Duration.ofMillis(200)); + long start = System.nanoTime(); + TokenUnavailableException e = expectFailure(source); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue("the attempt must be bounded", System.nanoTime() - start < 10_000_000_000L); + } + + @Test(timeout = 30_000) + public void testWorksBehindTheRefreshingProvider() { + OffsetDateTime expires = OffsetDateTime.now(ZoneOffset.UTC).plusHours(1); + AzureTokenSource source = new AzureTokenSource( + ctx -> Mono.just(new AccessToken("eyJ.cached", expires, expires.minusMinutes(50))), "api://qdb"); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("eyJ.cached", provider.getToken().toString()); + Assert.assertEquals(expires.toInstant().toEpochMilli(), provider.getTokenExpiresAtEpochMillis()); + Assert.assertTrue(provider.getLastSuccessEpochMillis() <= Instant.now().toEpochMilli()); + } + } + + private static void assertPermanent(Throwable libraryFailure, String fragment) { + TokenUnavailableException e = fetchFailure(libraryFailure); + Assert.assertFalse("expected permanent: " + e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains(fragment)); + } + + private static void assertRetryable(Throwable libraryFailure, String fragment, long retryAfterMillis) { + TokenUnavailableException e = fetchFailure(libraryFailure); + Assert.assertTrue("expected retryable: " + e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains(fragment)); + Assert.assertEquals(retryAfterMillis, e.getRetryAfterMillis()); + } + + private static TokenUnavailableException expectFailure(AzureTokenSource source) { + try { + source.fetchToken(); + Assert.fail("expected the fetch to fail"); + return null; + } catch (TokenUnavailableException e) { + return e; + } + } + + private static TokenUnavailableException fetchFailure(Throwable libraryFailure) { + return expectFailure(new AzureTokenSource(ctx -> Mono.error(libraryFailure), "api://qdb")); + } + + private static final class FakeResponse extends HttpResponse { + private final String retryAfter; + private final int status; + + FakeResponse(int status, String retryAfter) { + super(null); + this.status = status; + this.retryAfter = retryAfter; + } + + @Override + public Flux getBody() { + return Flux.empty(); + } + + @Override + public Mono getBodyAsByteArray() { + return Mono.empty(); + } + + @Override + public Mono getBodyAsString() { + return Mono.empty(); + } + + @Override + public Mono getBodyAsString(Charset charset) { + return Mono.empty(); + } + + @Override + public String getHeaderValue(String name) { + return "Retry-After".equalsIgnoreCase(name) ? retryAfter : null; + } + + @Override + public HttpHeaders getHeaders() { + HttpHeaders headers = new HttpHeaders(); + if (retryAfter != null) { + headers.set("Retry-After", retryAfter); + } + return headers; + } + + @Override + public int getStatusCode() { + return status; + } + } +} diff --git a/core/src/main/java/io/questdb/client/ConnectionHealth.java b/core/src/main/java/io/questdb/client/ConnectionHealth.java new file mode 100644 index 000000000..3ea731523 --- /dev/null +++ b/core/src/main/java/io/questdb/client/ConnectionHealth.java @@ -0,0 +1,290 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client; + +import java.time.Instant; + +/** + * An immutable snapshot of a QWP client's connection health (design/qwp-token-provider-spec.md, section 8.4). + * A sender in store-and-forward mode retries an outage indefinitely - a revoked credential, an unreachable + * cluster - while the application keeps writing; this snapshot is how an application, or a health endpoint, + * finds out that it has not reached the server for an hour even when no error handler is registered. + *

+ * Reading it never waits on I/O or on a connection thread: {@link Sender#health()}, + * {@code QwpQueryClient.health()} and {@link QuestDB#health()} return a published snapshot, safe to call from any + * thread at any rate. It never contains a credential. + */ +public final class ConnectionHealth { + /** + * Value of the timestamp getters when there is nothing to report. + */ + public static final long NONE = -1L; + private final long failedRounds; + private final Failure lastFailure; + private final long lastConnectedAtEpochMillis; + private final long outageSinceEpochMillis; + private final State state; + + public ConnectionHealth( + State state, + long lastConnectedAtEpochMillis, + long outageSinceEpochMillis, + long failedRounds, + Failure lastFailure + ) { + this.state = state; + this.lastConnectedAtEpochMillis = lastConnectedAtEpochMillis; + this.outageSinceEpochMillis = outageSinceEpochMillis; + this.failedRounds = failedRounds; + this.lastFailure = lastFailure; + } + + private static String instant(long epochMillis) { + return epochMillis == NONE ? "none" : Instant.ofEpochMilli(epochMillis).toString(); + } + + /** + * Number of failed connect rounds - walks over the configured endpoints that ended without a connection - + * since {@link #getOutageSinceEpochMillis()}. Reset when a connection is established. + */ + public long getFailedRounds() { + return failedRounds; + } + + /** + * The most recent failed connect round: its class, a sanitized message and its time. Kept after the + * connection recovers. Null when no round has failed. + */ + public Failure getLastFailure() { + return lastFailure; + } + + /** + * Wall-clock time of the most recent successful upgrade, or {@link #NONE}. + */ + public long getLastConnectedAtEpochMillis() { + return lastConnectedAtEpochMillis; + } + + /** + * Start of the current period without a connection, or {@link #NONE} while connected. + */ + public long getOutageSinceEpochMillis() { + return outageSinceEpochMillis; + } + + public State getState() { + return state; + } + + @Override + public String toString() { + return "ConnectionHealth{state=" + state + + ", lastConnectedAt=" + instant(lastConnectedAtEpochMillis) + + ", outageSince=" + instant(outageSinceEpochMillis) + + ", failedRounds=" + failedRounds + + ", lastFailure=" + lastFailure + '}'; + } + + /** + * The class of a failed connect round. + */ + public enum FailureClass { + /** + * The token provider could not supply a credential; no endpoint was contacted. + */ + CREDENTIAL_UNAVAILABLE, + /** + * An endpoint rejected the credential with {@code 401} or {@code 403}; see {@link Failure#getStatusCode()}. + */ + AUTH_REJECTED, + /** + * Every reachable endpoint has the wrong role, such as a replica during a failover. + */ + ROLE_REJECTED, + /** + * No endpoint could be reached: connection refused, timed out, or dropped. + */ + TRANSPORT, + /** + * Anything else: another HTTP status at the upgrade, a protocol or capability mismatch. + */ + OTHER + } + + /** + * The connection state. + */ + public enum State { + /** + * No successful upgrade yet. + */ + CONNECTING, + /** + * Connected. + */ + CONNECTED, + /** + * The connection was lost and is being re-established. For a query client, which connects on demand: the + * last connect or failover reconnect failed and the next operation will try again. + */ + RECONNECTING, + /** + * Terminal: the client will not connect again. + */ + FAILED, + /** + * Closed by the application. + */ + CLOSED + } + + /** + * Health across many connections - every pooled connection of a {@link QuestDB} handle: the number of + * connections in each state, the oldest outage and the most recent failure. + */ + public static final class Aggregate { + private final int[] counts; + private final Failure lastFailure; + private final long oldestOutageSinceEpochMillis; + + private Aggregate(int[] counts, long oldestOutageSinceEpochMillis, Failure lastFailure) { + this.counts = counts; + this.oldestOutageSinceEpochMillis = oldestOutageSinceEpochMillis; + this.lastFailure = lastFailure; + } + + /** + * Aggregates the given snapshots. + */ + public static Aggregate of(Iterable healths) { + int[] counts = new int[State.values().length]; + long oldest = NONE; + Failure last = null; + for (ConnectionHealth h : healths) { + counts[h.state.ordinal()]++; + if (h.outageSinceEpochMillis != NONE && (oldest == NONE || h.outageSinceEpochMillis < oldest)) { + oldest = h.outageSinceEpochMillis; + } + if (h.lastFailure != null && (last == null || h.lastFailure.epochMillis > last.epochMillis)) { + last = h.lastFailure; + } + } + return new Aggregate(counts, oldest, last); + } + + /** + * Number of connections in {@code state}. + */ + public int count(State state) { + return counts[state.ordinal()]; + } + + /** + * The most recent failed connect round across all connections, or null. + */ + public Failure getLastFailure() { + return lastFailure; + } + + /** + * The oldest {@code outage_since} across all connections, or {@link #NONE} when all are connected. + */ + public long getOldestOutageSinceEpochMillis() { + return oldestOutageSinceEpochMillis; + } + + /** + * Number of connections aggregated. + */ + public int total() { + int n = 0; + for (int c : counts) { + n += c; + } + return n; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("ConnectionHealth.Aggregate{"); + State[] states = State.values(); + for (int i = 0; i < states.length; i++) { + sb.append(states[i].name().toLowerCase(java.util.Locale.ROOT)).append('=').append(counts[i]).append(", "); + } + return sb.append("oldestOutageSince=").append(instant(oldestOutageSinceEpochMillis)) + .append(", lastFailure=").append(lastFailure).append('}').toString(); + } + } + + /** + * A failed connect round. Never carries a credential. + */ + public static final class Failure { + private final long epochMillis; + private final FailureClass failureClass; + private final String message; + private final int statusCode; + + public Failure(FailureClass failureClass, int statusCode, String message, long epochMillis) { + this.failureClass = failureClass; + this.statusCode = statusCode; + this.message = message; + this.epochMillis = epochMillis; + } + + /** + * Wall-clock time of the failure. + */ + public long getEpochMillis() { + return epochMillis; + } + + public FailureClass getFailureClass() { + return failureClass; + } + + /** + * A sanitized description of the failure, at most 256 characters. + */ + public String getMessage() { + return message; + } + + /** + * The HTTP status of the upgrade rejection - {@code 401} or {@code 403} for + * {@link FailureClass#AUTH_REJECTED} - or {@code 0} when none applies. + */ + public int getStatusCode() { + return statusCode; + } + + @Override + public String toString() { + return "Failure{class=" + failureClass + (statusCode > 0 ? ", status=" + statusCode : "") + + ", at=" + instant(epochMillis) + ", message=" + message + '}'; + } + } +} diff --git a/core/src/main/java/io/questdb/client/HttpTokenProvider.java b/core/src/main/java/io/questdb/client/HttpTokenProvider.java index cf28bbf50..02b318e68 100644 --- a/core/src/main/java/io/questdb/client/HttpTokenProvider.java +++ b/core/src/main/java/io/questdb/client/HttpTokenProvider.java @@ -47,6 +47,18 @@ * resolution to the OS. A provider that builds its own HTTP client should bound its connect and TLS * handshake likewise, or a black-holed token endpoint stalls a flush for the OS connect timeout. An exception from {@link #getToken()} fails the * in-flight flush (HTTP) or the connection attempt (WebSocket). + *

+ * Failure classification (WebSocket). A {@link io.questdb.client.cutlass.auth.TokenUnavailableException} marked + * retryable is a transient credential outage: a {@code Sender} whose initial connect retries + * ({@code initial_connect_retry=on}) keeps retrying it within {@code reconnect_max_duration_millis}. Any other + * exception counts as a permanent credential failure, so startup fails fast, as it does for an OIDC device-flow + * provider that is not signed in yet. Once a sender has connected, every credential failure is retried while + * store-and-forward keeps the rows. + *

+ * For a token that rotates on a schedule - a Microsoft Entra ID managed identity or service principal, an OAuth + * client-credentials grant - wrap a {@link io.questdb.client.cutlass.auth.TokenSource} in a + * {@link io.questdb.client.cutlass.auth.RefreshingTokenProvider}, which caches the token and refreshes it in the + * background before it expires. * * @see QuestDB#connect(CharSequence, HttpTokenProvider) * @see QuestDBBuilder#httpTokenProvider(HttpTokenProvider) @@ -106,4 +118,23 @@ static void validateToken(CharSequence token) { * @return the current HTTP authentication token */ CharSequence getToken(); + + /** + * Tells the provider that a server rejected a token it handed out, so that a caching provider can refresh + * early. WebSocket ingest and query clients call this when an upgrade that presented {@code token} is + * answered with {@code 401} - and, when the server sent a {@code WWW-Authenticate: Bearer} challenge that + * carries an {@code error}, only when that error is {@code invalid_token}. They then call {@link #getToken()} + * again and, if the token changed, retry the same endpoint once. A {@code 403} is never reported: it is an + * authorization decision that a new token does not change. + *

+ * The call may block briefly while the provider refreshes - a {@code RefreshingTokenProvider} waits up to its + * {@code forced_wait}, 5 s by default - and must honour an interrupt by returning promptly with the + * interrupt flag still set. It should not throw; the clients ignore anything it throws. The default does + * nothing, which suits a provider that always returns its freshest token anyway. + * + * @param token the rejected token, without the {@code "Bearer "} prefix + * @param httpStatus the HTTP status of the rejection + */ + default void onTokenRejected(CharSequence token, int httpStatus) { + } } diff --git a/core/src/main/java/io/questdb/client/QuestDB.java b/core/src/main/java/io/questdb/client/QuestDB.java index cf669a166..8cfc437b7 100644 --- a/core/src/main/java/io/questdb/client/QuestDB.java +++ b/core/src/main/java/io/questdb/client/QuestDB.java @@ -160,6 +160,18 @@ static QuestDB connect(CharSequence configurationString, HttpTokenProvider token */ Sender borrowSender(); + /** + * Aggregated connection health of every pooled connection - ingest senders and query clients: the number of + * connections in each state, the oldest outage and the most recent failed connect round + * (design/qwp-token-provider-spec.md, section 8.4). Cheap and safe to call from any thread, so it can back a + * health endpoint; it never waits on I/O and never contains a credential. + * + * @return the aggregate + */ + default ConnectionHealth.Aggregate health() { + throw new UnsupportedOperationException("connection health is not available from this QuestDB implementation"); + } + /** * Shuts down the pools and their published clients. Idempotent. Threads * currently blocked in {@link #borrowSender()} or {@link Query#submit()} diff --git a/core/src/main/java/io/questdb/client/QuestDBBuilder.java b/core/src/main/java/io/questdb/client/QuestDBBuilder.java index e846ad129..b37cdcf42 100644 --- a/core/src/main/java/io/questdb/client/QuestDBBuilder.java +++ b/core/src/main/java/io/questdb/client/QuestDBBuilder.java @@ -192,6 +192,10 @@ public QuestDB build() { throw new IllegalArgumentException( "httpTokenProvider cannot be combined with token, username, or password in the configuration"); } + if (httpTokenProvider != null && view.has("token_provider")) { + throw new IllegalArgumentException( + "httpTokenProvider cannot be combined with token_provider in the configuration"); + } // Validate the single cluster config exactly as both pools will, but // without connecting: the full Sender parse plus validateParameters // (ingress value keys are registry-STRING, so only the real parse diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 645d7b254..57ff7bf59 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -25,6 +25,8 @@ package io.questdb.client; import io.questdb.client.cutlass.auth.AuthUtils; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.line.AbstractLineTcpSender; import io.questdb.client.cutlass.line.LineChannel; import io.questdb.client.cutlass.line.LineSenderException; @@ -77,6 +79,7 @@ import java.util.Base64; import java.util.List; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Supplier; /** @@ -623,6 +626,22 @@ default long getAckedFsn() { return -1L; } + /** + * A snapshot of this sender's connection health: whether it is connected, since when it has been without a + * connection, how many connect rounds have failed since, and the most recent failure with its class + * (design/qwp-token-provider-spec.md, section 8.4). A store-and-forward sender retries an outage - a revoked + * credential, an unreachable cluster - indefinitely while the application keeps writing; this is how to see + * that it has not reached the server. Cheap and safe to call from any thread: it never waits on I/O or on the + * sender's I/O thread, so it can back a health endpoint. It never contains a credential. + * + * @return the current health + * @throws UnsupportedOperationException for transports that do not track connection health: only the + * WebSocket (QWP) sender does + */ + default ConnectionHealth health() { + throw new UnsupportedOperationException("connection health is only available for WebSocket (QWP) senders"); + } + /** * Add a column with a 32-bit signed integer value. * @@ -947,6 +966,8 @@ enum Transport { */ final class LineSenderBuilder { private static final int AUTO_FLUSH_DISABLED = 0; + // Warn once per process when an application-supplied token provider is used over ws:: (spec section 9). + private static final AtomicBoolean CLEARTEXT_PROVIDER_WARNED = new AtomicBoolean(); private static final String TLS_ROOTS_INSECURE_CONFIG_ERROR = "tls_roots cannot be combined with tls_verify=unsafe_off; remove tls_verify to use custom roots, or remove tls_roots to disable certificate validation"; // close() drain timeout. Default applied at build() time. 0 or -1 // means "fast close" (skip the drain entirely); any positive value @@ -1041,6 +1062,8 @@ final class LineSenderBuilder { OrphanScanner.QUARANTINE_SLOT_INFIX; private final ObjList hosts = new ObjList<>(); private final IntList ports = new IntList(); + // Optional authentication-outage deadline (spec section 8.5); 0 = not set, the default. + private long authFailureMaxDurationMillis; private long authTimeoutMillis = QwpWebSocketSender.DEFAULT_AUTH_TIMEOUT_MS; private int autoFlushBytes = PARAMETER_NOT_SET_EXPLICITLY; private int autoFlushIntervalMillis = PARAMETER_NOT_SET_EXPLICITLY; @@ -1093,6 +1116,10 @@ final class LineSenderBuilder { private int httpTimeout = PARAMETER_NOT_SET_EXPLICITLY; private String httpToken; private HttpTokenProvider httpTokenProvider; + // A token_provider selected by the connect string (wss:: only). Resolved through the process-wide + // TokenProviderRegistry when build() connects - never at parse time, so validating a configuration + // fetches no token - and released when the sender closes. + private TokenProviderSpec tokenProviderSpec; // Drives the initial-connect strategy. null means "not set // explicitly", which build() resolves to SYNC when any reconnect_* // knob was tuned by the user, otherwise OFF. SYNC retries on the @@ -1274,6 +1301,33 @@ public LineSenderBuilder connectTimeoutMillis(int millis) { return this; } + /** + * Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5): the sender + * becomes terminal when a connect round fails with an authentication-class failure - the token provider + * could not supply a credential, or an endpoint answered {@code 401}/{@code 403} - once that outage has + * lasted this long. Not set by default: such outages are retried indefinitely while store-and-forward + * keeps the rows. The clock starts at the first authentication-class failure of an outage and only a + * successful upgrade resets it; failures of other classes neither reset nor fire it. Applies while the + * sender is established and during an {@code async} initial connect; orphan drains are unaffected. When + * it fires, the error names the failure class and the elapsed time, goes to the error handler as + * terminal, and is thrown from later producer calls; unacknowledged rows stay in on-disk + * store-and-forward for a later sender or an orphan drain. WebSocket transport only. Connect-string key: + * {@code auth_failure_max_duration_millis}. + * + * @param millis the deadline, {@code > 0} + * @return this instance for method chaining + */ + public LineSenderBuilder authFailureMaxDurationMillis(long millis) { + if (protocol != PARAMETER_NOT_SET_EXPLICITLY && protocol != PROTOCOL_WEBSOCKET) { + throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); + } + if (millis <= 0) { + throw new LineSenderException("auth_failure_max_duration_millis must be > 0: ").put(millis); + } + this.authFailureMaxDurationMillis = millis; + return this; + } + /** * Per-endpoint timeout on the WebSocket upgrade response read. Default * {@value QwpWebSocketSender#DEFAULT_AUTH_TIMEOUT_MS} ms. @@ -1455,355 +1509,21 @@ public Sender build() { } if (protocol == PROTOCOL_WEBSOCKET) { - if (hosts.size() < 1) { - throw new LineSenderException("WebSocket transport requires at least one host:port pair"); - } - - int actualAutoFlushRows = autoFlushRows == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_ROWS : autoFlushRows; - int actualAutoFlushBytes = autoFlushBytes == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_BYTES : autoFlushBytes; - long actualAutoFlushIntervalNanos = autoFlushIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS - : TimeUnit.MILLISECONDS.toNanos(autoFlushIntervalMillis); - - Supplier wsAuthHeader = buildWebSocketAuthHeader(); - - ClientTlsConfiguration wsTlsConfig = null; - if (tlsEnabled) { - assert trustStorePassword == null || trustStorePath != null; - wsTlsConfig = new ClientTlsConfiguration( - trustStorePath, - trustStorePassword, - tlsValidationMode == TlsValidationMode.DEFAULT - ? ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL - : ClientTlsConfiguration.TLS_VALIDATION_MODE_NONE - ); + if (tokenProviderSpec == null) { + return buildWebSocket(httpTokenProvider); } - - // Setting sfDir enables store-and-forward (mmap'd, recoverable - // across sender restarts); omitting it gives memory-only mode - // (same lock-free architecture, no disk involvement). - // Durability-combination validation lives in validateParameters - // so build() and no-connect validation apply the same rules. - long actualSfMaxSegmentBytes = sfMaxSegmentBytes == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_SEGMENT_BYTES - : sfMaxSegmentBytes; - // Default cap depends on backing: RAM (memory mode) is tight - // by default; disk (SF mode) is cheap so the default is - // generous enough that normal traffic never hits it. - long defaultMaxTotal = sfDir == null - ? DEFAULT_MAX_BYTES_MEMORY - : DEFAULT_MAX_BYTES_SF; - long actualSfMaxTotalBytes = sfMaxTotalBytes == PARAMETER_NOT_SET_EXPLICITLY - ? Math.max(defaultMaxTotal, actualSfMaxSegmentBytes * 2) - : sfMaxTotalBytes; - long actualCloseFlushTimeoutMillis = closeFlushTimeoutMillis == CLOSE_FLUSH_TIMEOUT_NOT_SET - ? DEFAULT_CLOSE_FLUSH_TIMEOUT_MILLIS - : closeFlushTimeoutMillis; - long actualReconnectMaxDurationMillis = - reconnectMaxDurationMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS - : reconnectMaxDurationMillis; - long actualReconnectInitialBackoffMillis = - reconnectInitialBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_INITIAL_BACKOFF_MILLIS - : reconnectInitialBackoffMillis; - long actualReconnectMaxBackoffMillis = - reconnectMaxBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_BACKOFF_MILLIS - : reconnectMaxBackoffMillis; - // Resolve the initial-connect mode. An explicit user choice - // (via initialConnectMode/initialConnectRetry, or the - // initial_connect_retry conf key) wins unconditionally -- - // including initial_connect_retry=off paired with a tuned - // reconnect budget. When the user left it unset and tuned - // any reconnect_* knob, promote to SYNC so the budget they - // wrote actually applies to the first connect: the knob - // name reads as a generic retry budget but the underlying - // path only governs reconnects from an established - // connection, and silently ignoring the budget on the - // initial connect is the canonical footgun this implicit - // upgrade removes. - InitialConnectMode actualInitialConnectMode; - if (initialConnectMode != null) { - actualInitialConnectMode = initialConnectMode; - } else if (reconnectMaxDurationMillis != PARAMETER_NOT_SET_EXPLICITLY - || reconnectInitialBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY - || reconnectMaxBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY) { - actualInitialConnectMode = InitialConnectMode.SYNC; - } else { - actualInitialConnectMode = InitialConnectMode.OFF; - } - long actualDurableAckKeepaliveIntervalMillis = - durableAckKeepaliveIntervalMillis == DURABLE_ACK_KEEPALIVE_NOT_SET - ? CursorWebSocketSendLoop.DEFAULT_DURABLE_ACK_KEEPALIVE_INTERVAL_MILLIS - : durableAckKeepaliveIntervalMillis; - int actualMaxFrameRejections = maxFrameRejections != PARAMETER_NOT_SET_EXPLICITLY - ? maxFrameRejections - : CursorWebSocketSendLoop.DEFAULT_MAX_HEAD_FRAME_REJECTIONS; - long actualPoisonMinEscalationWindowMillis = poisonMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY - ? poisonMinEscalationWindowMillis - : CursorWebSocketSendLoop.DEFAULT_POISON_MIN_ESCALATION_WINDOW_MILLIS; - long actualCatchUpCapGapMinEscalationWindowMillis = - catchUpCapGapMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY - ? catchUpCapGapMinEscalationWindowMillis - : CursorWebSocketSendLoop.DEFAULT_CATCHUP_CAP_GAP_MIN_ESCALATION_WINDOW_MILLIS; - - // sfDir is the parent (group root); the actual slot lives - // under sfDir/senderId. This is what the engine sees — the - // slot lock and segment files all live one level deeper than - // the user-supplied path. Memory mode skips this composition - // (slotPath stays null). - // - // The slot ctor inside CursorSendEngine creates the slot - // directory itself, but Files.mkdir is non-recursive — so we - // must ensure the parent group root exists first. - String slotPath; - if (sfDir == null) { - slotPath = null; - } else { - if (!Files.exists(sfDir)) { - int rc = Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT); - // mkdir is non-zero on failure, but "already exists" - // is one such failure. Multiple SF senders sharing one - // sf_dir can be built concurrently (the pool calls - // build() outside its lock), so two threads can both - // pass the exists() check and race into mkdir; the - // loser gets EEXIST. Treat a benign creation race -- - // the dir now exists -- as success and only fail when - // the directory is genuinely absent afterwards. - if (rc != 0 && !Files.exists(sfDir)) { - throw new LineSenderException( - "could not create sf_dir: " + sfDir + " rc=" + rc); - } - } - if (sfDurability == SfDurability.PERIODIC - && Files.fsyncParentDir(sfDir) != 0) { - throw new LineSenderException( - "could not sync parent directory for sf_dir: " + sfDir); - } - slotPath = sfDir + "/" + senderId; - } - long actualSfAppendDeadlineNanos = - sfAppendDeadlineMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS - : sfAppendDeadlineMillis * 1_000_000L; - long actualSfSyncIntervalNanos = sfDurability == SfDurability.PERIODIC - ? (sfSyncIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_SF_SYNC_INTERVAL_MILLIS : sfSyncIntervalMillis) * 1_000_000L - : 0L; - QwpWebSocketSender connected = null; - // The parent-anchored logical lock is stable across a slot rename. Keep it - // from before the directory-local lock is acquired until connect() has either - // adopted that engine or quarantine has closed, renamed and recreated it. - // This closes the inode-swap window in which an already-queued orphan drainer - // could otherwise acquire the renamed directory's old .lock and later operate - // on the fresh slot through the original pathname. - try (SlotLock logicalSlotLock = slotPath == null - ? null - : SlotLock.acquireLogical(slotPath)) { - // The constructor's own recovery seed can also fail terminally, and - // not only as UnreplayableSlotException: when SegmentRing.openExisting - // had to skip an unreadable segment it throws SfRecoveryException (it - // constructs UnreplayableSlotException nowhere), and where it cannot - // even prove the chain's identity -- no manifest -- it quarantines the - // corrupt files and returns an EMPTY recovery rather than refusing. - // Either way the frame range cannot be shown already-acked, so recovery - // sets the slot aside rather than risk seeding the ack cursor past - // frames that were never delivered. All three types below are load - // bearing; narrowing this catch to UnreplayableSlotException would - // restore the permanent build() brick for the segment-skip case. That verdict gets - // the exact same quarantine-and-continue treatment as the connect()-time - // verdict below -- constructing cursorEngine is not inside the loop below, - // so a throw here would otherwise escape build() entirely, uncaught. - // quarantineTornSlot(null, ...) renames the WHOLE slot directory aside - // (not just the unreadable segment file) before building the replacement - // at the original slotPath, so the replacement starts on a genuinely empty - // directory with nothing left to skip -- it cannot throw the same way - // twice, which is what makes looping unnecessary here. - boolean quarantined = false; - CursorSendEngine cursorEngine; - try { - try { - cursorEngine = new CursorSendEngine( - slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos); - } catch (SfSanitizedResidueException first) { - // NOT terminal, and it must be intercepted ahead of its - // SfRecoveryException parent below. Recovery durably zeroed - // proven-dead sealed residue BEFORE failing closed, so the - // chain on disk is already healed: quarantining here would - // set aside a slot whose backlog replays perfectly. Retry - // once over the healed chain; a repeat is genuine and takes - // the terminal arm. - LOG.info("sf slot {}: sealed residue sanitized during recovery ({}); " - + "retrying over the healed chain", - slotPath, first.getMessage()); - cursorEngine = new CursorSendEngine( - slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos); - } - } catch (UnreplayableSlotException | SfRecoveryException - | MmapSegmentCorruptionException e) { - // The terminal recovery verdicts, and the only ones build() - // sets a slot aside for. UnreplayableSlotException says the - // symbol dictionary cannot be rebuilt from any source; - // SfRecoveryException and MmapSegmentCorruptionException say - // the durable chain itself is proven corrupt or incomplete. - // None of the three clears on a retry, and senderId is stable - // with a not-fully-drained slot retained on close -- so - // without this arm every restart re-recovers the same slot and - // throws again, and the application cannot construct a Sender - // at all, not even to BUFFER new rows. - // - // Deliberately NOT catching plain MmapSegmentException or - // SfOperationalException: those are operational (EMFILE, - // ENOMEM, an unreadable-but-possibly-intact file). Aborting - // startup on them is correct; quarantining on them would - // convert a transient into the permanent loss of a healthy - // slot's durable frames. - if (slotPath == null) { - throw e; - } - quarantined = true; - cursorEngine = quarantineTornSlot( - null, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos, errorHandler); - } - int actualErrorInboxCapacity = errorInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY - ? errorInboxCapacity - : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher.DEFAULT_CAPACITY; - int actualConnectionListenerInboxCapacity = connectionListenerInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY - ? connectionListenerInboxCapacity - : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderConnectionDispatcher.DEFAULT_CAPACITY; - List wsEndpoints = - new ArrayList<>(hosts.size()); - for (int i = 0, n = hosts.size(); i < n; i++) { - wsEndpoints.add(new QwpWebSocketSender.Endpoint(hosts.getQuick(i), ports.getQuick(i))); - } - // The recovery seed inside connect() is the authority on whether a recovered - // slot can be replayed: it rebuilds the dictionary from its intact prefix and - // then from the surviving frames' own delta sections, and throws - // UnreplayableSlotException only once neither source holds the missing ids. - // Quarantining on anything weaker would set aside slots that recovery can - // still rescue, so build() waits for that verdict rather than pre-judging it. - while (connected == null) { - try { - connected = QwpWebSocketSender.connectWithCredentialSupplier( - wsEndpoints, - wsTlsConfig, - actualAutoFlushRows, - actualAutoFlushBytes, - actualAutoFlushIntervalNanos, - wsAuthHeader, - requestDurableAck, - cursorEngine, - actualCloseFlushTimeoutMillis, - actualReconnectMaxDurationMillis, - actualReconnectInitialBackoffMillis, - actualReconnectMaxBackoffMillis, - actualInitialConnectMode, - errorHandler, - actualErrorInboxCapacity, - actualDurableAckKeepaliveIntervalMillis, - authTimeoutMillis, - connectTimeoutMillis == PARAMETER_NOT_SET_EXPLICITLY ? 0 : connectTimeoutMillis, - connectionListener, - actualConnectionListenerInboxCapacity, - actualMaxFrameRejections, - actualPoisonMinEscalationWindowMillis, - actualCatchUpCapGapMinEscalationWindowMillis - ); - } catch (UnreplayableSlotException e) { - // The one failure build() recovers from. The slot's frames reference ids - // that nothing still holds, so they can never go on the wire -- but that is - // no reason to take the producer down with them. Before this, the throw - // escaped build() and, because senderId is stable and a not-fully-drained - // slot is retained on close, every retry re-recovered the same slot and - // threw again: the application could not construct a Sender at all, so it - // could not even BUFFER new rows. An already-lost batch became an unbounded - // outage of everything after it. - // - // Set the slot aside instead, keep its bytes for forensics and resend, and - // start the producer on a clean one. Once only: a second such failure would - // mean the FRESH slot is unreplayable, which cannot happen, so let it out - // rather than loop. - if (quarantined || slotPath == null) { - try { - // close(false): we still hold the logical slot lock. - cursorEngine.close(false); - } catch (Throwable ignored) { - // best-effort - } - throw e; - } - quarantined = true; - cursorEngine = quarantineTornSlot( - cursorEngine, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos, errorHandler); - } catch (Throwable t) { - // connect() failed before ownership of cursorEngine - // transferred — close it ourselves. close(false) - // because logicalSlotLock is still held here: a fresh - // slot is fully drained, so the default close would - // unlink the very lock file this scope holds. - try { - cursorEngine.close(false); - } catch (Throwable ignored) { - // best-effort - } - throw t; - } - } - } - // connect() succeeded — `connected` now owns cursorEngine - // via setCursorEngine(engine, true). From here on, ANY - // failure must close `connected` (which closes the engine - // through ownsCursorEngine), not cursorEngine directly: - // closing the engine alone would leak the I/O thread, - // dispatcher daemon, drainer pool, microbatch buffers and - // WebSocketClient inside the abandoned `connected`. - connected.setTransactional(transactional); + // token_provider: share the process-wide provider for this configuration. The lease is the + // sender's from here on: released when the sender closes, or right here if build() fails. + TokenProviderRegistry.Lease lease = TokenProviderRegistry.global().acquire(tokenProviderSpec); try { - // Install the drainer listener BEFORE startOrphanDrainers - // below: drainers must see the listener at submit time so - // no early drainer event is lost to a late installation. - if (drainerListener != null) { - connected.setDrainerListener(drainerListener); - } - // Once the foreground sender is up, dispatch drainers - // for any sibling orphan slots. Scan AFTER we acquire - // our own slot lock so we never accidentally try to - // adopt our own data; the OrphanScanner.scan filter - // also excludes our sender_id. - if (drainOrphans && sfDir != null) { - io.questdb.client.std.ObjList orphans = - io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner - .scan(sfDir, senderId, orphanDrainBase, orphanDrainSlotCount); - if (orphans.size() > 0) { - org.slf4j.LoggerFactory.getLogger(LineSenderBuilder.class) - .info("dispatching drainers for {} orphan slot(s) under {} " - + "(max_background_drainers={})", - orphans.size(), sfDir, maxBackgroundDrainers); - connected.startOrphanDrainers( - orphans, - maxBackgroundDrainers, - actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, - actualSfSyncIntervalNanos); - } - } - return connected; - } catch (Throwable t) { - try { - connected.close(); - } catch (Throwable ignored) { - // best-effort + QwpWebSocketSender sender = buildWebSocket(lease.provider()); + sender.setCredentialLease(lease); + lease = null; + return sender; + } finally { + if (lease != null) { + lease.close(); } - throw t; } } @@ -2282,6 +2002,9 @@ public LineSenderBuilder httpTimeoutMillis(int httpTimeoutMillis) { * @return this instance for method chaining */ public LineSenderBuilder httpToken(String token) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("token cannot be combined with token_provider"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -2341,6 +2064,10 @@ public LineSenderBuilder httpToken(String token) { * @return this instance for method chaining */ public LineSenderBuilder httpTokenProvider(HttpTokenProvider httpTokenProvider) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("an application-supplied token provider cannot be combined with " + + "token_provider in the configuration"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -2370,6 +2097,9 @@ public LineSenderBuilder httpTokenProvider(HttpTokenProvider httpTokenProvider) * @see #httpToken(String) */ public LineSenderBuilder httpUsernamePassword(String username, String password) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("username/password cannot be combined with token_provider"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -3424,7 +3154,371 @@ private void appendAddress(String host, int port) { ports.add(port); } - private Supplier buildWebSocketAuthHeader() { + private QwpWebSocketSender buildWebSocket(HttpTokenProvider tokenProvider) { + if (hosts.size() < 1) { + throw new LineSenderException("WebSocket transport requires at least one host:port pair"); + } + if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { + // design/qwp-token-provider-spec.md, section 9: a configured token_provider is rejected on ws::, + // but an application-supplied provider is the application's call - warn once instead. + LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in " + + "cleartext; use wss:: in production"); + } + + int actualAutoFlushRows = autoFlushRows == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_ROWS : autoFlushRows; + int actualAutoFlushBytes = autoFlushBytes == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_BYTES : autoFlushBytes; + long actualAutoFlushIntervalNanos = autoFlushIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS + : TimeUnit.MILLISECONDS.toNanos(autoFlushIntervalMillis); + + Supplier wsAuthHeader = buildWebSocketAuthHeader(tokenProvider); + + ClientTlsConfiguration wsTlsConfig = null; + if (tlsEnabled) { + assert trustStorePassword == null || trustStorePath != null; + wsTlsConfig = new ClientTlsConfiguration( + trustStorePath, + trustStorePassword, + tlsValidationMode == TlsValidationMode.DEFAULT + ? ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL + : ClientTlsConfiguration.TLS_VALIDATION_MODE_NONE + ); + } + + // Setting sfDir enables store-and-forward (mmap'd, recoverable + // across sender restarts); omitting it gives memory-only mode + // (same lock-free architecture, no disk involvement). + // Durability-combination validation lives in validateParameters + // so build() and no-connect validation apply the same rules. + long actualSfMaxSegmentBytes = sfMaxSegmentBytes == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_SEGMENT_BYTES + : sfMaxSegmentBytes; + // Default cap depends on backing: RAM (memory mode) is tight + // by default; disk (SF mode) is cheap so the default is + // generous enough that normal traffic never hits it. + long defaultMaxTotal = sfDir == null + ? DEFAULT_MAX_BYTES_MEMORY + : DEFAULT_MAX_BYTES_SF; + long actualSfMaxTotalBytes = sfMaxTotalBytes == PARAMETER_NOT_SET_EXPLICITLY + ? Math.max(defaultMaxTotal, actualSfMaxSegmentBytes * 2) + : sfMaxTotalBytes; + long actualCloseFlushTimeoutMillis = closeFlushTimeoutMillis == CLOSE_FLUSH_TIMEOUT_NOT_SET + ? DEFAULT_CLOSE_FLUSH_TIMEOUT_MILLIS + : closeFlushTimeoutMillis; + long actualReconnectMaxDurationMillis = + reconnectMaxDurationMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS + : reconnectMaxDurationMillis; + long actualReconnectInitialBackoffMillis = + reconnectInitialBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_INITIAL_BACKOFF_MILLIS + : reconnectInitialBackoffMillis; + long actualReconnectMaxBackoffMillis = + reconnectMaxBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_BACKOFF_MILLIS + : reconnectMaxBackoffMillis; + // Resolve the initial-connect mode. An explicit user choice + // (via initialConnectMode/initialConnectRetry, or the + // initial_connect_retry conf key) wins unconditionally -- + // including initial_connect_retry=off paired with a tuned + // reconnect budget. When the user left it unset and tuned + // any reconnect_* knob, promote to SYNC so the budget they + // wrote actually applies to the first connect: the knob + // name reads as a generic retry budget but the underlying + // path only governs reconnects from an established + // connection, and silently ignoring the budget on the + // initial connect is the canonical footgun this implicit + // upgrade removes. + InitialConnectMode actualInitialConnectMode; + if (initialConnectMode != null) { + actualInitialConnectMode = initialConnectMode; + } else if (reconnectMaxDurationMillis != PARAMETER_NOT_SET_EXPLICITLY + || reconnectInitialBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY + || reconnectMaxBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY) { + actualInitialConnectMode = InitialConnectMode.SYNC; + } else { + actualInitialConnectMode = InitialConnectMode.OFF; + } + long actualDurableAckKeepaliveIntervalMillis = + durableAckKeepaliveIntervalMillis == DURABLE_ACK_KEEPALIVE_NOT_SET + ? CursorWebSocketSendLoop.DEFAULT_DURABLE_ACK_KEEPALIVE_INTERVAL_MILLIS + : durableAckKeepaliveIntervalMillis; + int actualMaxFrameRejections = maxFrameRejections != PARAMETER_NOT_SET_EXPLICITLY + ? maxFrameRejections + : CursorWebSocketSendLoop.DEFAULT_MAX_HEAD_FRAME_REJECTIONS; + long actualPoisonMinEscalationWindowMillis = poisonMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY + ? poisonMinEscalationWindowMillis + : CursorWebSocketSendLoop.DEFAULT_POISON_MIN_ESCALATION_WINDOW_MILLIS; + long actualCatchUpCapGapMinEscalationWindowMillis = + catchUpCapGapMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY + ? catchUpCapGapMinEscalationWindowMillis + : CursorWebSocketSendLoop.DEFAULT_CATCHUP_CAP_GAP_MIN_ESCALATION_WINDOW_MILLIS; + + // sfDir is the parent (group root); the actual slot lives + // under sfDir/senderId. This is what the engine sees — the + // slot lock and segment files all live one level deeper than + // the user-supplied path. Memory mode skips this composition + // (slotPath stays null). + // + // The slot ctor inside CursorSendEngine creates the slot + // directory itself, but Files.mkdir is non-recursive — so we + // must ensure the parent group root exists first. + String slotPath; + if (sfDir == null) { + slotPath = null; + } else { + if (!Files.exists(sfDir)) { + int rc = Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT); + // mkdir is non-zero on failure, but "already exists" + // is one such failure. Multiple SF senders sharing one + // sf_dir can be built concurrently (the pool calls + // build() outside its lock), so two threads can both + // pass the exists() check and race into mkdir; the + // loser gets EEXIST. Treat a benign creation race -- + // the dir now exists -- as success and only fail when + // the directory is genuinely absent afterwards. + if (rc != 0 && !Files.exists(sfDir)) { + throw new LineSenderException( + "could not create sf_dir: " + sfDir + " rc=" + rc); + } + } + if (sfDurability == SfDurability.PERIODIC + && Files.fsyncParentDir(sfDir) != 0) { + throw new LineSenderException( + "could not sync parent directory for sf_dir: " + sfDir); + } + slotPath = sfDir + "/" + senderId; + } + long actualSfAppendDeadlineNanos = + sfAppendDeadlineMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS + : sfAppendDeadlineMillis * 1_000_000L; + long actualSfSyncIntervalNanos = sfDurability == SfDurability.PERIODIC + ? (sfSyncIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_SF_SYNC_INTERVAL_MILLIS : sfSyncIntervalMillis) * 1_000_000L + : 0L; + QwpWebSocketSender connected = null; + // The parent-anchored logical lock is stable across a slot rename. Keep it + // from before the directory-local lock is acquired until connect() has either + // adopted that engine or quarantine has closed, renamed and recreated it. + // This closes the inode-swap window in which an already-queued orphan drainer + // could otherwise acquire the renamed directory's old .lock and later operate + // on the fresh slot through the original pathname. + try (SlotLock logicalSlotLock = slotPath == null + ? null + : SlotLock.acquireLogical(slotPath)) { + // The constructor's own recovery seed can also fail terminally, and + // not only as UnreplayableSlotException: when SegmentRing.openExisting + // had to skip an unreadable segment it throws SfRecoveryException (it + // constructs UnreplayableSlotException nowhere), and where it cannot + // even prove the chain's identity -- no manifest -- it quarantines the + // corrupt files and returns an EMPTY recovery rather than refusing. + // Either way the frame range cannot be shown already-acked, so recovery + // sets the slot aside rather than risk seeding the ack cursor past + // frames that were never delivered. All three types below are load + // bearing; narrowing this catch to UnreplayableSlotException would + // restore the permanent build() brick for the segment-skip case. That verdict gets + // the exact same quarantine-and-continue treatment as the connect()-time + // verdict below -- constructing cursorEngine is not inside the loop below, + // so a throw here would otherwise escape build() entirely, uncaught. + // quarantineTornSlot(null, ...) renames the WHOLE slot directory aside + // (not just the unreadable segment file) before building the replacement + // at the original slotPath, so the replacement starts on a genuinely empty + // directory with nothing left to skip -- it cannot throw the same way + // twice, which is what makes looping unnecessary here. + boolean quarantined = false; + CursorSendEngine cursorEngine; + try { + try { + cursorEngine = new CursorSendEngine( + slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos); + } catch (SfSanitizedResidueException first) { + // NOT terminal, and it must be intercepted ahead of its + // SfRecoveryException parent below. Recovery durably zeroed + // proven-dead sealed residue BEFORE failing closed, so the + // chain on disk is already healed: quarantining here would + // set aside a slot whose backlog replays perfectly. Retry + // once over the healed chain; a repeat is genuine and takes + // the terminal arm. + LOG.info("sf slot {}: sealed residue sanitized during recovery ({}); " + + "retrying over the healed chain", + slotPath, first.getMessage()); + cursorEngine = new CursorSendEngine( + slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos); + } + } catch (UnreplayableSlotException | SfRecoveryException + | MmapSegmentCorruptionException e) { + // The terminal recovery verdicts, and the only ones build() + // sets a slot aside for. UnreplayableSlotException says the + // symbol dictionary cannot be rebuilt from any source; + // SfRecoveryException and MmapSegmentCorruptionException say + // the durable chain itself is proven corrupt or incomplete. + // None of the three clears on a retry, and senderId is stable + // with a not-fully-drained slot retained on close -- so + // without this arm every restart re-recovers the same slot and + // throws again, and the application cannot construct a Sender + // at all, not even to BUFFER new rows. + // + // Deliberately NOT catching plain MmapSegmentException or + // SfOperationalException: those are operational (EMFILE, + // ENOMEM, an unreadable-but-possibly-intact file). Aborting + // startup on them is correct; quarantining on them would + // convert a transient into the permanent loss of a healthy + // slot's durable frames. + if (slotPath == null) { + throw e; + } + quarantined = true; + cursorEngine = quarantineTornSlot( + null, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos, errorHandler); + } + int actualErrorInboxCapacity = errorInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY + ? errorInboxCapacity + : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher.DEFAULT_CAPACITY; + int actualConnectionListenerInboxCapacity = connectionListenerInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY + ? connectionListenerInboxCapacity + : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderConnectionDispatcher.DEFAULT_CAPACITY; + List wsEndpoints = + new ArrayList<>(hosts.size()); + for (int i = 0, n = hosts.size(); i < n; i++) { + wsEndpoints.add(new QwpWebSocketSender.Endpoint(hosts.getQuick(i), ports.getQuick(i))); + } + // The recovery seed inside connect() is the authority on whether a recovered + // slot can be replayed: it rebuilds the dictionary from its intact prefix and + // then from the surviving frames' own delta sections, and throws + // UnreplayableSlotException only once neither source holds the missing ids. + // Quarantining on anything weaker would set aside slots that recovery can + // still rescue, so build() waits for that verdict rather than pre-judging it. + while (connected == null) { + try { + connected = QwpWebSocketSender.connectWithCredentialSupplier( + wsEndpoints, + wsTlsConfig, + actualAutoFlushRows, + actualAutoFlushBytes, + actualAutoFlushIntervalNanos, + wsAuthHeader, + requestDurableAck, + cursorEngine, + actualCloseFlushTimeoutMillis, + actualReconnectMaxDurationMillis, + actualReconnectInitialBackoffMillis, + actualReconnectMaxBackoffMillis, + actualInitialConnectMode, + errorHandler, + actualErrorInboxCapacity, + actualDurableAckKeepaliveIntervalMillis, + authTimeoutMillis, + connectTimeoutMillis == PARAMETER_NOT_SET_EXPLICITLY ? 0 : connectTimeoutMillis, + connectionListener, + actualConnectionListenerInboxCapacity, + actualMaxFrameRejections, + actualPoisonMinEscalationWindowMillis, + actualCatchUpCapGapMinEscalationWindowMillis + ); + } catch (UnreplayableSlotException e) { + // The one failure build() recovers from. The slot's frames reference ids + // that nothing still holds, so they can never go on the wire -- but that is + // no reason to take the producer down with them. Before this, the throw + // escaped build() and, because senderId is stable and a not-fully-drained + // slot is retained on close, every retry re-recovered the same slot and + // threw again: the application could not construct a Sender at all, so it + // could not even BUFFER new rows. An already-lost batch became an unbounded + // outage of everything after it. + // + // Set the slot aside instead, keep its bytes for forensics and resend, and + // start the producer on a clean one. Once only: a second such failure would + // mean the FRESH slot is unreplayable, which cannot happen, so let it out + // rather than loop. + if (quarantined || slotPath == null) { + try { + // close(false): we still hold the logical slot lock. + cursorEngine.close(false); + } catch (Throwable ignored) { + // best-effort + } + throw e; + } + quarantined = true; + cursorEngine = quarantineTornSlot( + cursorEngine, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos, errorHandler); + } catch (Throwable t) { + // connect() failed before ownership of cursorEngine + // transferred — close it ourselves. close(false) + // because logicalSlotLock is still held here: a fresh + // slot is fully drained, so the default close would + // unlink the very lock file this scope holds. + try { + cursorEngine.close(false); + } catch (Throwable ignored) { + // best-effort + } + throw t; + } + } + } + // connect() succeeded — `connected` now owns cursorEngine + // via setCursorEngine(engine, true). From here on, ANY + // failure must close `connected` (which closes the engine + // through ownsCursorEngine), not cursorEngine directly: + // closing the engine alone would leak the I/O thread, + // dispatcher daemon, drainer pool, microbatch buffers and + // WebSocketClient inside the abandoned `connected`. + connected.setTransactional(transactional); + if (authFailureMaxDurationMillis > 0) { + // The I/O loop may already be running (async initial connect), but it tracks the outage clock + // from the first authentication-class failure regardless; this only arms the deadline. + connected.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); + } + try { + // Install the drainer listener BEFORE startOrphanDrainers + // below: drainers must see the listener at submit time so + // no early drainer event is lost to a late installation. + if (drainerListener != null) { + connected.setDrainerListener(drainerListener); + } + // Once the foreground sender is up, dispatch drainers + // for any sibling orphan slots. Scan AFTER we acquire + // our own slot lock so we never accidentally try to + // adopt our own data; the OrphanScanner.scan filter + // also excludes our sender_id. + if (drainOrphans && sfDir != null) { + io.questdb.client.std.ObjList orphans = + io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner + .scan(sfDir, senderId, orphanDrainBase, orphanDrainSlotCount); + if (orphans.size() > 0) { + org.slf4j.LoggerFactory.getLogger(LineSenderBuilder.class) + .info("dispatching drainers for {} orphan slot(s) under {} " + + "(max_background_drainers={})", + orphans.size(), sfDir, maxBackgroundDrainers); + connected.startOrphanDrainers( + orphans, + maxBackgroundDrainers, + actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, + actualSfSyncIntervalNanos); + } + } + return connected; + } catch (Throwable t) { + try { + connected.close(); + } catch (Throwable ignored) { + // best-effort + } + throw t; + } + } + + private Supplier buildWebSocketAuthHeader(HttpTokenProvider provider) { // A constant credential goes through fixedAuthHeader, not a bare lambda: the tag is what lets // the store-and-forward drainer tell a permanently-wrong password from a rotating token that a // fresh pull can repair, and so decide whether a 401 may quarantine an orphan slot for good. @@ -3437,21 +3531,13 @@ private Supplier buildWebSocketAuthHeader() { String header = "Bearer " + httpToken; return QwpWebSocketSender.fixedAuthHeader(header); } - if (httpTokenProvider != null) { - // pull a fresh token at each (re)handshake so a long-lived WebSocket follows token - // refreshes; validateToken rejects a null/empty/blank return, or a token carrying a - // control or non-ASCII char (both forbidden by the HttpTokenProvider contract), rather - // than send a malformed or CR/LF-injected "Bearer " header - final HttpTokenProvider provider = httpTokenProvider; - return () -> { - // snapshot before validating: the concatenation below re-reads the sequence, and a - // provider is free to reuse a mutable buffer, so validating the live sequence checks - // bytes the header need not carry. See HttpTokenProvider.validateToken. - CharSequence pulled = provider.getToken(); - CharSequence token = pulled == null ? null : pulled.toString(); - HttpTokenProvider.validateToken(token); - return "Bearer " + token; - }; + if (provider != null) { + // Pull a fresh token at each (re)handshake so a long-lived WebSocket follows token refreshes. + // The supplier snapshots and validates every pull (validateToken rejects a null/empty/blank + // return, or a token carrying a control or non-ASCII char, rather than send a malformed or + // CR/LF-injected "Bearer " header), and it carries the provider's onTokenRejected back-channel + // that the connect walk uses for its one retry after a 401. + return QwpWebSocketSender.tokenProviderAuthHeader(provider); } return null; } @@ -3973,6 +4059,14 @@ private LineSenderBuilder fromConfig(CharSequence configurationString) { // genuine value-parse error names the offending key. String reservedKey = Chars.toString(sink); pos = getValue(configurationString, pos, sink, reservedKey); + } else if (Chars.equals("auth_failure_max_duration_millis", sink)) { + throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); + } else if (Chars.equals("token_provider", sink) + || Chars.equals("azure_resource", sink) + || Chars.equals("azure_client_id", sink)) { + // Dynamic bearer credentials are defined for QWP over wss:: only (decision D9). + throw new LineSenderException(Chars.toString(sink) + + " is only supported with the wss:: schema (QWP over WebSocket)"); } else { // sf-client.md §4.6: parser must reject unknown keys. // Forward-compat is via the spec, not silent ignore — silent @@ -4020,6 +4114,16 @@ private LineSenderBuilder fromConfigWebSocket(CharSequence configurationString) ConfigString cs = ConfigString.parse(configurationString); ConfigView view = new ConfigView(cs); validateWsConfig(view, tlsEnabled); + // Validates token_provider and its keys (wss:: only, exclusive with static credentials, a + // supported provider) without fetching anything; build() acquires the provider. + TokenProviderSpec spec = TokenProviderSpec.parse(view, tlsEnabled); + if (spec != null) { + if (httpTokenProvider != null) { + throw new LineSenderException("token_provider cannot be combined with an " + + "application-supplied token provider"); + } + tokenProviderSpec = spec; + } view.getHostPorts("addr", DEFAULT_WEBSOCKET_PORT, this::appendAddress); @@ -4055,6 +4159,9 @@ private LineSenderBuilder fromConfigWebSocket(CharSequence configurationString) // int, and an over-int value must reject, not wrap. connectTimeoutMillis(view.getInt("connect_timeout", 0)); } + if (view.has("auth_failure_max_duration_millis")) { + authFailureMaxDurationMillis(view.getLong("auth_failure_max_duration_millis", 0)); + } s = view.getStr("auto_flush_rows"); if (s != null) { @@ -4335,6 +4442,12 @@ public java.util.Map wsConfigSnapshotForTest() { m.put("tls_verify", tlsValidationMode == null ? null : tlsValidationMode.name()); m.put("tls_roots", trustStorePath); m.put("tls_roots_password", trustStorePassword == null ? null : new String(trustStorePassword)); + m.put("auth_failure_max_duration_millis", authFailureMaxDurationMillis); + m.put("token_provider", tokenProviderSpec == null ? null : tokenProviderSpec.name()); + m.put("azure_resource", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + m.put("azure_client_id", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); return m; } diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java b/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java new file mode 100644 index 000000000..a84be9fe4 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java @@ -0,0 +1,145 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.str.DisplaySafe; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; + +/** + * Redaction rules for bearer credentials and for error text that crosses the client boundary, as required by + * the dynamic-credential specification (design/qwp-token-provider-spec.md, section 9). + *

    + *
  • A token is never rendered. {@link #describeToken(CharSequence)} gives its length and at most the first + * 8 hex digits of its SHA-256 - enough to correlate a rotation across log lines, useless to replay.
    • + *
  • Error text that came from a library or an identity endpoint goes through + * {@link #sanitizeErrorText(CharSequence)} before it reaches a client error or a log line: control, + * bidirectional and other non-displayable characters are removed, and the result is capped at + * {@value #MAX_ERROR_TEXT_LENGTH} characters.
    • + *
+ */ +public final class CredentialRedaction { + /** + * Upper bound, in characters, on error text from a library or endpoint once sanitized. + */ + public static final int MAX_ERROR_TEXT_LENGTH = 256; + private static final int FINGERPRINT_HEX_DIGITS = 8; + private static final char[] HEX = "0123456789abcdef".toCharArray(); + private static final String TRUNCATION_MARKER = "..."; + + private CredentialRedaction() { + } + + /** + * Renders a token as {@code }: its length plus the first 8 hex digits of + * its SHA-256. Never the token itself. + * + * @param token the token, may be null + * @return a description that is safe to log + */ + public static String describeToken(CharSequence token) { + if (token == null) { + return ""; + } + return "'; + } + + /** + * The first 8 hex digits of the SHA-256 of {@code token}, encoded as UTF-8. + * + * @param token the token, never null + * @return 8 lowercase hex digits + */ + public static String fingerprint(CharSequence token) { + final byte[] digest; + try { + digest = MessageDigest.getInstance("SHA-256").digest(token.toString().getBytes(StandardCharsets.UTF_8)); + } catch (NoSuchAlgorithmException e) { + // every Java platform is required to provide SHA-256 + throw new IllegalStateException("SHA-256 is not available", e); + } + final char[] out = new char[FINGERPRINT_HEX_DIGITS]; + for (int i = 0; i < FINGERPRINT_HEX_DIGITS / 2; i++) { + out[2 * i] = HEX[(digest[i] >> 4) & 0xf]; + out[2 * i + 1] = HEX[digest[i] & 0xf]; + } + return new String(out); + } + + /** + * Prepares untrusted error text - from a token source, an identity library or an endpoint - for a client + * error or a log line: removes every code point {@link DisplaySafe} rejects (controls including CR/LF, + * bidirectional overrides, zero-width and other format characters, lone surrogates) and caps the result at + * {@value #MAX_ERROR_TEXT_LENGTH} characters, ending in {@code ...} when it had to cut. + *

+ * This does not, and cannot, remove a secret the text already carries. Keeping tokens and response bodies + * out of error text is the job of whoever produces it. + * + * @param text untrusted text, may be null + * @return the sanitized text, or {@code null} when {@code text} is null + */ + public static String sanitizeErrorText(CharSequence text) { + if (text == null) { + return null; + } + final StringBuilder sb = new StringBuilder(Math.min(text.length(), MAX_ERROR_TEXT_LENGTH)); + final int limit = MAX_ERROR_TEXT_LENGTH - TRUNCATION_MARKER.length(); + for (int i = 0, n = text.length(); i < n; ) { + final int cp = Character.codePointAt(text, i); + final int count = Character.charCount(cp); + if (DisplaySafe.isDisplaySafe(cp)) { + if (sb.length() + count > limit) { + // Only cut when something displayable is left to drop: text that fits exactly is kept whole. + if (hasDisplayableFrom(text, i, MAX_ERROR_TEXT_LENGTH - sb.length())) { + sb.append(TRUNCATION_MARKER); + return sb.toString(); + } + } + sb.appendCodePoint(cp); + } + i += count; + } + return sb.toString(); + } + + // True when the displayable remainder of text, starting at from, does not fit in budget characters. + private static boolean hasDisplayableFrom(CharSequence text, int from, int budget) { + int needed = 0; + for (int i = from, n = text.length(); i < n; ) { + final int cp = Character.codePointAt(text, i); + final int count = Character.charCount(cp); + if (DisplaySafe.isDisplaySafe(cp)) { + needed += count; + if (needed > budget) { + return true; + } + } + i += count; + } + return false; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java new file mode 100644 index 000000000..95713352b --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java @@ -0,0 +1,148 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.time.Instant; + +/** + * A bearer token together with the instant it expires, as returned by a {@link TokenSource}. This is the + * {@code TokenResult} of the dynamic-credential specification (design/qwp-token-provider-spec.md, section 4). + *

+ * The token is opaque and is stored without the {@code "Bearer "} prefix. It must be non-blank printable ASCII + * ({@code 0x20}-{@code 0x7e}); the constructors reject anything else, never echoing the token in the message. + *

+ * {@code expiresAtEpochMillis} is an absolute wall-clock instant. A source that receives a relative lifetime + * (such as {@code expires_in}) should add it to its own local time of receipt, which makes the cache robust to + * clock skew between the client and the identity platform; a source that cannot determine the expiry must + * supply a conservative value. {@code refreshAtEpochMillis}, when present, is the earliest instant the platform + * suggests refreshing; {@link #NO_REFRESH_AT} means none. + *

+ * {@link #toString()} never renders the token: it shows the length, an 8-hex-digit SHA-256 fingerprint and the + * timestamps. + */ +public final class ExpiringToken { + /** + * Value of {@code refreshAtEpochMillis} meaning the platform gave no refresh hint. + */ + public static final long NO_REFRESH_AT = 0; + private final long expiresAtEpochMillis; + private final long refreshAtEpochMillis; + private final String token; + + /** + * @param token the token, without the {@code "Bearer "} prefix + * @param expiresAtEpochMillis the absolute expiry, in milliseconds since the Unix epoch + * @throws IllegalArgumentException if the token is null, blank or not printable ASCII + */ + public ExpiringToken(String token, long expiresAtEpochMillis) { + this(token, expiresAtEpochMillis, NO_REFRESH_AT); + } + + /** + * @param token the token, without the {@code "Bearer "} prefix + * @param expiresAtEpochMillis the absolute expiry, in milliseconds since the Unix epoch + * @param refreshAtEpochMillis the earliest instant the platform suggests refreshing, or + * {@link #NO_REFRESH_AT} (any value {@code <= 0}) for none + * @throws IllegalArgumentException if the token is null, blank or not printable ASCII + */ + public ExpiringToken(String token, long expiresAtEpochMillis, long refreshAtEpochMillis) { + String problem = describeInvalidToken(token); + if (problem != null) { + throw new IllegalArgumentException(problem); + } + this.token = token; + this.expiresAtEpochMillis = expiresAtEpochMillis; + this.refreshAtEpochMillis = refreshAtEpochMillis > 0 ? refreshAtEpochMillis : NO_REFRESH_AT; + } + + /** + * Applies the token rules of the specification's section 3: non-null, not blank, and every character within + * printable ASCII. Returns a token-free description of the first violation, or {@code null} when the token + * is acceptable. + */ + static String describeInvalidToken(CharSequence token) { + if (token == null || token.length() == 0) { + return "token is null or empty"; + } + boolean blank = true; + for (int i = 0, n = token.length(); i < n; i++) { + char c = token.charAt(i); + if (c < 0x20 || c > 0x7e) { + return "token contains a control or non-ASCII character"; + } + if (c != ' ') { + blank = false; + } + } + return blank ? "token is blank" : null; + } + + private static String instant(long epochMillis) { + try { + return Instant.ofEpochMilli(epochMillis).toString(); + } catch (RuntimeException e) { + return Long.toString(epochMillis); + } + } + + /** + * @return the absolute expiry, in milliseconds since the Unix epoch + */ + public long getExpiresAtEpochMillis() { + return expiresAtEpochMillis; + } + + /** + * @return the platform's earliest suggested refresh instant, or {@link #NO_REFRESH_AT} when none was given + */ + public long getRefreshAtEpochMillis() { + return refreshAtEpochMillis; + } + + /** + * @return the token, without the {@code "Bearer "} prefix + */ + public String getToken() { + return token; + } + + /** + * @return true when the platform gave a refresh hint + */ + public boolean hasRefreshAt() { + return refreshAtEpochMillis != NO_REFRESH_AT; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("ExpiringToken{token=") + .append(CredentialRedaction.describeToken(token)) + .append(", expiresAt=").append(instant(expiresAtEpochMillis)); + if (hasRefreshAt()) { + sb.append(", refreshAt=").append(instant(refreshAtEpochMillis)); + } + return sb.append('}').toString(); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java new file mode 100644 index 000000000..796eb3aea --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -0,0 +1,1027 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.HttpTokenProvider; +import io.questdb.client.std.Chars; +import io.questdb.client.std.QuietCloseable; +import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.time.Instant; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.ThreadLocalRandom; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.DoubleSupplier; + +/** + * A shared, proactively refreshing cache in front of a {@link TokenSource}: the refreshing provider of the + * dynamic-credential specification (design/qwp-token-provider-spec.md, section 5). Hand one instance to + * {@code Sender.builder(...).httpTokenProvider(...)}, {@code QwpQueryClient.withBearerTokenProvider(...)} or + * {@code QuestDB.builder().httpTokenProvider(...)}; every connection then reads the cached token, and the + * provider fetches a new one in the background well before the current one expires. + * 

{@code
+ * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
+ * TokenRequestContext scope = new TokenRequestContext().addScopes("api:///.default");
+ * RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> {
+ *     AccessToken t = credential.getTokenSync(scope);
+ *     return new ExpiringToken(t.getToken(), t.getExpiresAt().toInstant().toEpochMilli());
+ * }).build();
+ * try (QuestDB db = QuestDB.connect("wss::addr=qdb1:9000,qdb2:9000;", tokens)) {
+ *     ...
+ * } finally {
+ *     tokens.close();
+ * }
+ * }
+ *

Behaviour

+ *
    + *
  • Prefetch. The first fetch starts as soon as the provider is built, on the provider's own daemon + * thread. {@link #awaitReady(long)} lets an application gate startup on it.
    • + *
  • Proactive refresh. After a fetch received at {@code f} with expiry {@code e}, the next fetch is + * scheduled at a jittered half-life, {@code f + (e - f) * U(0.45, 0.55)}, no later than + * {@code e - min(refresh_margin, (e - f) / 2)}, no earlier than {@code f + min(min_refresh_interval, + * (e - f) / 2)}, and no later than the platform's refresh hint when it gave one. While the source is healthy, + * even an idle client therefore always holds a usable token.
    • + *
  • Hand-out. A token is usable only while {@code now < e - handout_floor}. {@link #getToken()} + * returns a usable token at once - a volatile read, no I/O, no lock. With no usable token it starts a fetch + * (unless one is running or the provider is backing off after failures) and waits up to {@code cold_wait}, + * then throws a {@link TokenUnavailableException} carrying the classification of the most recent failed + * fetch, or a retryable one when no fetch failed. It never hands out an unusable token. Any number of + * concurrent callers cause at most one fetch.
    • + *
  • Failures. A failed fetch is retried with jittered exponential backoff, from + * {@code backoff_initial} up to {@code backoff_max}, honouring the source's {@code retry_after} up to + * {@code retry_after_max}. Retries continue whatever the classification for as long as the provider is open, + * the current token keeps being served while it remains usable, and failures are logged as warnings at most + * once a minute, with only the classification and a sanitized message.
    • + *
  • Forced refresh. {@link #onTokenRejected(CharSequence, int)} with {@code 401} for the current + * token starts a fetch, at most once per {@code forced_min_interval}, and waits up to {@code forced_wait} for + * it. A stale token (one that has already rotated) and any other status are ignored.
    • + *
  • Interrupts. A caller interrupted while waiting gets a retryable {@link TokenUnavailableException} + * promptly, with its interrupt flag still set.
    • + *
  • Close. {@link #close()} stops the refresher and wakes every waiter; afterwards + * {@link #getToken()} fails with a permanent error. Clients never close a provider the application supplied: + * close it yourself once every client using it is closed.
    • + *
  • Clocks. Expiry comparisons use the wall clock; delays and waits use a monotonic clock.
    • + *
+ * Nothing here ever renders the token: {@link #toString()}, log lines and error messages show at most its + * length and an 8-hex-digit SHA-256 fingerprint. + */ +public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable { + public static final long DEFAULT_BACKOFF_INITIAL_MILLIS = 500; + public static final long DEFAULT_BACKOFF_MAX_MILLIS = 60_000; + public static final long DEFAULT_COLD_WAIT_MILLIS = 30_000; + public static final long DEFAULT_FORCED_MIN_INTERVAL_MILLIS = 30_000; + public static final long DEFAULT_FORCED_WAIT_MILLIS = 5_000; + public static final long DEFAULT_HANDOUT_FLOOR_MILLIS = 60_000; + public static final long DEFAULT_MIN_REFRESH_INTERVAL_MILLIS = 30_000; + public static final long DEFAULT_REFRESH_MARGIN_MILLIS = 300_000; + public static final long DEFAULT_RETRY_AFTER_MAX_MILLIS = 300_000; + /** + * Value returned by the observability getters when there is nothing to report. + */ + public static final long NONE = -1; + private static final long CLOSE_AWAIT_MILLIS = 1_000; + private static final long FAILURE_LOG_INTERVAL_NANOS = TimeUnit.MINUTES.toNanos(1); + private static final Logger LOG = LoggerFactory.getLogger(RefreshingTokenProvider.class); + // Delays are bookkept as monotonic deadlines compared by subtraction, which is only sound while every + // difference stays below 2^63. Clamping each delay to 2^61 ns (~73 years) keeps that true for any token + // lifetime a source can report, including Long.MAX_VALUE "never expires". + private static final long MAX_DELAY_NANOS = Long.MAX_VALUE >> 2; + private static final AtomicInteger THREAD_IDS = new AtomicInteger(); + // A waiter re-checks its deadline at least this often when the clock is not the system clock, so a test + // that advances a fake clock is noticed. With the system clock the waits are untimed beyond their deadline. + private static final long WAIT_SLICE_NANOS = TimeUnit.MILLISECONDS.toNanos(20); + private final long backoffInitialMillis; + private final long backoffMaxMillis; + private final Clock clock; + private final long coldWaitMillis; + private final long forcedMinIntervalNanos; + private final long forcedWaitNanos; + private final long handoutFloorMillis; + private final ReentrantLock lock = new ReentrantLock(); + private final long maxWaitSliceNanos; + private final long minRefreshIntervalMillis; + private final String name; + private final DoubleSupplier random; + private final long refreshMarginMillis; + private final long retryAfterMaxMillis; + private final Scheduler scheduler; + private final TokenSource source; + // Signalled whenever a fetch completes and on close. + private final Condition stateChanged = lock.newCondition(); + // ---- guarded by lock ---- + private boolean backoffActive; + private long backoffUntilNanos; + private volatile boolean closed; + private long completedFetches; + private volatile int consecutiveFailures; + private boolean failureLogged; + private volatile boolean fetchRunning; + private boolean forcedRefreshed; + // The current token, or null. Immutable snapshot, published with a volatile write so the warm path in + // getToken() needs no lock. + private volatile Held held; + private volatile FetchFailure lastFailure; + private long lastFailureLogNanos; + private long lastForcedNanos; + private volatile long lastSuccessEpochMillis = NONE; + private long scheduledDueNanos; + private long scheduledSeq; + private ScheduledTask scheduledTask; + + private RefreshingTokenProvider(Builder b) { + this.source = b.source; + this.name = b.name; + this.clock = b.clock; + this.maxWaitSliceNanos = b.clock == SystemClock.INSTANCE ? Long.MAX_VALUE : WAIT_SLICE_NANOS; + this.scheduler = b.scheduler != null ? b.scheduler : new DefaultScheduler(); + this.random = b.random; + this.refreshMarginMillis = b.refreshMarginMillis; + this.minRefreshIntervalMillis = b.minRefreshIntervalMillis; + this.handoutFloorMillis = b.handoutFloorMillis; + this.coldWaitMillis = b.coldWaitMillis; + this.backoffInitialMillis = b.backoffInitialMillis; + this.backoffMaxMillis = b.backoffMaxMillis; + this.retryAfterMaxMillis = b.retryAfterMaxMillis; + this.forcedMinIntervalNanos = TimeUnit.MILLISECONDS.toNanos(b.forcedMinIntervalMillis); + this.forcedWaitNanos = TimeUnit.MILLISECONDS.toNanos(b.forcedWaitMillis); + } + + /** + * Starts building a provider around {@code source}. + * + * @param source obtains new tokens; called only from the provider's refresher thread + * @return a builder whose defaults are the specification's + */ + public static Builder builder(TokenSource source) { + if (source == null) { + throw new IllegalArgumentException("token source must not be null"); + } + return new Builder(source); + } + + /** + * The refresh instant of the specification's section 5.2 for a token received at {@code f} that expires at + * {@code e}, given a uniform sample {@code u} in {@code [0, 1)}. Pure; exposed for the conformance tests. + */ + @TestOnly + public static long computeRefreshAtMillis( + long f, + long e, + long refreshAt, + double u, + long refreshMarginMillis, + long minRefreshIntervalMillis + ) { + final long life = e - f; + final long half = life / 2; + final long margin = Math.min(refreshMarginMillis, half); + final long floor = Math.min(minRefreshIntervalMillis, half); + final double fraction = 0.45 + 0.10 * Math.max(0.0, Math.min(1.0, u)); + long r = f + (long) (life * fraction); + r = Math.min(r, e - margin); + r = Math.max(r, f + floor); + if (refreshAt > 0) { + r = Math.min(r, Math.max(refreshAt, f + floor)); + } + return r; + } + + private static String instant(long epochMillis) { + try { + return Instant.ofEpochMilli(epochMillis).toString(); + } catch (RuntimeException e) { + return Long.toString(epochMillis); + } + } + + private static long millisToNanos(long millis) { + return Math.min(TimeUnit.MILLISECONDS.toNanos(Math.max(0, millis)), MAX_DELAY_NANOS); + } + + /** + * Waits until the provider holds a usable token, starting a fetch if none is running and the provider is + * not backing off. Use it to gate application startup on the first fetch. + * + * @param timeoutMillis the longest to wait + * @return true once a usable token is held; false on timeout, when the provider is closed, or when the + * calling thread is interrupted (its interrupt flag is then left set) + */ + public boolean awaitReady(long timeoutMillis) { + lock.lock(); + try { + final long start = clock.monotonicNanos(); + final long deadline = start + millisToNanos(timeoutMillis); + if (!usable(held, clock.wallClockMillis())) { + requestFetchNowLocked(start); + } + while (true) { + if (closed) { + return false; + } + if (usable(held, clock.wallClockMillis())) { + return true; + } + final long remaining = deadline - clock.monotonicNanos(); + if (remaining <= 0) { + return false; + } + try { + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return false; + } + } + } finally { + lock.unlock(); + } + } + + /** + * Stops scheduled fetches, interrupts a fetch in progress and wakes every waiter. Afterwards + * {@link #getToken()} fails with a permanent {@link TokenUnavailableException}. Idempotent. Waits at most + * one second for the refresher thread to exit. + */ + @Override + public void close() { + lock.lock(); + try { + if (closed) { + return; + } + closed = true; + cancelScheduledLocked(); + held = null; + stateChanged.signalAll(); + } finally { + lock.unlock(); + } + scheduler.shutdown(); + LOG.debug("token provider {} closed", name); + } + + /** + * @return the number of consecutive failed fetches since the last successful one + */ + public int getConsecutiveFailures() { + return consecutiveFailures; + } + + /** + * The most recent failed fetch - its classification, sanitized message and time - or null when no fetch + * has failed. Kept after the source recovers. + */ + public FetchFailure getLastFailure() { + return lastFailure; + } + + /** + * @return the wall-clock time of the last successful fetch, or {@link #NONE} + */ + public long getLastSuccessEpochMillis() { + return lastSuccessEpochMillis; + } + + /** + * @return the non-secret label this provider uses in log lines and errors + */ + public String getName() { + return name; + } + + /** + * Returns a usable token: at once when one is held, otherwise after waiting up to {@code cold_wait} for a + * fetch. + * + * @return the current token, without the {@code "Bearer "} prefix + * @throws TokenUnavailableException when no usable token arrives in time (classified as the most recent + * failed fetch, or retryable when none failed), when the calling thread + * is interrupted (retryable; the interrupt flag stays set), or when the + * provider is closed (permanent) + */ + @Override + public CharSequence getToken() { + if (closed) { + throw closedException(); + } + final Held h = held; + if (h != null) { + final long wallNow = clock.wallClockMillis(); + if (usable(h, wallNow)) { + // Warm path: a volatile read, no I/O and no lock. Only when the refresh is overdue - its + // scheduled fetch has not run, say after the host slept through it - does a caller take the + // lock, to start that fetch in the background. + if (!fetchRunning && consecutiveFailures == 0 + && (wallNow >= h.refreshAtEpochMillis || clock.monotonicNanos() - h.refreshAtNanos >= 0)) { + startOverdueRefresh(); + } + return h.token; + } + } + return awaitUsableToken(); + } + + /** + * @return the current token's expiry, or {@link #NONE} when no token is held + */ + public long getTokenExpiresAtEpochMillis() { + final Held h = held; + return h == null ? NONE : h.expiresAtEpochMillis; + } + + /** + * @return true once {@link #close()} has been called + */ + public boolean isClosed() { + return closed; + } + + /** + * Forced refresh. When a server answered {@code 401} to {@code token} and that token is still the current + * one, starts a fetch now - joining one already running, and respecting failure backoff - and waits up to + * {@code forced_wait} for it. Ignored for any other status, for a token that has already rotated, and when + * a forced refresh ran less than {@code forced_min_interval} ago. A forced fetch that returns the same token + * counts as a normal success. Never throws; returns promptly when interrupted, with the flag still set. + */ + @Override + public void onTokenRejected(CharSequence token, int httpStatus) { + if (httpStatus != 401 || token == null) { + return; + } + lock.lock(); + try { + if (closed) { + return; + } + final Held h = held; + if (h == null || !Chars.equals(h.token, token)) { + return; // stale: the token has already rotated, so the caller's next pull gets the new one + } + final long now = clock.monotonicNanos(); + if (forcedRefreshed && now - lastForcedNanos < forcedMinIntervalNanos) { + return; + } + forcedRefreshed = true; + lastForcedNanos = now; + LOG.info("token provider {}: the server rejected the current token with 401, refreshing early", name); + final long target = completedFetches + 1; + requestFetchNowLocked(now); + final long deadline = now + forcedWaitNanos; + while (!closed && completedFetches < target) { + final long remaining = deadline - clock.monotonicNanos(); + if (remaining <= 0) { + break; + } + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + lock.unlock(); + } + } + + @Override + public String toString() { + final Held h = held; + final FetchFailure f = lastFailure; + final StringBuilder sb = new StringBuilder("RefreshingTokenProvider{name=").append(name); + if (h == null) { + sb.append(", token="); + } else { + sb.append(", token=").append(CredentialRedaction.describeToken(h.token)) + .append(", expiresAt=").append(instant(h.expiresAtEpochMillis)) + .append(", refreshAt=").append(instant(h.refreshAtEpochMillis)); + } + final long success = lastSuccessEpochMillis; + if (success != NONE) { + sb.append(", lastSuccess=").append(instant(success)); + } + sb.append(", consecutiveFailures=").append(consecutiveFailures); + if (f != null) { + sb.append(", lastFailure=").append(f); + } + if (closed) { + sb.append(", closed"); + } + return sb.append('}').toString(); + } + + private CharSequence awaitUsableToken() { + lock.lock(); + try { + final long start = clock.monotonicNanos(); + final long deadline = start + millisToNanos(coldWaitMillis); + // A caller that arrives without a usable token starts one fetch - unless one is running or the + // provider is backing off - and then waits for it or for the fetches already scheduled. It does not + // start another each time a fetch completes: a source that keeps returning a token inside the + // hand-out floor would otherwise be fetched in a tight loop for the whole wait. + if (!usable(held, clock.wallClockMillis())) { + requestFetchNowLocked(start); + } + while (true) { + if (closed) { + throw closedException(); + } + final Held h = held; + if (usable(h, clock.wallClockMillis())) { + return h.token; + } + final long now = clock.monotonicNanos(); + // The earliest permitted fetch lies beyond the caller's deadline: nothing can arrive in time, + // so say so now rather than parking the caller for nothing. + if (!fetchRunning && inBackoffLocked(now) && backoffUntilNanos - deadline > 0) { + throw unavailableLocked(now, start); + } + final long remaining = deadline - now; + if (remaining <= 0) { + throw unavailableLocked(now, start); + } + try { + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new TokenUnavailableException("interrupted while waiting for a token from " + name, true); + } + } + } finally { + lock.unlock(); + } + } + + private long backoffMillis(int failures, long retryAfterMillis) { + final int shift = Math.min(failures - 1, 40); + long base = backoffInitialMillis << shift; + if (base < 0 || (base >> shift) != backoffInitialMillis) { + base = backoffMaxMillis; // overflowed + } + base = Math.min(base, backoffMaxMillis); + final long half = base / 2; + long delay = half + (long) (half * uniform()); + if (retryAfterMillis >= 0) { + delay = Math.max(delay, Math.min(retryAfterMillis, retryAfterMaxMillis)); + } + return delay; + } + + private void cancelScheduledLocked() { + scheduledSeq++; + final ScheduledTask t = scheduledTask; + scheduledTask = null; + if (t != null) { + t.cancel(); + } + } + + private FetchFailure checkResult(ExpiringToken result, long receivedAt) { + if (result == null) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned null", receivedAt); + } + final String problem = ExpiringToken.describeInvalidToken(result.getToken()); + if (problem != null) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned an unusable token: " + problem, receivedAt); + } + if (result.getExpiresAtEpochMillis() <= receivedAt) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned a token that expired at " + instant(result.getExpiresAtEpochMillis()) + + ", not after it was received at " + instant(receivedAt), + receivedAt); + } + return null; + } + + private FetchFailure classify(Throwable t, long receivedAt) { + if (t instanceof TokenUnavailableException) { + final TokenUnavailableException e = (TokenUnavailableException) t; + return new FetchFailure(e.isRetryable(), e.getRetryAfterMillis(), e.getMessage(), receivedAt); + } + // An unclassified failure is retryable (the specification's rule when unsure). Name its type: the + // message alone ("connect timed out") often does not say what failed. + final String message = t.getMessage(); + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + message == null ? t.getClass().getName() : t.getClass().getName() + ": " + message, receivedAt); + } + + private TokenUnavailableException closedException() { + return new TokenUnavailableException("token provider " + name + " is closed", false); + } + + private boolean inBackoffLocked(long now) { + return backoffActive && now - backoffUntilNanos < 0; + } + + private String onFailureLocked(FetchFailure failure, long receivedNanos) { + final int failures = consecutiveFailures + 1; + consecutiveFailures = failures; + lastFailure = failure; + final long delayMillis = backoffMillis(failures, failure.retryAfterMillis); + final long delayNanos = millisToNanos(delayMillis); + backoffActive = true; + backoffUntilNanos = receivedNanos + delayNanos; + scheduleFetchLocked(delayNanos, receivedNanos); + if (failureLogged && receivedNanos - lastFailureLogNanos < FAILURE_LOG_INTERVAL_NANOS) { + return null; + } + failureLogged = true; + lastFailureLogNanos = receivedNanos; + return "token provider " + name + ": fetch failed [retryable=" + failure.retryable + + ", consecutiveFailures=" + failures + ", retryInMillis=" + delayMillis + "]: " + failure.message; + } + + private void onSuccessLocked(ExpiringToken result, long receivedAt, long receivedNanos) { + final long refreshAt = computeRefreshAtMillis( + receivedAt, + result.getExpiresAtEpochMillis(), + result.getRefreshAtEpochMillis(), + uniform(), + refreshMarginMillis, + minRefreshIntervalMillis + ); + final long delayNanos = millisToNanos(refreshAt - receivedAt); + final int failures = consecutiveFailures; + held = new Held(result.getToken(), result.getExpiresAtEpochMillis(), refreshAt, receivedNanos + delayNanos); + lastSuccessEpochMillis = receivedAt; + consecutiveFailures = 0; + backoffActive = false; + scheduleFetchLocked(delayNanos, receivedNanos); + if (failures > 0) { + LOG.info("token provider {}: fetch succeeded after {} consecutive failure(s) [token={}, expiresAt={}]", + name, failures, CredentialRedaction.describeToken(result.getToken()), + instant(result.getExpiresAtEpochMillis())); + } else if (LOG.isDebugEnabled()) { + LOG.debug("token provider {}: fetched [token={}, expiresAt={}, refreshAt={}]", + name, CredentialRedaction.describeToken(result.getToken()), + instant(result.getExpiresAtEpochMillis()), instant(refreshAt)); + } + } + + // Starts a fetch now unless one is running, one is already due, or the provider is in failure backoff. + private void requestFetchNowLocked(long now) { + if (closed || fetchRunning || inBackoffLocked(now)) { + return; + } + if (scheduledTask != null && now - scheduledDueNanos >= 0) { + return; // already due: the scheduler runs it as soon as it can + } + scheduleFetchLocked(0, now); + } + + private void runFetch(long seq) { + lock.lock(); + try { + if (closed || seq != scheduledSeq || fetchRunning) { + return; // cancelled, superseded, or a fetch is already in flight + } + scheduledTask = null; + fetchRunning = true; + } finally { + lock.unlock(); + } + ExpiringToken result = null; + Throwable error = null; + try { + result = source.fetchToken(); + } catch (Throwable t) { + // Every failure, an Error included, is a failed fetch to retry with backoff: rethrowing would only + // be swallowed by the scheduler and leave the provider with no fetch scheduled, ever again. + error = t; + } + final long receivedAt = clock.wallClockMillis(); + final long receivedNanos = clock.monotonicNanos(); + FetchFailure failure = error != null ? classify(error, receivedAt) : checkResult(result, receivedAt); + if (failure != null) { + failure = failure.sanitized(); + } + String warning = null; + lock.lock(); + try { + fetchRunning = false; + completedFetches++; + if (!closed) { + if (failure == null) { + onSuccessLocked(result, receivedAt, receivedNanos); + } else { + warning = onFailureLocked(failure, receivedNanos); + } + } + stateChanged.signalAll(); + } finally { + lock.unlock(); + } + if (warning != null) { + LOG.warn(warning); + } + } + + private void scheduleFetchLocked(long delayNanos, long now) { + if (closed) { + return; + } + cancelScheduledLocked(); + final long seq = scheduledSeq; + scheduledDueNanos = now + delayNanos; + scheduledTask = scheduler.schedule(() -> runFetch(seq), delayNanos); + } + + private void startOverdueRefresh() { + lock.lock(); + try { + requestFetchNowLocked(clock.monotonicNanos()); + } finally { + lock.unlock(); + } + } + + private double uniform() { + return random.getAsDouble(); + } + + private TokenUnavailableException unavailableLocked(long now, long start) { + final FetchFailure f = consecutiveFailures > 0 ? lastFailure : null; + final long waitedMillis = TimeUnit.NANOSECONDS.toMillis(Math.max(0, now - start)); + if (f == null) { + return new TokenUnavailableException("no usable token from " + name + " after waiting " + + waitedMillis + " ms", true); + } + final long retryAfter = backoffActive && backoffUntilNanos - now > 0 + ? TimeUnit.NANOSECONDS.toMillis(backoffUntilNanos - now) + : TokenUnavailableException.NO_RETRY_AFTER; + return new TokenUnavailableException("no usable token from " + name + " after waiting " + waitedMillis + + " ms [retryable=" + f.retryable + ", consecutiveFailures=" + consecutiveFailures + "]: " + + f.message, f.retryable, retryAfter); + } + + private boolean usable(Held h, long wallNow) { + return h != null && wallNow < h.expiresAtEpochMillis - handoutFloorMillis; + } + + /** + * Time source. Expiry comparisons use {@link #wallClockMillis()}; delays and waits use + * {@link #monotonicNanos()}. + */ + public interface Clock { + long monotonicNanos(); + + long wallClockMillis(); + } + + /** + * Runs fetches on the provider's background context. The default owns one daemon thread. A test may supply + * its own to run fetches deterministically. + */ + public interface Scheduler { + /** + * Runs {@code task} once, after {@code delayNanos} on the monotonic clock. + */ + ScheduledTask schedule(Runnable task, long delayNanos); + + /** + * Called once from {@link RefreshingTokenProvider#close()}: stop running tasks and interrupt a task in + * progress. + */ + void shutdown(); + } + + /** + * Handle to a task submitted to a {@link Scheduler}. + */ + public interface ScheduledTask { + /** + * Best-effort: a task that has already started may still run, and the provider ignores it. + */ + void cancel(); + } + + /** + * Builder for {@link RefreshingTokenProvider}. The defaults are the specification's (section 5.1). + */ + public static final class Builder { + private final TokenSource source; + private long backoffInitialMillis = DEFAULT_BACKOFF_INITIAL_MILLIS; + private long backoffMaxMillis = DEFAULT_BACKOFF_MAX_MILLIS; + private Clock clock = SystemClock.INSTANCE; + private long coldWaitMillis = DEFAULT_COLD_WAIT_MILLIS; + private long forcedMinIntervalMillis = DEFAULT_FORCED_MIN_INTERVAL_MILLIS; + private long forcedWaitMillis = DEFAULT_FORCED_WAIT_MILLIS; + private long handoutFloorMillis = DEFAULT_HANDOUT_FLOOR_MILLIS; + private long minRefreshIntervalMillis = DEFAULT_MIN_REFRESH_INTERVAL_MILLIS; + private String name = "token-provider"; + private DoubleSupplier random = () -> ThreadLocalRandom.current().nextDouble(); + private long refreshMarginMillis = DEFAULT_REFRESH_MARGIN_MILLIS; + private long retryAfterMaxMillis = DEFAULT_RETRY_AFTER_MAX_MILLIS; + private Scheduler scheduler; + + private Builder(TokenSource source) { + this.source = source; + } + + private static long nonNegative(String name, long value) { + if (value < 0) { + throw new IllegalArgumentException(name + " must be >= 0: " + value); + } + return value; + } + + /** + * First retry delay after a failed fetch. Default 500 ms. + */ + public Builder backoffInitialMillis(long millis) { + if (millis <= 0) { + throw new IllegalArgumentException("backoff_initial must be > 0: " + millis); + } + this.backoffInitialMillis = millis; + return this; + } + + /** + * Largest retry delay after failed fetches. Default 60 s. + */ + public Builder backoffMaxMillis(long millis) { + if (millis <= 0) { + throw new IllegalArgumentException("backoff_max must be > 0: " + millis); + } + this.backoffMaxMillis = millis; + return this; + } + + /** + * Builds the provider and starts its first fetch in the background. + */ + public RefreshingTokenProvider build() { + if (backoffMaxMillis < backoffInitialMillis) { + throw new IllegalArgumentException("backoff_max must be >= backoff_initial [backoff_max=" + + backoffMaxMillis + ", backoff_initial=" + backoffInitialMillis + ']'); + } + RefreshingTokenProvider provider = new RefreshingTokenProvider(this); + provider.lock.lock(); + try { + provider.scheduleFetchLocked(0, provider.clock.monotonicNanos()); + } finally { + provider.lock.unlock(); + } + return provider; + } + + /** + * Test seam: the clock for expiry comparisons, delays and waits. + */ + @TestOnly + public Builder clock(Clock clock) { + if (clock == null) { + throw new IllegalArgumentException("clock must not be null"); + } + this.clock = clock; + return this; + } + + /** + * The longest {@code getToken()} waits when no usable token is held. Default 30 s. + */ + public Builder coldWaitMillis(long millis) { + this.coldWaitMillis = nonNegative("cold_wait", millis); + return this; + } + + /** + * Minimum spacing between forced refreshes. Default 30 s. + */ + public Builder forcedMinIntervalMillis(long millis) { + this.forcedMinIntervalMillis = nonNegative("forced_min_interval", millis); + return this; + } + + /** + * The longest {@code onTokenRejected} waits for a forced refresh. Default 5 s. + */ + public Builder forcedWaitMillis(long millis) { + this.forcedWaitMillis = nonNegative("forced_wait", millis); + return this; + } + + /** + * A token is usable only while {@code now < expires_at - handout_floor}. Default 60 s. + */ + public Builder handoutFloorMillis(long millis) { + this.handoutFloorMillis = nonNegative("handout_floor", millis); + return this; + } + + /** + * Lower bound on the time between a fetch and the next scheduled refresh. Default 30 s. + */ + public Builder minRefreshIntervalMillis(long millis) { + this.minRefreshIntervalMillis = nonNegative("min_refresh_interval", millis); + return this; + } + + /** + * A non-secret label for log lines, errors and the refresher thread, e.g. + * {@code azure[resource=api://...]}. Default {@code token-provider}. + */ + public Builder name(String name) { + if (name == null || name.isEmpty()) { + throw new IllegalArgumentException("name must not be empty"); + } + this.name = name; + return this; + } + + /** + * Test seam: the source of uniform samples in {@code [0, 1)} for the refresh and backoff jitter. + */ + @TestOnly + public Builder random(DoubleSupplier uniform) { + if (uniform == null) { + throw new IllegalArgumentException("random must not be null"); + } + this.random = uniform; + return this; + } + + /** + * Upper bound on how close to expiry a scheduled refresh may run. Default 5 min. + */ + public Builder refreshMarginMillis(long millis) { + this.refreshMarginMillis = nonNegative("refresh_margin", millis); + return this; + } + + /** + * Cap applied to a source's {@code retry_after}. Default 5 min. + */ + public Builder retryAfterMaxMillis(long millis) { + this.retryAfterMaxMillis = nonNegative("retry_after_max", millis); + return this; + } + + /** + * Test seam: where fetches run. The provider calls {@link Scheduler#shutdown()} from {@code close()}. + */ + @TestOnly + public Builder scheduler(Scheduler scheduler) { + if (scheduler == null) { + throw new IllegalArgumentException("scheduler must not be null"); + } + this.scheduler = scheduler; + return this; + } + } + + /** + * A failed fetch: its classification, sanitized message and time. Never carries the token. + */ + public static final class FetchFailure { + private final long epochMillis; + private final String message; + private final long retryAfterMillis; + private final boolean retryable; + + FetchFailure(boolean retryable, long retryAfterMillis, String message, long epochMillis) { + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? TokenUnavailableException.NO_RETRY_AFTER : retryAfterMillis; + this.message = message; + this.epochMillis = epochMillis; + } + + /** + * @return the wall-clock time of the failure + */ + public long getEpochMillis() { + return epochMillis; + } + + /** + * @return the failure message, sanitized and at most 256 characters + */ + public String getMessage() { + return message; + } + + /** + * @return the source's suggested wait before retrying, or {@link TokenUnavailableException#NO_RETRY_AFTER} + */ + public long getRetryAfterMillis() { + return retryAfterMillis; + } + + /** + * @return whether the source classified the failure as retryable + */ + public boolean isRetryable() { + return retryable; + } + + @Override + public String toString() { + return "FetchFailure{retryable=" + retryable + ", at=" + instant(epochMillis) + ", message=" + message + '}'; + } + + FetchFailure sanitized() { + final String clean = CredentialRedaction.sanitizeErrorText(message); + return new FetchFailure(retryable, retryAfterMillis, clean == null || clean.isEmpty() + ? "token source failed without a message" : clean, epochMillis); + } + } + + private static final class DefaultScheduler implements Scheduler { + private final ScheduledThreadPoolExecutor executor; + private volatile Thread thread; + + DefaultScheduler() { + final String threadName = "qdb-token-refresh-" + THREAD_IDS.incrementAndGet(); + this.executor = new ScheduledThreadPoolExecutor(1, r -> { + Thread t = new Thread(r, threadName); + t.setDaemon(true); + thread = t; + return t; + }); + executor.setRemoveOnCancelPolicy(true); + executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false); + executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false); + } + + @Override + public ScheduledTask schedule(Runnable task, long delayNanos) { + final ScheduledFuture future = executor.schedule(task, delayNanos, TimeUnit.NANOSECONDS); + return () -> future.cancel(false); + } + + @Override + public void shutdown() { + executor.shutdownNow(); + if (Thread.currentThread() == thread) { + return; // closed from inside a fetch: the thread exits once this task unwinds + } + // Interrupt-neutral, like the other close paths in this library: a carried interrupt flag would + // turn the bounded wait into an immediate return. + final boolean interrupted = Thread.interrupted(); + try { + executor.awaitTermination(CLOSE_AWAIT_MILLIS, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + } + + private static final class Held { + final long expiresAtEpochMillis; + final long refreshAtEpochMillis; + final long refreshAtNanos; + final String token; + + Held(String token, long expiresAtEpochMillis, long refreshAtEpochMillis, long refreshAtNanos) { + this.token = token; + this.expiresAtEpochMillis = expiresAtEpochMillis; + this.refreshAtEpochMillis = refreshAtEpochMillis; + this.refreshAtNanos = refreshAtNanos; + } + } + + private static final class SystemClock implements Clock { + static final SystemClock INSTANCE = new SystemClock(); + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis(); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java new file mode 100644 index 000000000..2e96e1fd0 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java @@ -0,0 +1,83 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.util.Map; + +/** + * Service-provider interface behind the {@code token_provider} connect-string key (design/qwp-token-provider-spec.md, + * section 7). An implementation is found through {@link java.util.ServiceLoader}: list it in + * {@code META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory}, or declare it with + * {@code provides} in a module descriptor. The optional {@code questdb-client-azure} artifact supplies the + * {@code azure} factory this way. + *

+ * A factory turns the provider-specific connect-string keys into a {@link TokenSource}. The client wraps that + * source in a {@link RefreshingTokenProvider} that the process-wide {@link TokenProviderRegistry} shares between + * every client built from an equivalent connect string, so a process runs one refresher per configuration. + *

+ * The connect-string vocabulary is fixed - unknown keys are rejected before a factory ever sees them - so a + * factory receives only the keys the client defines for it, already normalized: for {@code azure}, the + * {@code azure_resource} (with any trailing {@code /.default} removed) and the optional, lower-cased + * {@code azure_client_id}. Values in these keys are never secret: secrets come from the platform's standard + * configuration, never from the connect string. + */ +public interface TokenProviderFactory { + + /** + * Creates the token source for one configuration. Called once per distinct configuration, when the first + * client using it connects. It must not perform network I/O: the provider's refresher thread calls + * {@link TokenSource#fetchToken()} later. + * + * @param params the provider-specific keys, normalized; unmodifiable + * @return the token source + * @throws IllegalArgumentException when the parameters cannot be used + */ + TokenSource createSource(Map params); + + /** + * A non-secret label for log lines and errors, e.g. {@code azure[resource=api://...]}. + * + * @param params the provider-specific keys, normalized + * @return the label + */ + default String describe(Map params) { + return params.isEmpty() ? name() : name() + params; + } + + /** + * The value of {@code token_provider} this factory serves, such as {@code azure}. + */ + String name(); + + /** + * Validates the provider-specific keys when a connect string is parsed, before any client connects. Must be + * pure: it runs during configuration validation that must not fetch a token (section 7.2). + * + * @param params the provider-specific keys, normalized; unmodifiable + * @throws IllegalArgumentException naming the offending key + */ + default void validate(Map params) { + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java new file mode 100644 index 000000000..7c55a6f74 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java @@ -0,0 +1,302 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.QuietCloseable; +import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.ServiceConfigurationError; +import java.util.ServiceLoader; +import java.util.TreeSet; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Process-wide registry of the token providers that connect strings select with {@code token_provider} + * (design/qwp-token-provider-spec.md, section 7.4). Every client instance built from a connect string - each + * sender, each query client, each pooled connection - acquires a {@link Lease} when it connects and releases it + * when it closes. All leases for the same configuration - the provider name plus its normalized parameters - + * share one {@link RefreshingTokenProvider}, so the process runs one refresher, and makes at most one call to + * the identity platform at a time, per configuration. + *

+ * When the last lease is released the provider stays alive for the registry's linger period (60 s for the + * global registry) and is closed only if no new lease arrives in the meantime: a pool that recycles its + * connections, or an application that closes one sender and opens another, keeps the warm cache. + *

+ * Providers that an application supplies itself never pass through here; sharing those is the application's + * responsibility. + */ +public final class TokenProviderRegistry { + public static final long DEFAULT_LINGER_MILLIS = 60_000; + private static final TokenProviderRegistry GLOBAL = new TokenProviderRegistry(DEFAULT_LINGER_MILLIS); + private static final Logger LOG = LoggerFactory.getLogger(TokenProviderRegistry.class); + @TestOnly + private static volatile List factoriesForTesting; + private final Map entries = new HashMap<>(); + private final long lingerMillis; + private final Object lock = new Object(); + private ScheduledThreadPoolExecutor lingerTimer; + + /** + * Creates a private registry. Production code uses {@link #global()}; a separate instance is for tests and + * for applications that want to scope provider sharing themselves. + * + * @param lingerMillis how long a provider outlives its last lease; {@code <= 0} closes it at once + */ + public TokenProviderRegistry(long lingerMillis) { + this.lingerMillis = lingerMillis; + } + + /** + * Finds the factory serving {@code name} through {@link ServiceLoader}, trying the thread context class + * loader first and then the loader of this library. + * + * @return the factory, or null when none is installed + */ + public static TokenProviderFactory findFactory(String name) { + for (TokenProviderFactory factory : factories()) { + if (name.equals(factory.name())) { + return factory; + } + } + return null; + } + + /** + * The registry every client built from a connect string uses. + */ + public static TokenProviderRegistry global() { + return GLOBAL; + } + + /** + * Test seam: replaces {@link ServiceLoader} discovery with a fixed list of factories. Pass null to restore + * discovery. + */ + @TestOnly + public static void setFactoriesForTesting(List factories) { + factoriesForTesting = factories; + } + + /** + * The names of every installed factory, sorted: the values {@code token_provider} accepts. + */ + public static List supportedProviders() { + TreeSet names = new TreeSet<>(); + for (TokenProviderFactory factory : factories()) { + names.add(factory.name()); + } + return new ArrayList<>(names); + } + + private static List factories() { + final List override = factoriesForTesting; + if (override != null) { + return override; + } + final List found = new ArrayList<>(); + final ClassLoader context = Thread.currentThread().getContextClassLoader(); + load(context == null ? ServiceLoader.load(TokenProviderFactory.class) + : ServiceLoader.load(TokenProviderFactory.class, context), found); + final ClassLoader own = TokenProviderFactory.class.getClassLoader(); + if (own != context) { + load(ServiceLoader.load(TokenProviderFactory.class, own), found); + } + return found; + } + + private static void load(ServiceLoader loader, List into) { + final Iterator it = loader.iterator(); + while (true) { + final TokenProviderFactory factory; + try { + if (!it.hasNext()) { + return; + } + factory = it.next(); + } catch (ServiceConfigurationError e) { + // one broken provider must not hide the others + LOG.warn("could not load a token provider factory: {}", + CredentialRedaction.sanitizeErrorText(String.valueOf(e.getMessage()))); + continue; + } + boolean duplicate = false; + for (int i = 0, n = into.size(); i < n; i++) { + if (into.get(i).getClass() == factory.getClass()) { + duplicate = true; + break; + } + } + if (!duplicate) { + into.add(factory); + } + } + } + + /** + * Acquires a lease on the shared provider for {@code spec}, creating the provider - and starting its first + * fetch - when none is alive. Cancels a pending linger close. + * + * @throws IllegalArgumentException when the factory rejects the parameters + */ + public Lease acquire(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + if (entry == null || entry.provider.isClosed()) { + TokenSource source = spec.factory().createSource(spec.params()); + if (source == null) { + throw new IllegalStateException("token provider factory " + spec.name() + " returned no token source"); + } + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).name(spec.describe()).build(); + entry = new Entry(spec.registryKey(), provider); + entries.put(entry.key, entry); + LOG.info("started token provider {}", entry.provider.getName()); + } + entry.refs++; + if (entry.lingerTask != null) { + entry.lingerTask.cancel(false); + entry.lingerTask = null; + } + return new Lease(entry); + } + } + + /** + * Whether a provider for {@code spec} is alive - leased, or lingering after its last lease. + */ + public boolean isActive(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + return entry != null && !entry.provider.isClosed(); + } + } + + /** + * Number of leases currently held on the provider for {@code spec}. + */ + public int leaseCount(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + return entry == null ? 0 : entry.refs; + } + } + + private void expire(Entry entry) { + synchronized (lock) { + if (entry.refs > 0 || entries.get(entry.key) != entry) { + return; // re-leased during the linger, or already replaced + } + entries.remove(entry.key); + entry.lingerTask = null; + } + LOG.info("closing token provider {}: no client has used it for {} ms", entry.provider.getName(), lingerMillis); + entry.provider.close(); + } + + private ScheduledThreadPoolExecutor lingerTimer() { + if (lingerTimer == null) { + lingerTimer = new ScheduledThreadPoolExecutor(1, r -> { + Thread t = new Thread(r, "qdb-token-provider-registry"); + t.setDaemon(true); + return t; + }); + lingerTimer.setRemoveOnCancelPolicy(true); + lingerTimer.setKeepAliveTime(5, TimeUnit.SECONDS); + lingerTimer.allowCoreThreadTimeOut(true); + } + return lingerTimer; + } + + private void release(Entry entry) { + RefreshingTokenProvider toClose = null; + synchronized (lock) { + if (--entry.refs > 0) { + return; + } + if (lingerMillis <= 0) { + entries.remove(entry.key, entry); + toClose = entry.provider; + } else { + entry.lingerTask = lingerTimer().schedule(() -> expire(entry), lingerMillis, TimeUnit.MILLISECONDS); + } + } + if (toClose != null) { + toClose.close(); + } + } + + private static final class Entry { + final String key; + final RefreshingTokenProvider provider; + ScheduledFuture lingerTask; + int refs; + + Entry(String key, RefreshingTokenProvider provider) { + this.key = key; + this.provider = provider; + } + } + + /** + * One client's hold on a shared provider. Release it exactly when the client closes; releasing twice is + * harmless. + */ + public final class Lease implements QuietCloseable { + private final Entry entry; + private final AtomicBoolean released = new AtomicBoolean(); + + private Lease(Entry entry) { + this.entry = entry; + } + + @Override + public void close() { + if (released.compareAndSet(false, true)) { + release(entry); + } + } + + /** + * The shared provider. Do not close it: release the lease instead. + */ + public RefreshingTokenProvider provider() { + return entry.provider; + } + + @Override + public String toString() { + return "TokenProviderRegistry.Lease{" + entry.provider.getName() + (released.get() ? ", released}" : "}"); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java new file mode 100644 index 000000000..82e09019e --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java @@ -0,0 +1,248 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.impl.ConfigView; + +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.TreeMap; + +/** + * A validated {@code token_provider} selection from a {@code ws}/{@code wss} connect string: the provider name, + * its normalized parameters, and the factory that serves it (design/qwp-token-provider-spec.md, section 7). + *

+ * {@link #parse(ConfigView, boolean)} enforces every rule of section 7.2 and never fetches a token: it only + * looks the factory up. A client turns the spec into a provider when it connects, through + * {@link TokenProviderRegistry#acquire(TokenProviderSpec)}. + * 

+ * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_client_id=<uami-client-id>;
+ *
+ */ +public final class TokenProviderSpec { + /** + * The {@code token_provider} value of the Microsoft Entra ID provider, served by the optional + * {@code questdb-client-azure} module. + */ + public static final String AZURE = "azure"; + /** + * Reserved for a zero-dependency Azure IMDS provider (decision D5); not implemented. + */ + public static final String AZURE_IMDS = "azure_imds"; + public static final String AZURE_MODULE = "org.questdb:questdb-client-azure"; + public static final String KEY_AZURE_CLIENT_ID = "azure_client_id"; + public static final String KEY_AZURE_RESOURCE = "azure_resource"; + public static final String KEY_TOKEN_PROVIDER = "token_provider"; + private static final String AZURE_DEFAULT_SCOPE_SUFFIX = "/.default"; + private static final String[] STATIC_CREDENTIAL_KEYS = {"token", "username", "password"}; + private final TokenProviderFactory factory; + private final String name; + private final Map params; + private final String registryKey; + + private TokenProviderSpec(TokenProviderFactory factory, String name, Map params) { + this.factory = factory; + this.name = name; + this.params = Collections.unmodifiableMap(params); + StringBuilder key = new StringBuilder(name); + for (Map.Entry e : params.entrySet()) { + key.append('|').append(e.getKey()).append('=').append(e.getValue()); + } + this.registryKey = key.toString(); + } + + /** + * Validates and resolves the {@code token_provider} keys of a {@code ws}/{@code wss} connect string. Never + * fetches a token and never starts a provider. Rejects, naming the offending key: + *
    + *
  • {@code token_provider} combined with {@code token}, {@code username} or {@code password};
    • + *
  • an empty, unknown, reserved or uninstalled {@code token_provider}, listing the supported values;
    • + *
  • a provider-specific key the selected provider does not accept, such as {@code azure_resource} + * without {@code token_provider=azure};
    • + *
  • a missing required provider key;
    • + *
  • {@code token_provider} on {@code ws::}: the bearer token would cross the network in cleartext + * (decision D4).
    • + *
+ * Combining {@code token_provider} with a provider the application supplies is rejected by the builders. + * + * @param view the parsed connect string + * @param tls true for the {@code wss} schema + * @return the spec, or null when the string selects no provider + * @throws IllegalArgumentException on any violation + */ + public static TokenProviderSpec parse(ConfigView view, boolean tls) { + final String name = view.getStr(KEY_TOKEN_PROVIDER); + final String resource = view.getStr(KEY_AZURE_RESOURCE); + final String clientId = view.getStr(KEY_AZURE_CLIENT_ID); + if (name == null) { + if (resource != null) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + if (clientId != null) { + throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + return null; + } + for (String key : STATIC_CREDENTIAL_KEYS) { + if (view.has(key)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " cannot be combined with " + key); + } + } + if (name.trim().isEmpty()) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " must not be empty; " + supportedValues()); + } + if (!tls) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " requires the wss:: schema: over ws:: the bearer " + + "token would cross the network in cleartext"); + } + if (AZURE_IMDS.equals(name)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE_IMDS + + " is reserved and not implemented by this client; " + supportedValues()); + } + final TokenProviderFactory factory = TokenProviderRegistry.findFactory(name); + if (factory == null) { + if (AZURE.equals(name)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE + " requires the " + AZURE_MODULE + + " module on the class path or module path; " + supportedValues()); + } + throw new IllegalArgumentException("unsupported " + KEY_TOKEN_PROVIDER + ": " + safe(name) + "; " + + supportedValues()); + } + if (!AZURE.equals(name)) { + if (resource != null) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + if (clientId != null) { + throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + } + final Map params = new TreeMap<>(); + if (AZURE.equals(name)) { + if (resource == null) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE + " requires " + KEY_AZURE_RESOURCE); + } + params.put(KEY_AZURE_RESOURCE, normalizeAzureResource(resource)); + if (clientId != null) { + params.put(KEY_AZURE_CLIENT_ID, normalizeGuid(clientId)); + } + } + factory.validate(Collections.unmodifiableMap(params)); + return new TokenProviderSpec(factory, name, params); + } + + /** + * Lower-cases a GUID after checking its {@code 8-4-4-4-12} hex shape. + */ + static String normalizeGuid(String value) { + if (value.length() != 36) { + throw invalidClientId(value); + } + for (int i = 0; i < 36; i++) { + char c = value.charAt(i); + boolean dash = i == 8 || i == 13 || i == 18 || i == 23; + if (dash ? c != '-' : Character.digit(c, 16) < 0) { + throw invalidClientId(value); + } + } + return value.toLowerCase(Locale.ROOT); + } + + private static IllegalArgumentException invalidClientId(String value) { + return new IllegalArgumentException("invalid " + KEY_AZURE_CLIENT_ID + + ": expected a GUID such as 00000000-0000-0000-0000-000000000000, got " + safe(value)); + } + + // The application ID URI (api://) or the client ID of the QuestDB app registration. A trailing + // "/.default" is stripped: the client always requests /.default. + private static String normalizeAzureResource(String value) { + String resource = value; + if (resource.endsWith(AZURE_DEFAULT_SCOPE_SUFFIX)) { + resource = resource.substring(0, resource.length() - AZURE_DEFAULT_SCOPE_SUFFIX.length()); + } + if (resource.isEmpty()) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " must not be empty"); + } + for (int i = 0, n = resource.length(); i < n; i++) { + char c = resource.charAt(i); + if (c <= 0x20 || c > 0x7e) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + + " must be an application ID URI (api://) or a client ID without spaces or " + + "non-ASCII characters, got " + safe(resource)); + } + } + return resource; + } + + private static String safe(String value) { + return CredentialRedaction.sanitizeErrorText(value); + } + + private static String supportedValues() { + List names = TokenProviderRegistry.supportedProviders(); + if (names.isEmpty()) { + return "supported values: none installed (token_provider=azure needs " + AZURE_MODULE + ')'; + } + return "supported values: " + names; + } + + /** + * A non-secret label for log lines and errors. + */ + public String describe() { + return factory.describe(params); + } + + public TokenProviderFactory factory() { + return factory; + } + + /** + * The provider name, the value of {@code token_provider}. + */ + public String name() { + return name; + } + + /** + * The normalized provider-specific keys; unmodifiable. + */ + public Map params() { + return params; + } + + /** + * The registry key: the provider name plus the normalized parameters. Equivalent connect strings share it. + */ + public String registryKey() { + return registryKey; + } + + @Override + public String toString() { + return "TokenProviderSpec{" + registryKey + '}'; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java new file mode 100644 index 000000000..31575460d --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java @@ -0,0 +1,63 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +/** + * Obtains a new bearer token from an identity platform - Microsoft Entra ID, an OAuth client-credentials + * endpoint, a secrets manager - together with its expiry. A {@link RefreshingTokenProvider} wraps a source and + * caches its tokens, refreshing them in the background before they expire. This is the token-source contract of + * the dynamic-credential specification (design/qwp-token-provider-spec.md, section 4). + *

+ * Threading. The provider calls {@link #fetchToken()} only from its own background refresher thread, never from + * a connection thread, and never concurrently with itself. The call may block, but it should bound its own + * network operations; 30 seconds per attempt is recommended. A provider being closed interrupts its refresher, + * so a source blocked in an interruptible wait should let the interrupt end it. + *

+ * Failures. Throw {@link TokenUnavailableException} to classify a failure: + *

    + *
  • retryable - network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. Pass the platform's + * {@code Retry-After}, when it gave one, as {@code retryAfterMillis};
    • + *
  • permanent - configuration that is missing or wrong: no credential configured, identity not found, + * invalid client.
    • + *
+ * When unsure, report the failure as retryable. Any other exception is treated as retryable. The provider keeps + * retrying either way - an operator can repair a "permanent" failure, such as a missing role assignment, + * without restarting the application - but the classification decides how a caller waiting for a token reacts. + *

+ * Secrets. A failure message must not contain a token or a raw response body: report only the shape of a + * problem, such as a missing or invalid field. Never attach an object that serializes a raw HTTP request, such + * as a client-credentials body carrying a {@code client_secret}, to an exception. + */ +@FunctionalInterface +public interface TokenSource { + + /** + * Obtains a new token from the identity platform. + * + * @return the token and its expiry; never null + * @throws TokenUnavailableException to report a classified failure; any other exception counts as retryable + */ + ExpiringToken fetchToken(); +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java new file mode 100644 index 000000000..6e08754b8 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java @@ -0,0 +1,128 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.cutlass.line.LineSenderException; + +/** + * No token could be obtained. This is the {@code TokenError} of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 4) and also its {@code token-unavailable} error. + *

    + *
  • A {@link TokenSource} throws it to classify a failed fetch.
    • + *
  • A {@link RefreshingTokenProvider} throws it from {@code getToken()} when it holds no usable token and + * cannot get one in time. It then carries the classification of the provider's most recent failed fetch, or + * is retryable when no fetch has failed (the wait timed out, or the caller was interrupted).
    • + *
+ * {@link #isRetryable()} tells a caller whether trying again can succeed. A permanent failure is configuration + * that is missing or wrong; a retryable one is a network failure, a timeout, throttling or a server error. + * {@link #getRetryAfterMillis()} is the platform's suggested wait before the next attempt, or + * {@link #NO_RETRY_AFTER}. + *

+ * Connection code treats an exception from an application-supplied token provider by type: only a + * {@code TokenUnavailableException} marked retryable is retried while a {@code Sender} with + * {@code initial_connect_retry=on} starts up; anything else fails startup fast (decision D8). + *

+ * The message must never contain a token or a raw response body. + */ +public class TokenUnavailableException extends LineSenderException { + /** + * Value of {@link #getRetryAfterMillis()} meaning the platform suggested no wait. + */ + public static final long NO_RETRY_AFTER = -1; + private final long retryAfterMillis; + private final boolean retryable; + + /** + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + */ + public TokenUnavailableException(CharSequence message, boolean retryable) { + this(message, retryable, NO_RETRY_AFTER); + } + + /** + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + * @param retryAfterMillis the platform's suggested wait before retrying, or {@link #NO_RETRY_AFTER} + */ + public TokenUnavailableException(CharSequence message, boolean retryable, long retryAfterMillis) { + super(message, retryable); + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? NO_RETRY_AFTER : retryAfterMillis; + } + + /** + * Carries a cause. Attach only a cause that is safe to log: never one whose message holds a token or a raw + * response body, and never an object that serializes a raw HTTP request. + * + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + * @param retryAfterMillis the platform's suggested wait before retrying, or {@link #NO_RETRY_AFTER} + * @param cause the underlying failure, may be null + */ + public TokenUnavailableException(String message, boolean retryable, long retryAfterMillis, Throwable cause) { + super(message, cause); + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? NO_RETRY_AFTER : retryAfterMillis; + } + + /** + * A permanent failure: configuration that is missing or wrong. + */ + public static TokenUnavailableException permanent(CharSequence message) { + return new TokenUnavailableException(message, false); + } + + /** + * A retryable failure without a suggested wait. + */ + public static TokenUnavailableException retryable(CharSequence message) { + return new TokenUnavailableException(message, true); + } + + /** + * A retryable failure with the platform's suggested wait, such as an HTTP {@code Retry-After}. + */ + public static TokenUnavailableException retryable(CharSequence message, long retryAfterMillis) { + return new TokenUnavailableException(message, true, retryAfterMillis); + } + + /** + * @return the platform's suggested wait before the next attempt, in milliseconds, or + * {@link #NO_RETRY_AFTER} when it gave none + */ + public long getRetryAfterMillis() { + return retryAfterMillis; + } + + /** + * @return true when retrying can succeed (a network failure, timeout, throttling or server error), false for + * configuration that is missing or wrong + */ + @Override + public boolean isRetryable() { + return retryable; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java new file mode 100644 index 000000000..914f97f00 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java @@ -0,0 +1,158 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.http; + +/** + * Reads the {@code error} parameter of a {@code Bearer} challenge in a {@code WWW-Authenticate} header + * (RFC 6750, section 3; RFC 7235, section 4.1). Clients use it to decide whether a {@code 401} is worth a token + * refresh: {@code invalid_token} is, {@code insufficient_scope} is not. + *

+ * The parser is tolerant. It walks a comma-separated list of challenges - {@code scheme [token68 | auth-param, + * ...]} - accepts quoted and unquoted parameter values, and ignores what it cannot read. Only the first + * {@code error} parameter of a {@code Bearer} challenge counts. + */ +public final class BearerChallenge { + /** + * The {@code error} value that means the presented token itself is bad, so a new one can help. + */ + public static final String INVALID_TOKEN = "invalid_token"; + + private BearerChallenge() { + } + + /** + * @param wwwAuthenticate the value of one or more {@code WWW-Authenticate} headers (joined with commas), or + * null + * @return the {@code error} parameter of the first {@code Bearer} challenge that carries one, or null when + * there is no such challenge + */ + public static String bearerError(CharSequence wwwAuthenticate) { + if (wwwAuthenticate == null) { + return null; + } + final int n = wwwAuthenticate.length(); + boolean inBearer = false; + int i = 0; + while (i < n) { + char c = wwwAuthenticate.charAt(i); + if (c == ',' || isWhitespace(c)) { + i++; + continue; + } + final int start = i; + while (i < n && isTokenChar(wwwAuthenticate.charAt(i))) { + i++; + } + if (i == start) { + i++; // not a token character: skip it + continue; + } + final String token = wwwAuthenticate.subSequence(start, i).toString(); + int j = skipWhitespace(wwwAuthenticate, i); + if (j < n && wwwAuthenticate.charAt(j) == '=') { + // auth-param: name = ( token / quoted-string ) + j = skipWhitespace(wwwAuthenticate, j + 1); + final String value; + if (j < n && wwwAuthenticate.charAt(j) == '"') { + final StringBuilder sb = new StringBuilder(); + j++; + while (j < n) { + char q = wwwAuthenticate.charAt(j++); + if (q == '\\' && j < n) { + sb.append(wwwAuthenticate.charAt(j++)); + } else if (q == '"') { + break; + } else { + sb.append(q); + } + } + value = sb.toString(); + } else { + final int valueStart = j; + while (j < n && wwwAuthenticate.charAt(j) != ',' && !isWhitespace(wwwAuthenticate.charAt(j))) { + j++; + } + value = wwwAuthenticate.subSequence(valueStart, j).toString(); + } + if (inBearer && "error".equalsIgnoreCase(token)) { + return value; + } + i = j; + } else { + // a new challenge's auth-scheme (or a token68, which carries no parameters) + inBearer = "Bearer".equalsIgnoreCase(token); + i = j; + } + } + return null; + } + + /** + * Whether a {@code 401} carrying {@code wwwAuthenticate} warrants refreshing the token: true when the header + * has no {@code Bearer} challenge with an {@code error} parameter (the status code alone decides), or when + * that error is {@code invalid_token}. + */ + public static boolean allowsTokenRefresh(CharSequence wwwAuthenticate) { + final String error = bearerError(wwwAuthenticate); + return error == null || INVALID_TOKEN.equals(error); + } + + private static boolean isTokenChar(char c) { + if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) { + return true; + } + switch (c) { + case '!': + case '#': + case '$': + case '%': + case '&': + case '\'': + case '*': + case '+': + case '-': + case '.': + case '^': + case '_': + case '`': + case '|': + case '~': + return true; + default: + return false; + } + } + + private static boolean isWhitespace(char c) { + return c == ' ' || c == '\t'; + } + + private static int skipWhitespace(CharSequence s, int i) { + while (i < s.length() && isWhitespace(s.charAt(i))) { + i++; + } + return i; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java b/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java index 1290fcab5..8c8d07bd8 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java @@ -163,6 +163,11 @@ public abstract class WebSocketClient implements QuietCloseable { private int serverNegotiatedZstdLevel; private int serverQwpVersion = 1; private String upgradeRejectRole; + // WWW-Authenticate value(s) from the most recent rejected upgrade, joined with ", " when the server sent + // several, or null when absent. Lets a 401 carrying "Bearer error=insufficient_scope" be told apart from an + // expired token, so a dynamic credential is refreshed only when a new token can help. Reset to null on every + // upgrade() invocation. + private String upgradeRejectWwwAuthenticate; // Server-advertised zone identifier from the most recent rejected upgrade, // captured from the X-QuestDB-Zone response header on a 421. Null when the // header was absent or empty. Per failover.md §5 servers SHOULD emit this @@ -374,6 +379,14 @@ public String getUpgradeRejectRole() { return upgradeRejectRole; } + /** + * {@code WWW-Authenticate} value(s) on the most recent rejected upgrade, joined with {@code ", "} when the + * server sent several, or null when absent. + */ + public String getUpgradeRejectWwwAuthenticate() { + return upgradeRejectWwwAuthenticate; + } + /** * Zone identifier from {@code X-QuestDB-Zone} on the most recent rejected * upgrade, or null when the header was absent or empty (after trimming). @@ -648,6 +661,7 @@ public void upgrade(CharSequence path, int timeout, CharSequence authorizationHe return; // Already upgraded } upgradeRejectRole = null; + upgradeRejectWwwAuthenticate = null; upgradeRejectZone = null; upgradeStatusCode = 0; @@ -878,6 +892,34 @@ private static String extractRoleHeader(String response) { return null; } + // Every value of the named header, joined with ", " (RFC 7230 section 3.2.2), or null when absent. + private static String extractHeaderValues(String response, String headerName) { + int headerLen = headerName.length(); + int responseLen = response.length(); + StringBuilder values = null; + int lineStart = response.indexOf("\r\n"); + while (lineStart >= 0 && lineStart + 2 + headerLen <= responseLen) { + int hStart = lineStart + 2; + if (response.regionMatches(true, hStart, headerName, 0, headerLen)) { + int valueStart = hStart + headerLen; + int lineEnd = response.indexOf('\r', valueStart); + if (lineEnd < 0) { + lineEnd = responseLen; + } + String value = response.substring(valueStart, lineEnd).trim(); + if (!value.isEmpty()) { + if (values == null) { + values = new StringBuilder(value); + } else { + values.append(", ").append(value); + } + } + } + lineStart = response.indexOf("\r\n", hStart); + } + return values == null ? null : values.toString(); + } + private static String extractZoneHeader(String response) { int headerLen = QUESTDB_ZONE_HEADER_NAME.length(); int responseLen = response.length(); @@ -1346,6 +1388,9 @@ private void validateUpgradeResponse(int headerEnd) { if (!response.startsWith("HTTP/1.1 101")) { String statusLine = response.split("\r\n")[0]; upgradeStatusCode = parseStatusCode(statusLine); + if (upgradeStatusCode == 401) { + upgradeRejectWwwAuthenticate = extractHeaderValues(response, "WWW-Authenticate:"); + } if (upgradeStatusCode == 421) { upgradeRejectRole = extractRoleHeader(response); upgradeRejectZone = extractZoneHeader(response); diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java index d2a5714a2..e14c3a7da 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java @@ -24,26 +24,61 @@ package io.questdb.client.cutlass.qwp.client; +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.http.BearerChallenge; import io.questdb.client.cutlass.http.client.HttpClientException; /** - * WebSocket upgrade rejected with {@code 401} or {@code 403}. Terminal across all - * configured endpoints: a rejected credential is uniformly rejected across the - * cluster, so failing fast surfaces the configuration error immediately. Path - * mismatches ({@code 404}) are NOT routed through this exception because a single - * misconfigured node mid-deploy can return 404 while peers are healthy. + * WebSocket upgrade rejected with {@code 401} or {@code 403}: the {@code auth-rejected} failure class of the + * dynamic-credential specification (design/qwp-token-provider-spec.md, section 8.1). The message starts with + * {@code auth-rejected} so every report of it names the class. + *

+ * A credential is uniformly accepted or rejected across a cluster, so the connect walk stops at the first such + * rejection instead of trying the remaining endpoints. What happens next depends on the phase (section 8.3): + * startup fails, an established store-and-forward sender keeps retrying, an orphan drainer rides a rotating + * credential out before quarantining, and a query fails. Before any of that, a {@code 401} against a dynamic + * credential earns one same-endpoint retry with a refreshed token (section 8.2) - see + * {@link #isTokenRefreshable()}. Path mismatches ({@code 404}) are NOT routed through this exception because a + * single misconfigured node mid-deploy can return 404 while peers are healthy. */ public final class QwpAuthFailedException extends HttpClientException { + private static final int MAX_BEARER_ERROR_LENGTH = 64; + private final String bearerError; private final String host; private final int port; private final int statusCode; public QwpAuthFailedException(int statusCode, String host, int port) { - super("WebSocket upgrade rejected with HTTP "); - put(statusCode).put(" for ").put(host).put(':').put(port); + this(statusCode, host, port, null); + } + + /** + * @param wwwAuthenticate the {@code WWW-Authenticate} header value(s) of the rejection, or null + */ + public QwpAuthFailedException(int statusCode, String host, int port, String wwwAuthenticate) { + super("auth-rejected: WebSocket upgrade rejected with HTTP "); + put(statusCode); + String error = BearerChallenge.bearerError(wwwAuthenticate); + if (error != null) { + error = CredentialRedaction.sanitizeErrorText(error); + if (error.length() > MAX_BEARER_ERROR_LENGTH) { + error = error.substring(0, MAX_BEARER_ERROR_LENGTH); + } + put(" [error=").put(error).put(']'); + } + put(" for ").put(host).put(':').put(port); this.statusCode = statusCode; this.host = host; this.port = port; + this.bearerError = error; + } + + /** + * The {@code error} parameter of the rejection's {@code WWW-Authenticate: Bearer} challenge (for example + * {@code invalid_token} or {@code insufficient_scope}), sanitized, or null when the server sent none. + */ + public String getBearerError() { + return bearerError; } public String getHost() { @@ -57,4 +92,13 @@ public int getPort() { public int getStatusCode() { return statusCode; } + + /** + * Whether a new token can help: the status is {@code 401}, and the server either sent no {@code Bearer} + * challenge with an {@code error} or sent {@code error="invalid_token"} (RFC 6750, section 3.1). A + * {@code 403} never qualifies - it is an authorization decision a new token does not change (decision D2). + */ + public boolean isTokenRefreshable() { + return statusCode == 401 && (bearerError == null || BearerChallenge.INVALID_TOKEN.equals(bearerError)); + } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java new file mode 100644 index 000000000..a26b681b4 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java @@ -0,0 +1,182 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client; + +import io.questdb.client.ConnectionHealth; +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; +import io.questdb.client.cutlass.line.LineSenderException; + +/** + * Tracks one QWP client's connection health (design/qwp-token-provider-spec.md, section 8.4) and publishes it as + * an immutable {@link ConnectionHealth} snapshot. Connection code reports transitions - a successful upgrade, a + * lost connection, a failed connect round, a terminal failure, close - from whichever thread observes them; + * readers get the last published snapshot with a single volatile read and never wait on connection threads. + * Writers serialize on a private monitor that is never held across I/O. + */ +public final class QwpConnectionHealthTracker { + private static final int MAX_CAUSE_DEPTH = 16; + private final Object lock = new Object(); + private long failedRounds; + private ConnectionHealth.Failure lastFailure; + private long lastConnectedAt = ConnectionHealth.NONE; + private long outageSince; + private volatile ConnectionHealth snapshot; + private ConnectionHealth.State state = ConnectionHealth.State.CONNECTING; + + public QwpConnectionHealthTracker() { + // A client that has never connected has been without a connection since it was created. + this.outageSince = System.currentTimeMillis(); + publish(); + } + + /** + * Classifies a failed connect round's exception into the failure classes of section 8.4. + */ + public static ConnectionHealth.Failure classify(Throwable failure, long nowMillis) { + Throwable t = failure; + for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof QwpCredentialUnavailableException) { + return failure(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, 0, t, nowMillis); + } + if (t instanceof QwpAuthFailedException) { + return failure(ConnectionHealth.FailureClass.AUTH_REJECTED, + ((QwpAuthFailedException) t).getStatusCode(), t, nowMillis); + } + if (t instanceof QwpRoleMismatchException || t instanceof QwpIngressRoleRejectedException) { + return failure(ConnectionHealth.FailureClass.ROLE_REJECTED, 0, t, nowMillis); + } + if (t instanceof WebSocketUpgradeException) { + WebSocketUpgradeException e = (WebSocketUpgradeException) t; + return e.isRoleMismatch() + ? failure(ConnectionHealth.FailureClass.ROLE_REJECTED, 0, t, nowMillis) + : failure(ConnectionHealth.FailureClass.OTHER, Math.max(0, e.getStatusCode()), t, nowMillis); + } + if (t instanceof QwpVersionMismatchException || t instanceof QwpDurableAckMismatchException) { + return failure(ConnectionHealth.FailureClass.OTHER, 0, t, nowMillis); + } + } + return failure(failure instanceof HttpClientException || failure instanceof LineSenderException + ? ConnectionHealth.FailureClass.TRANSPORT + : ConnectionHealth.FailureClass.OTHER, 0, failure, nowMillis); + } + + private static ConnectionHealth.Failure failure( + ConnectionHealth.FailureClass failureClass, + int statusCode, + Throwable t, + long nowMillis + ) { + String message = CredentialRedaction.sanitizeErrorText(t.getMessage()); + return new ConnectionHealth.Failure(failureClass, statusCode, + message == null || message.isEmpty() ? t.getClass().getSimpleName() : message, nowMillis); + } + + /** + * The client was closed. Sticky. + */ + public void closed() { + synchronized (lock) { + state = ConnectionHealth.State.CLOSED; + publish(); + } + } + + /** + * An established connection was lost and is being re-established. + */ + public void connectionLost() { + synchronized (lock) { + if (state == ConnectionHealth.State.CONNECTED) { + state = ConnectionHealth.State.RECONNECTING; + outageSince = System.currentTimeMillis(); + failedRounds = 0; + publish(); + } + } + } + + /** + * The client will not connect again. Sticky; keeps the last connect-round failure. + */ + public void failed() { + synchronized (lock) { + if (state != ConnectionHealth.State.CLOSED) { + state = ConnectionHealth.State.FAILED; + publish(); + } + } + } + + /** + * A connect round - one walk over the configured endpoints - ended without a connection. + */ + public void roundFailed(Throwable failure) { + final long now = System.currentTimeMillis(); + final ConnectionHealth.Failure classified = classify(failure, now); + synchronized (lock) { + if (state == ConnectionHealth.State.FAILED || state == ConnectionHealth.State.CLOSED) { + return; + } + if (state == ConnectionHealth.State.CONNECTED) { + state = ConnectionHealth.State.RECONNECTING; + outageSince = now; + failedRounds = 0; + } + failedRounds++; + lastFailure = classified; + publish(); + } + } + + /** + * @return the current snapshot; never null + */ + public ConnectionHealth snapshot() { + return snapshot; + } + + /** + * A connect round ended with a successful upgrade. + */ + public void upgraded() { + synchronized (lock) { + if (state == ConnectionHealth.State.FAILED || state == ConnectionHealth.State.CLOSED) { + return; + } + state = ConnectionHealth.State.CONNECTED; + lastConnectedAt = System.currentTimeMillis(); + outageSince = ConnectionHealth.NONE; + failedRounds = 0; + publish(); + } + } + + // caller holds lock + private void publish() { + snapshot = new ConnectionHealth(state, lastConnectedAt, outageSince, failedRounds, lastFailure); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java index 2652a7f79..963119527 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java @@ -24,23 +24,28 @@ package io.questdb.client.cutlass.qwp.client; +import io.questdb.client.cutlass.auth.TokenUnavailableException; import io.questdb.client.cutlass.line.LineSenderException; /** * Signals that the client could not OBTAIN an Authorization credential for a * (re)connect handshake: the configured {@code httpTokenProvider} threw instead of - * returning a token -- a failed silent refresh, or no sign-in yet. + * returning a token -- a failed silent refresh, or no sign-in yet. This is the + * {@code credential-unavailable} failure class of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 8.1). *

* Distinct from {@link QwpAuthFailedException}, which means the server rejected a - * credential the client did present (a terminal auth failure). A credential the client - * cannot ACQUIRE is instead handled by connection phase, exactly like a transport outage: + * credential the client did present. A credential the client cannot ACQUIRE is instead + * handled by connection phase, exactly like a transport outage: * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under * Invariant B -- the IdP becomes reachable again, or the user completes an interactive * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer - * store-and-forward promised to keep alive). Only the foreground/SYNC initial connect - * fails fast, because a connectivity error is the caller's to see during initialization, - * not after the drainer is running. + * store-and-forward promised to keep alive). During initialization the classification + * decides: {@link #isRetryable()} is true only when the provider threw a + * {@link TokenUnavailableException} marked retryable, which a SYNC initial connect keeps + * retrying within its budget; any other provider failure is permanent and fails startup + * fast (decision D8), as does any failure on an OFF initial connect. *

* It exists so the send loop can tell "the provider failed" apart from "the network * failed", and it carries the provider's own exception so a handler can surface that @@ -52,7 +57,9 @@ * connect in {@code QwpWebSocketSender} - both catch it and rethrow * {@link #providerFailure()}, so a token-provider failure reaches the caller as the * provider's own exception; the running background drainer catches it and retries under - * the invariant above. It is public because both of those packages handle it, and + * the invariant above. The query client ({@code QwpQueryClient}) does throw it from + * {@code connect()}, with a message that starts with {@code credential-unavailable}. It is + * public because both of those packages handle it, and * because {@code QwpWebSocketSender.newReconnectFactory()} is public: a caller that * drives {@code ReconnectFactory.reconnect()} itself runs the endpoint walk directly and * so can receive this type unwrapped. Such a caller should treat it as the provider @@ -63,12 +70,30 @@ public class QwpCredentialUnavailableException extends LineSenderException { private final RuntimeException providerFailure; public QwpCredentialUnavailableException(RuntimeException providerFailure) { - super(providerFailure.getMessage() == null + this(providerFailure.getMessage() == null ? "token provider failed to supply a credential" : providerFailure.getMessage(), providerFailure); + } + + /** + * @param message the message, which must not contain a token + * @param providerFailure the exception the token provider threw + */ + public QwpCredentialUnavailableException(String message, RuntimeException providerFailure) { + super(message, providerFailure); this.providerFailure = providerFailure; } + /** + * True only when the provider threw a {@link TokenUnavailableException} marked retryable. Any other + * exception from a provider counts as a permanent credential failure (decision D8), so that startup fails + * fast, as it does for an OIDC device-flow provider that is not signed in yet. + */ + @Override + public boolean isRetryable() { + return providerFailure instanceof TokenUnavailableException && ((TokenUnavailableException) providerFailure).isRetryable(); + } + /** * The exception the token provider threw, for a caller that must surface the * provider's own error rather than this wrapper. Never null: the wrapper is only diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index bfc924e52..380fc0eae 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -25,12 +25,14 @@ package io.questdb.client.cutlass.qwp.client; import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.http.client.WebSocketClientFactory; import io.questdb.client.cutlass.http.client.WebSocketFrameHandler; -import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.impl.ConfigString; import io.questdb.client.impl.ConfigView; @@ -151,6 +153,8 @@ public class QwpQueryClient implements QuietCloseable { * is already in the client's kernel recv buffer by the time this wait starts. */ private static final int DEFAULT_SERVER_INFO_TIMEOUT_MS = 5_000; + // Warn once per process when an application-supplied token provider is used over ws:: (spec section 9). + private static final AtomicBoolean CLEARTEXT_PROVIDER_WARNED = new AtomicBoolean(); private static final Logger LOG = LoggerFactory.getLogger(QwpQueryClient.class); // Reusable typed bind-value sink. Populated on the user thread by the // {@link QwpBindSetter} passed to execute(); the pre-encoded bytes are @@ -166,6 +170,8 @@ public class QwpQueryClient implements QuietCloseable { private final List endpoints = new ArrayList<>(); private final AtomicBoolean executing = new AtomicBoolean(); private final Random failoverRandom = new Random(); + // Connection health (design/qwp-token-provider-spec.md, section 8.4). + private final QwpConnectionHealthTracker healthTracker = new QwpConnectionHealthTracker(); private long authTimeoutMs = DEFAULT_AUTH_TIMEOUT_MS; private String authorizationHeader; // Deterministic lifecycle barrier used by facade shutdown tests. Null in @@ -270,6 +276,9 @@ public class QwpQueryClient implements QuietCloseable { // currentRequestId. Volatile so a cancel from any thread is visible to the // worker thread's post-requestId read. private volatile boolean pendingCancel; + // Set when a failover reconnect inside execute() failed, leaving the client disconnected: the next + // execute() reconnects instead of throwing "not connected". Cleared by every successful connect. + private volatile boolean reconnectOnNextExecute; // Decoded SERVER_INFO from the current connection's handshake. Null before // connect() has succeeded; non-null on every established connection (the // server always emits the frame). Volatile so getServerInfo(), callable @@ -300,6 +309,12 @@ public class QwpQueryClient implements QuietCloseable { // token rotation. Mutually exclusive with the fixed authorizationHeader // synthesized by withBearerToken/withBasicAuth; null when unset. private HttpTokenProvider tokenProvider; + // The registry lease behind tokenProvider when the connect string selected a token_provider. Acquired on + // the first connect() - never by fromConfig(), so building or validating a client fetches no token - and + // released by close(). + private volatile TokenProviderRegistry.Lease tokenProviderLease; + // A token_provider selected by the connect string (wss:: only); null otherwise. + private TokenProviderSpec tokenProviderSpec; private char[] trustStorePassword; private String trustStorePath; private volatile WebSocketClient webSocketClient; @@ -385,6 +400,7 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { } ConfigView view = new ConfigView(cs); validateConfig(view, tls); + TokenProviderSpec tokenProviderSpec = TokenProviderSpec.parse(view, tls); List parsedEndpoints = new ArrayList<>(); view.getHostPorts("addr", DEFAULT_WS_PORT, (h, p) -> parsedEndpoints.add(new Endpoint(h, p))); @@ -484,6 +500,7 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { } if (hasBasic) client.withBasicAuth(username, password); if (token != null) client.withBearerToken(token); + client.tokenProviderSpec = tokenProviderSpec; if (cid != null) client.withClientId(cid); if (maxBatchRows > 0) client.withMaxBatchRows(maxBatchRows); if (zone != null) client.withZone(zone); @@ -561,6 +578,9 @@ public static void validateConfig(ConfigView view, boolean tls) { if (tlsRoots != null && "unsafe_off".equals(tlsVerify)) { throw new IllegalArgumentException(TLS_ROOTS_INSECURE_CONFIG_ERROR); } + // token_provider and its keys (design/qwp-token-provider-spec.md, section 7.2): resolves the factory, never + // fetches a token. + TokenProviderSpec.parse(view, tls); // Mirror fromConfig's effective values: a missing bound takes its // default, so the ordering is enforced even when only one key is set // (e.g. failover_backoff_max_ms alone, below the default initial backoff). @@ -640,6 +660,7 @@ public void close() { // scratch, double-freeing it. return; } + healthTracker.closed(); Runnable hook = beforeCloseHook; beforeCloseHook = null; if (hook != null) { @@ -720,6 +741,13 @@ public void close() { // (submitQuery copies its bytes into sendScratch), so it is safe to free // even when we otherwise leak the I/O thread and buffer pool. bindValues.close(); + // Release the token_provider lease: the registry keeps the shared provider alive for its linger + // period, which also covers an I/O thread that failed to join above. + TokenProviderRegistry.Lease lease = tokenProviderLease; + tokenProviderLease = null; + if (lease != null) { + lease.close(); + } if (wasInterrupted) { // Hand the caller's cancellation back exactly as it arrived. Restoring it here rather // than earlier keeps it out of the joins above, which is the whole point. @@ -758,6 +786,36 @@ public synchronized void connect() { if (connected) { return; } + try { + connectWalk(); + healthTracker.upgraded(); + } catch (RuntimeException e) { + healthTracker.roundFailed(e); + throw e; + } + } + + /** + * Connection health (design/qwp-token-provider-spec.md, section 8.4). A query client connects on demand, so + * {@link ConnectionHealth.State#RECONNECTING} means its last connect or failover reconnect failed and the next + * operation will try again. Cheap and safe from any thread; never contains a credential. + */ + public ConnectionHealth health() { + return healthTracker.snapshot(); + } + + // One connect round: resolve the credential, walk the endpoints. See connect(). + private void connectWalk() { + if (tokenProviderSpec != null && tokenProviderLease == null) { + // The connect string selected a token_provider: share the process-wide provider for it (spec section + // 7.4). The lease is this client's until close(). + tokenProviderLease = TokenProviderRegistry.global().acquire(tokenProviderSpec); + tokenProvider = tokenProviderLease.provider(); + } else if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { + // spec section 9: an application-supplied provider over ws:: is the application's call - warn once + LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in cleartext; " + + "use wss:: in production"); + } lastCloseTimedOut = false; if (hostTracker == null) { hostTracker = new QwpHostHealthTracker( @@ -774,8 +832,18 @@ public synchronized void connect() { // instead of being folded into "all endpoints unreachable", and avoids re-querying the provider // once per endpoint. String authHeader = resolveAuthorizationHeader(); + // One same-endpoint retry after a refreshable 401 per connect, for a token provider only + // (design/qwp-token-provider-spec.md, section 8.2). The retried attempt records no health penalty. + boolean authRetryAvailable = tokenProvider != null; + int retryIdx = -1; while (true) { - int i = hostTracker.pickNext(); + int i; + if (retryIdx >= 0) { + i = retryIdx; + retryIdx = -1; + } else { + i = hostTracker.pickNext(); + } if (i < 0) { break; } @@ -784,6 +852,17 @@ public synchronized void connect() { connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); + if (authRetryAvailable && ae.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshAfterRejection(authHeader, ae); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("QwpQueryClient {}:{} rejected the token with {}; retrying once with a refreshed token", + ep.host, ep.port, ae.getStatusCode()); + authHeader = refreshed; + retryIdx = i; + continue; + } + } throw ae; } catch (QwpIngressRoleRejectedException re) { lastTransportError = re; @@ -819,6 +898,7 @@ public synchronized void connect() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; + reconnectOnNextExecute = false; connected = true; return; } @@ -973,6 +1053,11 @@ public java.util.Map configSnapshotForTest() { m.put("tls_verify", tlsValidationMode); m.put("tls_roots", trustStorePath); m.put("tls_roots_password", trustStorePassword == null ? null : new String(trustStorePassword)); + m.put("token_provider", tokenProviderSpec == null ? null : tokenProviderSpec.name()); + m.put("azure_resource", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + m.put("azure_client_id", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); return m; } @@ -1126,6 +1211,9 @@ public QwpQueryClient withConnectTimeout(int connectTimeoutMs) { */ public QwpQueryClient withBasicAuth(String username, String password) { checkPreConnect("withBasicAuth"); + if (tokenProviderSpec != null) { + throw new IllegalStateException("withBasicAuth cannot be combined with token_provider in the configuration"); + } if (tokenProvider != null) { throw new IllegalStateException("withBasicAuth cannot be combined with withBearerTokenProvider"); } @@ -1146,6 +1234,9 @@ public QwpQueryClient withBasicAuth(String username, String password) { */ public QwpQueryClient withBearerToken(String token) { checkPreConnect("withBearerToken"); + if (tokenProviderSpec != null) { + throw new IllegalStateException("withBearerToken cannot be combined with token_provider in the configuration"); + } if (tokenProvider != null) { throw new IllegalStateException("withBearerToken cannot be combined with withBearerTokenProvider"); } @@ -1179,6 +1270,10 @@ public QwpQueryClient withBearerTokenProvider(HttpTokenProvider provider) { if (provider == null) { throw new IllegalArgumentException("provider must not be null"); } + if (tokenProviderSpec != null) { + throw new IllegalStateException( + "withBearerTokenProvider cannot be combined with token_provider in the configuration"); + } if (authorizationHeader != null) { throw new IllegalStateException("withBearerTokenProvider cannot be combined with withBearerToken or withBasicAuth"); } @@ -1578,7 +1673,20 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa throw new IllegalStateException("QwpQueryClient is closed"); } if (!connected) { - throw new IllegalStateException("QwpQueryClient not connected; call connect() first"); + if (!reconnectOnNextExecute) { + throw new IllegalStateException("QwpQueryClient not connected; call connect() first"); + } + // A previous failover reconnect failed. Reconnect on this operation rather than leave the client + // permanently unusable (design/qwp-token-provider-spec.md, section 8.3, "Egress recovery") -- a + // pooled client is never discarded by its pool on its own, so without this a single failed + // failover (say, a token outage) would poison the slot for the life of the pool. + try { + connect(); + } catch (RuntimeException e) { + handler.onError(-1L, WebSocketResponse.STATUS_INTERNAL_ERROR, + "reconnect failed: " + e.getMessage()); + return; + } } hostTracker.beginRound(false); long failoverDeadlineNanos; @@ -1602,6 +1710,9 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa return; } if (!failoverEnabled) { + // With failover off the transport failure stays latched: every later execute() reports it. + healthTracker.connectionLost(); + healthTracker.failed(); handler.onError(probe.interceptedRequestId, probe.interceptedStatus, probe.interceptedMessage); return; } @@ -1621,6 +1732,10 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa } cleanupFailedConnect(); connected = false; + healthTracker.connectionLost(); + // Every exit from here on - an exhausted deadline, an interrupted backoff, a failed reconnect - leaves + // the client disconnected; the next execute() reconnects rather than throw "not connected". + reconnectOnNextExecute = true; if (failoverInitialBackoffMs > 0L) { long base = failoverInitialBackoffMs << Math.min(attempt - 1, 30); if (base < 0L) base = failoverMaxBackoffMs; @@ -1661,18 +1776,31 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa } } try { - reconnectViaTracker(); + try { + reconnectViaTracker(); + healthTracker.upgraded(); + } catch (RuntimeException e) { + healthTracker.roundFailed(e); + throw e; + } } catch (QwpAuthFailedException authErr) { // failover.md S6: AuthError is terminal across all hosts. // Credentials are cluster-wide, so retrying floods server logs // without recovery. Surface a distinct message so monitoring - // can pull auth incidents apart from generic transport failures. + // can pull auth incidents apart from generic transport failures; + // it names the failure class (design/qwp-token-provider-spec.md, 8.3). handler.onError(probe.interceptedRequestId, probe.interceptedStatus, - "auth failure during failover reconnect [host=" + "auth-rejected during failover reconnect [host=" + authErr.getHost() + ':' + authErr.getPort() + ", status=" + authErr.getStatusCode() + ", last error: " + probe.interceptedMessage + ']'); return; + } catch (QwpCredentialUnavailableException credentialErr) { + // The token provider could not supply a credential; no endpoint was contacted. + handler.onError(probe.interceptedRequestId, probe.interceptedStatus, + "failover reconnect failed: " + credentialErr.getMessage() + + " [last error: " + probe.interceptedMessage + ']'); + return; } catch (RuntimeException reconnectErr) { handler.onError(probe.interceptedRequestId, probe.interceptedStatus, "failover reconnect failed after " + attempt + " attempt" @@ -1873,8 +2001,16 @@ private void reconnectViaTracker() { // reason as connect(): a provider failure is cluster-wide, so surface it directly rather than // as a per-endpoint transport error retried across every host. String authHeader = resolveAuthorizationHeader(); + boolean authRetryAvailable = tokenProvider != null; + int retryIdx = -1; while (true) { - int i = hostTracker.pickNext(); + int i; + if (retryIdx >= 0) { + i = retryIdx; + retryIdx = -1; + } else { + i = hostTracker.pickNext(); + } if (i < 0) { if (!retriedAfterReset) { hostTracker.beginRound(true); @@ -1888,6 +2024,17 @@ private void reconnectViaTracker() { connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); + if (authRetryAvailable && ae.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshAfterRejection(authHeader, ae); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("QwpQueryClient {}:{} rejected the token with {} on failover; retrying once with " + + "a refreshed token", ep.host, ep.port, ae.getStatusCode()); + authHeader = refreshed; + retryIdx = i; + continue; + } + } throw ae; } catch (QwpIngressRoleRejectedException re) { lastError = re; @@ -1917,6 +2064,7 @@ private void reconnectViaTracker() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; + reconnectOnNextExecute = false; connected = true; return; } @@ -1934,30 +2082,54 @@ private String resolveAuthorizationHeader() { // With a token provider, query it once per connect()/reconnect (the caller resolves before the // endpoint walk) so a reconnect presents a freshly refreshed token; validateToken rejects a // null/empty/blank return, or one carrying a control or non-ASCII character, before it reaches - // the "Bearer " header. A provider that throws (a failed silent refresh, or not signed in yet) - // fails connect()/reconnect as a LineSenderException, preserving the provider failure as its cause. + // the "Bearer " header. A provider that throws (a failed silent refresh, or not signed in yet), or + // returns a token that fails validation, fails connect()/reconnect with the credential-unavailable + // failure class (design/qwp-token-provider-spec.md, section 8.1): a QwpCredentialUnavailableException + // whose message names the class and whose cause is the provider failure. if (tokenProvider != null) { - CharSequence pulled; + String token; try { - pulled = tokenProvider.getToken(); - } catch (LineSenderException e) { - throw e; + CharSequence pulled = tokenProvider.getToken(); + // snapshot before validating, for the reason HttpTokenProvider.validateToken gives: the + // concatenation below re-reads the sequence, and the provider may be reusing its buffer + token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); } catch (RuntimeException e) { - throw new LineSenderException( - e.getMessage() == null + throw new QwpCredentialUnavailableException( + "credential-unavailable: " + (e.getMessage() == null ? "token provider failed to supply a credential" - : e.getMessage(), + : e.getMessage()), e); } - // snapshot before validating, for the reason HttpTokenProvider.validateToken gives: the - // concatenation below re-reads the sequence, and the provider may be reusing its buffer - CharSequence token = pulled == null ? null : pulled.toString(); - HttpTokenProvider.validateToken(token); return "Bearer " + token; } return authorizationHeader; } + /** + * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells the + * token provider that the token in {@code presentedHeader} was rejected, then resolves the header again. + * Returns the new header, or null when no credential could be obtained - the connect then ends with the + * rejection, carrying the provider's failure as a suppressed diagnostic. + */ + private String refreshAfterRejection(String presentedHeader, QwpAuthFailedException rejection) { + if (presentedHeader != null && presentedHeader.startsWith("Bearer ")) { + try { + tokenProvider.onTokenRejected(presentedHeader.substring("Bearer ".length()), + rejection.getStatusCode()); + } catch (RuntimeException e) { + // The contract says it must not throw; a provider that does still gets its token re-pulled. + LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); + } + } + try { + return resolveAuthorizationHeader(); + } catch (RuntimeException e) { + rejection.addSuppressed(e); + return null; + } + } + private long resolveQueryFlags(boolean resetSymbolDict) { if (!resetSymbolDict) { return 0L; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java index c0709f53d..388c0cad3 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java @@ -48,7 +48,8 @@ static HttpClientException classify(WebSocketClient client, String host, int por } int status = client.getUpgradeStatusCode(); if (status == 401 || status == 403) { - QwpAuthFailedException ae = new QwpAuthFailedException(status, host, port); + QwpAuthFailedException ae = new QwpAuthFailedException( + status, host, port, client.getUpgradeRejectWwwAuthenticate()); ae.initCause(ex); return ae; } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 2f9c5a122..588a37e07 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -25,6 +25,8 @@ package io.questdb.client.cutlass.qwp.client; import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.ConnectionHealth; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionEvent; import io.questdb.client.SenderConnectionListener; @@ -65,6 +67,7 @@ import io.questdb.client.std.Numbers; import io.questdb.client.std.NumericException; import io.questdb.client.std.ObjList; +import io.questdb.client.std.QuietCloseable; import io.questdb.client.std.bytes.DirectByteSlice; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.TestOnly; @@ -185,6 +188,10 @@ public class QwpWebSocketSender implements Sender { // work, and neither the foreground's reconnect nor close() can queue // behind a drainer's endpoint walk. private final ReentrantLock connectWalkLock = new ReentrantLock(); + // Connection health of the FOREGROUND connection (design/qwp-token-provider-spec.md, section 8.4): rounds and + // upgrades from buildAndConnect, connection loss and terminal failure from the I/O loop, close from close(). + // Background drainer walks never report here. + private final QwpConnectionHealthTracker healthTracker = new QwpConnectionHealthTracker(); private final QwpHostHealthTracker hostTracker; // Per-table encoded body byte counts captured during flushPendingRows' combined // encode. flushPendingRowsSplit uses them both for preflight sizing and to walk @@ -233,8 +240,13 @@ public class QwpWebSocketSender implements Sender { // Test-only lifecycle witness. close() invokes and clears it strictly after // publishing closed=true and before starting any drain or teardown work. private volatile Runnable closeStartedHook; + // Optional authentication-outage deadline (spec section 8.5) handed to the I/O loop; 0 = none. + private volatile long authFailureMaxDurationMillis; private boolean connected; private SenderConnectionDispatcher connectionDispatcher; + // A connect-string token_provider's registry lease (TokenProviderRegistry), released when this sender + // closes so the shared provider can linger and then stop. Null for any other credential. + private volatile QuietCloseable credentialLease; // Async-delivery sink for SenderConnectionEvent notifications. Default // installed at construction; the builder hook can swap before connect() // runs, and post-connect setConnectionListener() propagates to the live @@ -1005,6 +1017,22 @@ public static Supplier fixedAuthHeader(String header) { return header == null ? null : new FixedAuthHeader(header); } + /** + * Wraps a token provider as a DYNAMIC {@code Authorization} header supplier: each {@code get()} pulls the + * provider's current token, snapshots it, validates it ({@link HttpTokenProvider#validateToken}) and returns + * {@code "Bearer " + token}. Unlike a bare lambda, the wrapper also carries the provider's + * {@link HttpTokenProvider#onTokenRejected} back-channel, so a {@code 401} on an upgrade that presented this + * credential tells the provider - a caching provider then refreshes early - before the connect walk pulls + * again and retries the same endpoint once with the new token (design/qwp-token-provider-spec.md, section + * 8.2). + * + * @param provider the token provider, or null when no credential is configured + * @return a dynamic supplier, or null when {@code provider} is null + */ + public static Supplier tokenProviderAuthHeader(HttpTokenProvider provider) { + return provider == null ? null : new TokenProviderAuthHeader(provider); + } + @Override public void at(long timestamp, ChronoUnit unit) { checkNotClosed(); @@ -1302,6 +1330,7 @@ public void close() { } private void close0(boolean[] restoreInterrupt) { + healthTracker.closed(); Runnable hook = closeStartedHook; closeStartedHook = null; if (hook != null) { @@ -1453,6 +1482,9 @@ private void close0(boolean[] restoreInterrupt) { terminalError = captureCloseError(terminalError, e); } } + // Nothing pulls a credential any more (or, after a failed stop, the registry's linger outlasts the + // straggler), so the shared token provider may go. + releaseCredentialLease(); // Always free resources the I/O thread never touches: // encoder and table buffers are user-thread-only. @@ -1507,6 +1539,54 @@ public boolean isCloseCleanupComplete() { return closeCleanupComplete; } + /** + * {@inheritDoc} + *

+ * Tracks the foreground connection only; orphan-slot drainers report through + * {@link BackgroundDrainerListener} and the error handler. + */ + @Override + public ConnectionHealth health() { + return healthTracker.snapshot(); + } + + /** + * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5); see + * {@code Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)}. {@code <= 0} disarms it. May be called + * while the I/O loop runs. + */ + public void setAuthFailureMaxDurationMillis(long millis) { + this.authFailureMaxDurationMillis = millis; + CursorWebSocketSendLoop loop = cursorSendLoop; + if (loop != null) { + loop.setAuthFailureMaxDurationMillis(millis); + } + } + + /** + * Hands this sender the registry lease of the {@code token_provider} its credential comes from; the sender + * releases it when it closes. {@code Sender.LineSenderBuilder.build()} calls this right after connecting. A + * lease handed to a sender that is already closed is released at once. + */ + public void setCredentialLease(QuietCloseable lease) { + this.credentialLease = lease; + if (closed) { + releaseCredentialLease(); + } + } + + private void releaseCredentialLease() { + QuietCloseable lease = credentialLease; + credentialLease = null; + if (lease != null) { + try { + lease.close(); + } catch (Throwable e) { + LOG.error("Error releasing the token provider lease: {}", String.valueOf(e)); + } + } + } + /** * True once the store-and-forward slot flock has been released. False * means an I/O or manager worker did not stop and close() retained the @@ -3211,7 +3291,15 @@ private WebSocketClient buildAndConnect(ReconnectSupplier ctx, CursorWebSocketSe } connectWalkLock.lock(); try { - return connectWalk(ctx, cancellation); + WebSocketClient client = connectWalk(ctx, cancellation); + healthTracker.upgraded(); + return client; + } catch (RuntimeException e) { + // One failed connect round. A walk that a close() aborted is not a failure of the connection. + if (!ctx.isAborted()) { + healthTracker.roundFailed(e); + } + throw e; } finally { connectWalkLock.unlock(); } @@ -3232,6 +3320,47 @@ private static void clearInFlight(CursorWebSocketSendLoop.ConnectCancellation ca } } + /** + * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells a + * provider-backed credential that {@code presentedHeader} was rejected - a caching provider refreshes early, + * waiting briefly for the fetch - and pulls the credential again. Returns the new header, or null when no + * credential could be obtained, in which case the round's outcome stays the rejection. + *

+ * Both calls run caller-supplied provider code that may block, so on a cancellable walk this thread is + * published as being inside a credential pull, exactly like the pull before the walk: {@code close()} then + * breaks it with an interrupt. + */ + private String refreshCredentialAfterRejection( + String presentedHeader, + QwpAuthFailedException rejection, + ReconnectSupplier ctx, + CursorWebSocketSendLoop.ConnectCancellation cancellation + ) { + final Supplier supplier = authorizationHeaderSupplier; + if (cancellation != null) { + cancellation.publishCredentialPull(Thread.currentThread()); + if (cancellation.isCancelled()) { + cancellation.clearCredentialPull(); + throw new LineSenderException(ctx.abortMessage()); + } + } + try { + if (supplier instanceof TokenProviderAuthHeader) { + ((TokenProviderAuthHeader) supplier).onRejected(presentedHeader, rejection.getStatusCode()); + } + return supplier.get(); + } catch (RuntimeException e) { + // No fresh credential to retry with. Keep the provider's failure as a diagnostic of the rejection + // the round ends with. + rejection.addSuppressed(e); + return null; + } finally { + if (cancellation != null) { + cancellation.clearCredentialPull(); + } + } + } + private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLoop.ConnectCancellation cancellation) { // Background (drainer) factories share this connect walk -- endpoint // list and hostTracker HEALTH state (never the shared round: a @@ -3330,7 +3459,7 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo throw new LineSenderException(ctx.abortMessage()); } } - final String authHeader; + String authHeader; try { authHeader = authorizationHeaderSupplier == null ? null : authorizationHeaderSupplier.get(); } catch (RuntimeException e) { @@ -3349,11 +3478,24 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo cancellation.clearCredentialPull(); } } + // One retry after a 401 per round (design/qwp-token-provider-spec.md, section 8.2): when an upgrade that + // presented a DYNAMIC credential is rejected with a refreshable 401, the provider is told, the credential + // is pulled again and, if it changed, the SAME endpoint is retried at once. The retried attempt is not an + // endpoint failure: it records no health penalty, fires no event, and does not consume the round's pick. + // A static credential never gets it - re-presenting the same bytes cannot change the answer. + boolean authRetryAvailable = hasDynamicCredential(); + int retryIdx = -1; while (true) { if (ctx.isAborted()) { throw new LineSenderException(ctx.abortMessage()); } - int idx = background ? cursor.next() : hostTracker.pickNext(); + int idx; + if (retryIdx >= 0) { + idx = retryIdx; + retryIdx = -1; + } else { + idx = background ? cursor.next() : hostTracker.pickNext(); + } if (idx < 0) break; Endpoint ep = endpoints.get(idx); lastEndpoint = ep; @@ -3425,12 +3567,26 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo continue; } if (classified instanceof QwpAuthFailedException) { + QwpAuthFailedException rejection = (QwpAuthFailedException) classified; + if (authRetryAvailable && rejection.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshCredentialAfterRejection( + authHeader, rejection, ctx, cancellation); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("{}:{} rejected the token with {}; retrying the same endpoint once with a " + + "refreshed token", ep.host, ep.port, rejection.getStatusCode()); + authHeader = refreshed; + retryIdx = idx; + continue; + } + } // Auth is uniform across the cluster; we won't keep walking // endpoints. Fire AUTH_FAILED before throwing so the user - // listener observes the terminal classification at the - // moment the I/O thread gives up, ahead of the producer + // listener observes the classification of the round's final + // outcome at the moment the walk gives up, ahead of the producer // thread learning via LineSenderException on the next - // API call. + // API call. A 401 that earned the same-endpoint retry above + // fires nothing: only the round's final outcome is reported. if (!background) { dispatchConnectionEvent( SenderConnectionEvent.Kind.AUTH_FAILED, @@ -4090,6 +4246,8 @@ private void ensureConnected() { // the loop no longer fires a terminal budget-exhaustion event -- it // retries indefinitely.) cursorSendLoop.setConnectionDispatcher(connectionDispatcher); + cursorSendLoop.setConnectionHealthTracker(healthTracker); + cursorSendLoop.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); cursorSendLoop.start(); } catch (Throwable t) { // start() (or dispatcher construction) failed after cursorSendLoop was @@ -5338,6 +5496,42 @@ public String get() { } } + /** + * A dynamic {@code Authorization} header backed by an {@link HttpTokenProvider}. See + * {@link #tokenProviderAuthHeader(HttpTokenProvider)}. + */ + private static final class TokenProviderAuthHeader implements Supplier { + private static final String BEARER_PREFIX = "Bearer "; + private final HttpTokenProvider provider; + + private TokenProviderAuthHeader(HttpTokenProvider provider) { + this.provider = provider; + } + + @Override + public String get() { + // Snapshot before validating: the concatenation below re-reads the sequence, and a provider is free + // to reuse a mutable buffer, so validating the live sequence checks bytes the header need not carry. + // See HttpTokenProvider.validateToken. + CharSequence pulled = provider.getToken(); + String token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); + return BEARER_PREFIX + token; + } + + void onRejected(String presentedHeader, int httpStatus) { + if (presentedHeader == null || !presentedHeader.startsWith(BEARER_PREFIX)) { + return; + } + try { + provider.onTokenRejected(presentedHeader.substring(BEARER_PREFIX.length()), httpStatus); + } catch (RuntimeException e) { + // The contract says it must not throw; a provider that does still gets its token re-pulled. + LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); + } + } + } + private final class ReconnectSupplier implements CursorWebSocketSendLoop.ReconnectFactory { /** * Optional caller-owned liveness gate. {@code null} means this factory diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 6643bf036..a64409f56 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -403,6 +403,18 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // indefinitely and never gives up on a wall-clock budget). private volatile SenderConnectionDispatcher connectionDispatcher; private volatile SenderErrorDispatcher errorDispatcher; + // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), FOREGROUND only; + // 0 = none, the default. Written by the owner thread, read by the I/O thread. + private volatile long authFailureMaxDurationNanos; + // The outage clock of that deadline: started by the first authentication-class failure (credential-unavailable + // or a 401/403 upgrade rejection) of the current outage, reset only by a successful upgrade. Failures of other + // classes neither reset nor fire it. Tracked whether or not a deadline is armed, so arming it late - the + // builder does so right after an async connect has started - still measures the whole outage. I/O thread only. + private boolean authOutageActive; + private long authOutageStartNanos; + // Foreground connection health (spec section 8.4): the loop reports connection loss and its terminal failure. + // Null for orphan drainers. + private volatile io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker healthTracker; // The send cursor has two coordinate systems: // // FSN: durable frame sequence number in the local cursor engine. This is @@ -1051,18 +1063,32 @@ public static WebSocketClient connectWithRetry( contextLabel, e.getMessage()); throw e; } catch (QwpCredentialUnavailableException e) { - // A credential the client cannot ACQUIRE (the configured token provider threw) is NOT a - // transport outage: retrying the connect cannot conjure a token the provider will not hand - // over, so fail fast with the provider's own exception rather than burn the whole connect - // budget treating it as a reachable-server problem (which would block build() for up to - // maxDurationMillis, default 5 min, and surface a transport-shaped wrapper). Mirrors the - // foreground OFF-mode connect (QwpWebSocketSender) and the background reconnect loop above, - // which both give credential acquisition its own terminal handling; only this SYNC - // initial-connect path lacked it. QwpCredentialUnavailableException is a LineSenderException, - // disjoint from the HttpClientException-based terminal set above, so it reaches here. - LOG.error("{} could not acquire a credential, won't retry: {}", - contextLabel, e.getMessage()); - throw e.providerFailure(); + // A credential the client cannot ACQUIRE (the configured token provider threw) is classified by + // the provider (design/qwp-token-provider-spec.md, section 8.3 and decisions D6/D8): + // - a TokenUnavailableException marked retryable - the IdP is unreachable, timing out or + // throttling - is a transient outage like any other, so it consumes the connect budget and is + // retried within it (D6); + // - anything else is permanent - no sign-in yet, a missing or wrong configuration - and retrying + // cannot conjure a token the provider will not hand over, so fail fast with the provider's own + // exception rather than burn the whole budget (up to maxDurationMillis, default 5 min) and + // surface a transport-shaped wrapper (D8). This is how an OIDC device-flow provider that is + // not signed in behaves, and how every provider failure behaved before D6. + // An interrupt on the calling thread also ends the retries: a provider wait it cut short reports + // retryable, and re-entering it with the flag still set would only spin through the budget. + // QwpCredentialUnavailableException is a LineSenderException, disjoint from the + // HttpClientException-based terminal set above, so it reaches here. + if (!e.isRetryable() || Thread.currentThread().isInterrupted()) { + LOG.error("{} could not acquire a credential, won't retry: {}", + contextLabel, e.getMessage()); + throw e.providerFailure(); + } + lastError = e; + long now = System.nanoTime(); + if (now - lastLogNanos >= RECONNECT_LOG_THROTTLE_NANOS) { + LOG.warn("{} attempt {}: credential-unavailable (retryable): {}; retrying within connect budget", + contextLabel, attempts, e.getMessage()); + lastLogNanos = now; + } } catch (Throwable e) { if (e instanceof Error) { // JVM/programming failure (OOM, LinkageError): not a @@ -1105,7 +1131,11 @@ public static WebSocketClient connectWithRetry( backoffMillis = Math.min(backoffMillis * 2, maxBackoffMillis); } long elapsedMs = (System.nanoTime() - startNanos) / 1_000_000L; - String lastMsg = lastError == null ? "no attempts made" : lastError.getMessage(); + String lastMsg = lastError == null + ? "no attempts made" + : lastError instanceof QwpCredentialUnavailableException + ? "credential-unavailable: " + lastError.getMessage() + : lastError.getMessage(); throw new LineSenderException( contextLabel + " failed after " + elapsedMs + "ms / " + attempts + " attempts: " + lastMsg, @@ -1550,6 +1580,25 @@ public void setConnectionDispatcher(SenderConnectionDispatcher dispatcher) { this.connectionDispatcher = dispatcher; } + /** + * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). A + * FOREGROUND loop then latches a terminal when a connect round fails with an authentication-class failure - + * credential-unavailable, or a 401/403 upgrade rejection - and the current outage's first such failure lies + * at least this far back. {@code <= 0} disarms it. Ignored by orphan drainers, which have their own policy. + * May be called while the loop runs. + */ + public void setAuthFailureMaxDurationMillis(long millis) { + this.authFailureMaxDurationNanos = millis <= 0 ? 0 : TimeUnit.MILLISECONDS.toNanos(millis); + } + + /** + * Plugs the owning sender's connection-health tracker: the loop reports connection loss and its terminal + * failure there. Set before {@link #start()}; foreground loops only. + */ + public void setConnectionHealthTracker(io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker) { + this.healthTracker = tracker; + } + /** * Plug an async-delivery sink for {@link SenderError} notifications. * Idempotent — set once before {@link #start()}; later reassignment is @@ -1746,6 +1795,12 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM snapshotReplayTarget(); LOG.warn("cursor I/O loop entering {} loop: {}", phase, initial.getMessage()); + if (hasEverConnected) { + io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker = healthTracker; + if (tracker != null) { + tracker.connectionLost(); + } + } long outageStartNanos = System.nanoTime(); // INVARIANT B: a store-and-forward loop must NEVER terminate on a // wall-clock reconnect budget. A replica-only / all-endpoints-replica @@ -1790,6 +1845,8 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM try { WebSocketClient newClient = reconnectFactory.reconnect(connectCancellation); if (newClient != null) { + // a successful upgrade ends the authentication outage (spec section 8.5) + authOutageActive = false; if (!running) { // close() ran while this connect attempt was in // flight. Its latch await may have been interrupted @@ -1875,6 +1932,9 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM } resetCatchUpCapGapEpisode(); lastReconnectError = e; + if (e instanceof QwpAuthFailedException && authOutageDeadlineFired(e)) { + return; + } dispatchRetriedEndpointPolicyFailure( SenderError.Category.SECURITY_ERROR, "ws-upgrade-failed: " + e.getMessage()); long now = System.nanoTime(); @@ -1942,6 +2002,9 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM // cap-gap dwell (see MAX_CATCHUP_CAP_GAP_ATTEMPTS). resetCatchUpCapGapEpisode(); lastReconnectError = e; + if (authOutageDeadlineFired(e)) { + return; + } // Retrying must not be programmatically INVISIBLE, exactly as for the auth/upgrade and // durable-ack policy failures above: a revoked refresh token or a permanently unreachable IdP // is not self-healing, yet flush() keeps returning success while SF absorbs the rows. Without @@ -2048,6 +2111,53 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM phase, elapsedMs, attempts, lastMsg); } + /** + * The optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), evaluated + * when a connect round of a FOREGROUND loop fails with an authentication-class failure ({@code failure} is a + * credential-unavailable or a 401/403 rejection). Starts the outage clock on the first such failure; when a + * deadline is armed and the clock has reached it, latches a terminal that names the failure class and the + * elapsed time, reports it to the error handler, and returns true. Unacknowledged rows stay in + * store-and-forward. + */ + private boolean authOutageDeadlineFired(Throwable failure) { + if (reconnectPolicy != ReconnectPolicy.FOREGROUND) { + return false; + } + final long now = System.nanoTime(); + if (!authOutageActive) { + authOutageActive = true; + authOutageStartNanos = now; + } + final long deadline = authFailureMaxDurationNanos; + if (deadline <= 0 || now - authOutageStartNanos < deadline) { + return false; + } + final long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(now - authOutageStartNanos); + final String failureClass = failure instanceof QwpCredentialUnavailableException + ? "credential-unavailable" : "auth-rejected"; + final String message = "authentication outage deadline exceeded: " + failureClass + " persisted for " + + elapsedMillis + "ms (auth_failure_max_duration_millis=" + + TimeUnit.NANOSECONDS.toMillis(deadline) + "); last failure: " + failure.getMessage(); + LOG.error("{} -- the sender stops; unacknowledged rows stay in store-and-forward", message); + long fromFsn = engine.ackedFsn() + 1L; + long toFsn = Math.max(fromFsn, engine.publishedFsn()); + SenderError err = new SenderError( + SenderError.Category.SECURITY_ERROR, + SenderError.Policy.TERMINAL, + SenderError.NO_STATUS_BYTE, + message, + SenderError.NO_MESSAGE_SEQUENCE, + fromFsn, + toFsn, + null, + System.nanoTime() + ); + totalServerErrors.incrementAndGet(); + recordFatal(new LineSenderServerException(err)); + dispatchError(err); + return true; + } + /** * Reports an endpoint-policy rejection a FOREGROUND sender is riding out. *

@@ -2549,6 +2659,10 @@ private void positionCursorInSegment(MmapSegment seg, long targetFsn) { * every rethrow delivers the same instance. */ private void recordFatal(Throwable t) { + io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker = healthTracker; + if (tracker != null) { + tracker.failed(); + } if (terminalError == null) { terminalError = t instanceof LineSenderException ? (LineSenderException) t diff --git a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java index c9529a13e..38eebb6fc 100644 --- a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java +++ b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java @@ -57,10 +57,19 @@ public final class ConfigSchema { str("tls_roots_password", Side.COMMON); longRange("auth_timeout_ms", Side.COMMON, 0, OPEN_MAX, true, false); // > 0 longRange("connect_timeout", Side.COMMON, 0, OPEN_MAX, true, false); // > 0 + // Dynamic bearer credentials (design/qwp-token-provider-spec.md, section 7.1): a refreshing token + // provider selected by name, wss:: only, mutually exclusive with token/username/password. The values are + // never secret. Validated by TokenProviderSpec on both clients. + str("token_provider", Side.COMMON); + str("azure_resource", Side.COMMON); + str("azure_client_id", Side.COMMON); // INGRESS -- the WebSocket Sender applies. STRING in the registry; the // Sender parses suffix/mode values (off/on, 64k, durability) with its // own helpers, byte-for-byte. + // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). Not set by + // default; > 0 when set. + longRange("auth_failure_max_duration_millis", Side.INGRESS, 0, OPEN_MAX, true, false); str("auto_flush", Side.INGRESS); str("auto_flush_bytes", Side.INGRESS); str("auto_flush_interval", Side.INGRESS); diff --git a/core/src/main/java/io/questdb/client/impl/PooledSender.java b/core/src/main/java/io/questdb/client/impl/PooledSender.java index 7b4e5f802..06f3940f9 100644 --- a/core/src/main/java/io/questdb/client/impl/PooledSender.java +++ b/core/src/main/java/io/questdb/client/impl/PooledSender.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.Sender; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; @@ -276,6 +277,11 @@ public long getAckedFsn() { return slot.live(generation).getAckedFsn(); } + @Override + public ConnectionHealth health() { + return slot.live(generation).health(); + } + @Override public Sender intColumn(CharSequence name, int value) { slot.live(generation).intColumn(name, value); diff --git a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java index 8e858c019..4f3c7b566 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java +++ b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.QueryException; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; @@ -485,6 +486,21 @@ void discard(QueryWorker w, long gen) { } } + /** + * Adds the connection health of every pooled query client to {@code into}. Reading a client's health never + * waits on its I/O, so this is safe under the pool lock. + */ + void collectHealth(java.util.List into) { + lock.lock(); + try { + for (int i = 0, n = all.size(); i < n; i++) { + into.add(all.get(i).client().health()); + } + } finally { + lock.unlock(); + } + } + void reapIdle() { if (closed) { return; diff --git a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java index 574d6b59e..0f0c4c64f 100644 --- a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java +++ b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.QuestDB; import io.questdb.client.Query; @@ -206,6 +207,14 @@ public Sender borrowSender() { return senderPool.borrow(); } + @Override + public ConnectionHealth.Aggregate health() { + java.util.List healths = new java.util.ArrayList<>(); + senderPool.collectHealth(healths); + queryPool.collectHealth(healths); + return ConnectionHealth.Aggregate.of(healths); + } + // synchronized so concurrent close() callers serialize THROUGH the bounded // shutdown sequence, not merely through the `closed` flip. `closed` is set // before the teardown chain runs, so a plain volatile guard (or a bare CAS) diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java index 1a4390269..fd899158c 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderPool.java +++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionListener; @@ -1923,6 +1924,25 @@ public int availableSize() { } } + /** + * Adds the connection health of every live pooled sender to {@code into}. Reading a sender's health never + * waits on its I/O, so this is safe under the pool lock. + */ + public void collectHealth(java.util.List into) { + lock.lock(); + try { + for (int i = 0, n = all.size(); i < n; i++) { + try { + into.add(all.get(i).delegate().health()); + } catch (UnsupportedOperationException ignored) { + // a test double without connection health + } + } + } finally { + lock.unlock(); + } + } + /** Snapshot of the total number of live slots (idle + in-use). For tests and introspection. */ public int totalSize() { lock.lock(); diff --git a/core/src/main/java/module-info.java b/core/src/main/java/module-info.java index 8383221e4..4f4bc9852 100644 --- a/core/src/main/java/module-info.java +++ b/core/src/main/java/module-info.java @@ -71,4 +71,8 @@ exports io.questdb.client.cutlass.qwp.client.sf.cursor; exports io.questdb.client.cutlass.qwp.protocol; exports io.questdb.client.cutlass.qwp.websocket; + + // token_provider= in a connect string resolves through this SPI; the optional questdb-client-azure + // artifact provides the "azure" factory (design/qwp-token-provider-spec.md, section 7). + uses io.questdb.client.cutlass.auth.TokenProviderFactory; } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java new file mode 100644 index 000000000..629377106 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -0,0 +1,841 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.test.cutlass.auth.TokenTestKit.FakeClock; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ManualScheduler; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; + +/** + * Conformance tests C1-C8 and the cache half of C20 of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 10) for {@link RefreshingTokenProvider}. + *

+ * Schedule, backoff, hand-out and forced-refresh tests run on a fake clock and a scheduler the test drives, so + * they assert exact delays without waiting. The concurrency, interrupt and lifecycle tests run on the real + * refresher thread. + */ +public class RefreshingTokenProviderTest { + private static final long MIN = 60_000L; + private static final long HOUR = 60 * MIN; + private static final long DAY = 24 * HOUR; + // A fixed wall-clock origin keeps the expected instants literal. + private static final long T0 = 1_800_000_000_000L; + + @Test + public void testAlreadyExpiredResultIsARetryableFailure() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("EXPIRED", T0 - 1) + .thenToken("FRESH", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals(1, provider.getConsecutiveFailures()); + RefreshingTokenProvider.FetchFailure failure = provider.getLastFailure(); + Assert.assertNotNull(failure); + Assert.assertTrue("a result that has already expired is a retryable failure", failure.isRetryable()); + Assert.assertTrue(failure.getMessage(), failure.getMessage().contains("expired")); + Assert.assertEquals(RefreshingTokenProvider.NONE, provider.getTokenExpiresAtEpochMillis()); + // retried after backoff, not dropped + scheduler.advanceAndRunNext(); + Assert.assertEquals("FRESH", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + } + } + + @Test(timeout = 30_000) + public void testAwaitReadyGatesOnTheFirstFetch() { + ScriptedSource source = new ScriptedSource().thenToken("READY", System.currentTimeMillis() + HOUR); + source.setGate(); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertFalse("the first fetch has not finished", provider.awaitReady(50)); + source.openGate(); + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("READY", provider.getToken().toString()); + } + } + + @Test + public void testBackoffFollowsTheSpec() { + // section 5.4: base = min(backoff_max, backoff_initial * 2^(n-1)); delay = base/2 + U(0, base/2). + // u = 0 pins the delay to the lower bound base/2, u = 0.5 to 3/4 of base. + assertBackoffDelays(0.0, new long[]{250, 500, 1_000, 2_000, 4_000, 8_000, 16_000, 30_000, 30_000}); + assertBackoffDelays(0.5, new long[]{375, 750, 1_500, 3_000, 6_000, 12_000, 24_000, 45_000, 45_000}); + } + + @Test + public void testCloseFailsGetTokenPermanentlyAndStopsTheRefresher() { + ScriptedSource source = new ScriptedSource().thenToken("TOKEN", System.currentTimeMillis() + HOUR); + List before = refresherThreads(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + Assert.assertTrue(provider.awaitReady(10_000)); + List started = refresherThreads(); + started.removeAll(before); + Assert.assertEquals("one refresher thread per provider", 1, started.size()); + + provider.close(); + provider.close(); // idempotent + Assert.assertTrue(provider.isClosed()); + await(() -> !started.get(0).isAlive(), 5_000, "the refresher thread to exit"); + try { + provider.getToken(); + Assert.fail("getToken() after close() must fail"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("a closed provider is a permanent failure", e.isRetryable()); + } + Assert.assertFalse(provider.awaitReady(10)); + provider.onTokenRejected("TOKEN", 401); // no-op, no throw + } + + @Test(timeout = 30_000) + public void testColdBurstAfterTheWallClockJumpsCausesOneFetch() throws Exception { + // C3, second shape: a token is held, but the wall clock jumped past its expiry (a host that slept + // through its scheduled refresh). 64 callers find no usable token at once; exactly one fetch runs. + SkewedClock clock = new SkewedClock(); + ScriptedSource source = new ScriptedSource() + .thenToken("OLD", System.currentTimeMillis() + HOUR) + .then(() -> new ExpiringToken("NEW", clock.wallClockMillis() + HOUR)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).clock(clock).build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("OLD", provider.getToken().toString()); + source.setGate(); + clock.skewMillis = 2 * HOUR; + Assert.assertEquals(listOf("NEW"), burst(provider, 64, source)); + Assert.assertEquals("the prefetch plus exactly one fetch for the whole burst", 2, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdBurstCausesExactlyOneFetch() throws Exception { + // C3: 64 concurrent callers with no usable token cause exactly one fetch. + ScriptedSource source = new ScriptedSource().thenToken("TOKEN-1", System.currentTimeMillis() + HOUR); + source.setGate(); // the prefetch blocks until every caller is waiting + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertEquals(listOf("TOKEN-1"), burst(provider, 64, source)); + Assert.assertEquals(1, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdFailureFailsImmediatelyWhenBackoffOutlastsTheWait() { + // section 5.3: "If the earliest permitted fetch is later than now + cold_wait, fail immediately." + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("HTTP 429 from the token endpoint", 60_000)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(5_000) + .build()) { + await(() -> provider.getConsecutiveFailures() == 1, 10_000, "the prefetch to fail"); + long start = System.nanoTime(); + try { + provider.getToken(); + Assert.fail("expected the cold wait to fail"); + } catch (TokenUnavailableException e) { + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must fail at once, not wait out cold_wait; took " + elapsedMillis + " ms", + elapsedMillis < 2_000); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue("the remaining backoff is the hint: " + e.getRetryAfterMillis(), + e.getRetryAfterMillis() > 30_000); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("HTTP 429")); + } + Assert.assertEquals("the caller must not bypass the backoff with a fetch of its own", 1, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdFailurePassesAPermanentClassificationThrough() { + // C4: a permanent classification is passed through to the caller. + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.permanent("no credential is configured")); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(300) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + try { + provider.getToken(); + Assert.fail("expected a token-unavailable error"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("the source's permanent classification must reach the caller", e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no credential is configured")); + } + // the provider keeps retrying a "permanent" failure: an operator can repair it without a restart + int calls = source.calls(); + await(() -> source.calls() > calls, 5_000, "another retry of the permanent failure"); + } + } + + @Test(timeout = 30_000) + public void testColdFailureRaisesARetryableErrorWithinColdWait() { + // C4: a retryable error is raised within cold_wait. + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("IMDS answered HTTP 503")); + long coldWaitMillis = 400; + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(coldWaitMillis) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + long start = System.nanoTime(); + try { + provider.getToken(); + Assert.fail("expected a token-unavailable error"); + } catch (TokenUnavailableException e) { + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("HTTP 503")); + Assert.assertTrue("the wait must be bounded by cold_wait; took " + elapsedMillis + " ms", + elapsedMillis < coldWaitMillis + 2_000); + } + Assert.assertTrue("short failures inside cold_wait are retried, not surfaced on the first one", + source.calls() > 1); + } + } + + @Test + public void testCredentialRedactionHelpers() { + String token = "abc.def.ghi"; + String fingerprint = CredentialRedaction.fingerprint(token); + Assert.assertTrue(fingerprint, fingerprint.matches("[0-9a-f]{8}")); + String described = CredentialRedaction.describeToken(token); + Assert.assertEquals("', described); + Assert.assertEquals("", CredentialRedaction.describeToken(null)); + + // controls, CR/LF and bidi overrides are removed; the result is capped at 256 characters + String hostile = "line1\r\nFORGED: yes\u202Eevil\u0007" + repeat('x', 1_000); + String clean = CredentialRedaction.sanitizeErrorText(hostile); + Assert.assertTrue(clean.length() <= CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertTrue(clean.endsWith("...")); + Assert.assertFalse(clean.contains("\r") || clean.contains("\n") || clean.contains("\u202E") + || clean.contains("\u0007")); + Assert.assertTrue(clean.startsWith("line1FORGED: yesevil")); + // text that fits is kept whole + String exact = repeat('y', CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertEquals(exact, CredentialRedaction.sanitizeErrorText(exact)); + Assert.assertNull(CredentialRedaction.sanitizeErrorText(null)); + } + + @Test + public void testExpiringTokenRejectsInvalidTokensWithoutEchoingThem() { + String secret = "SECRET-" + UUID.randomUUID(); + assertRejected(null); + assertRejected(""); + assertRejected(" "); + assertRejected(secret + "\r\nX-Injected: 1"); + assertRejected(secret + "\u00e9"); + try { + new ExpiringToken(secret + '\n', T0); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertFalse("the token must never be echoed", e.getMessage().contains(secret)); + } + ExpiringToken ok = new ExpiringToken(secret, T0, -5); + Assert.assertFalse("a non-positive refresh hint means none", ok.hasRefreshAt()); + Assert.assertEquals(ExpiringToken.NO_REFRESH_AT, ok.getRefreshAtEpochMillis()); + Assert.assertTrue(new ExpiringToken(secret, T0 + HOUR, T0 + MIN).hasRefreshAt()); + } + + @Test(timeout = 30_000) + public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameToken() throws Exception { + // C8: rate limited; a stale token is ignored; a same-token result is accepted. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + HOUR) + .thenToken("T2", T0 + HOUR) + .thenToken("T2", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5) + .forcedMinIntervalMillis(30_000) + .forcedWaitMillis(5_000) + .build()) { + scheduler.runNext(); + Assert.assertEquals("T1", provider.getToken().toString()); + + // 403 never triggers a forced refresh (decision D2) + assertReturnsPromptly(() -> provider.onTokenRejected("T1", 403)); + Assert.assertNotEquals("a 403 must not schedule a fetch", 0, scheduler.peek().delayNanos); + + // 401 for the current token: a fetch starts now and the caller waits for it + Thread rejecter = start(() -> provider.onTokenRejected("T1", 401)); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the forced fetch to be scheduled"); + scheduler.runNext(); + rejecter.join(5_000); + Assert.assertFalse("onTokenRejected must return once the forced fetch completes", rejecter.isAlive()); + Assert.assertEquals("T2", provider.getToken().toString()); + Assert.assertEquals(2, source.calls()); + + // a stale token - it has already rotated - is ignored + assertReturnsPromptly(() -> provider.onTokenRejected("T1", 401)); + // rate limited: a second forced refresh within forced_min_interval is ignored + clock.advanceMillis(30_000 - 1); + assertReturnsPromptly(() -> provider.onTokenRejected("T2", 401)); + Assert.assertNotEquals("no forced fetch may be scheduled", 0, scheduler.peek().delayNanos); + Assert.assertEquals(2, source.calls()); + + // once the interval has passed, a forced fetch that returns the same token is a normal success + clock.advanceMillis(1); + rejecter = start(() -> provider.onTokenRejected("T2", 401)); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the second forced fetch to be scheduled"); + scheduler.runNext(); + rejecter.join(5_000); + Assert.assertFalse(rejecter.isAlive()); + Assert.assertEquals(3, source.calls()); + Assert.assertEquals("T2", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + Assert.assertNull("a same-token result is not a failure", provider.getLastFailure()); + Assert.assertEquals(clock.wallClockMillis(), provider.getLastSuccessEpochMillis()); + } + } + + @Test(timeout = 30_000) + public void testHandoutFloorTokenIsNeverHandedOut() throws Exception { + // C6: a token inside handout_floor is never handed out. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + 10 * MIN) + .thenToken("T2", T0 + 70 * MIN); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + clock.jumpWallMillis(10 * MIN - RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS - 1); + Assert.assertEquals("one millisecond outside the floor the token is still usable", + "T1", provider.getToken().toString()); + clock.jumpWallMillis(1); + + // exactly at expires_at - handout_floor the token is unusable: the caller must get a new one + AtomicReference got = new AtomicReference<>(); + Thread caller = start(() -> got.set(provider.getToken().toString())); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the cold caller to start a fetch"); + Assert.assertNull("nothing may be handed out before the fetch", got.get()); + scheduler.runNext(); + caller.join(5_000); + Assert.assertEquals("T2", got.get()); + } + } + + @Test(timeout = 30_000) + public void testHandoutFloorTokenIsNotHandedOutEvenWhenTheSourceRepeatsIt() throws Exception { + // C6, second shape: the source keeps returning the token that sits inside the floor. The caller waits + // out cold_wait and fails rather than receive it. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("T1", T0 + 10 * MIN); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5) + .coldWaitMillis(1_000) + .build()) { + scheduler.runNext(); + clock.jumpWallMillis(10 * MIN - 30_000); + AtomicReference got = new AtomicReference<>(); + Thread caller = start(() -> { + try { + got.set(provider.getToken().toString()); + } catch (TokenUnavailableException e) { + got.set(e); + } + }); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the cold caller to start a fetch"); + scheduler.runNext(); // the source returns T1 again, still inside the floor + Assert.assertNull("an unusable token must not be handed out", got.get()); + clock.advanceMillis(1_001); // cold_wait elapses + caller.join(5_000); + Assert.assertTrue("expected a token-unavailable error, got " + got.get(), + got.get() instanceof TokenUnavailableException); + Assert.assertTrue("no fetch failed, so the error is retryable", + ((TokenUnavailableException) got.get()).isRetryable()); + Assert.assertEquals("the caller started one fetch, not one per completed fetch", 2, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testInterruptedWaitEndsPromptlyAndKeepsTheFlag() throws Exception { + // C7: a cancelled wait returns promptly, and the cancellation signal is preserved. + ScriptedSource source = new ScriptedSource().thenToken("NEVER", System.currentTimeMillis() + HOUR); + source.setGate(); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + AtomicReference error = new AtomicReference<>(); + AtomicBoolean flagKept = new AtomicBoolean(); + Thread caller = start(() -> { + try { + provider.getToken(); + } catch (TokenUnavailableException e) { + error.set(e); + flagKept.set(Thread.currentThread().isInterrupted()); + } + }); + await(() -> caller.getState() == Thread.State.TIMED_WAITING, 5_000, "the caller to wait"); + long start = System.nanoTime(); + caller.interrupt(); + caller.join(5_000); + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertFalse(caller.isAlive()); + Assert.assertNotNull("an interrupted wait must end with a token-unavailable error", error.get()); + Assert.assertTrue("a cancelled wait is retryable", error.get().isRetryable()); + Assert.assertTrue("the interrupt flag must be preserved", flagKept.get()); + Assert.assertTrue("must return promptly; took " + elapsedMillis + " ms", elapsedMillis < 2_000); + + // a caller that arrives already interrupted fails at once, flag intact + Thread.currentThread().interrupt(); + try { + provider.getToken(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertTrue(Thread.interrupted()); + } + source.openGate(); + } + } + + @Test(timeout = 30_000) + public void testProactiveRefreshRunsWithoutACaller() { + // section 5.2: the provider starts a fetch at the refresh time without waiting for a caller. + long now = System.currentTimeMillis(); + ScriptedSource source = new ScriptedSource() + .thenToken("SHORT", now + 1_000) + .then(() -> new ExpiringToken("NEXT", System.currentTimeMillis() + HOUR)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .handoutFloorMillis(0) + .refreshMarginMillis(0) + .minRefreshIntervalMillis(0) + .build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + await(() -> source.calls() == 2, 10_000, "the proactive refresh"); + await(() -> provider.getTokenExpiresAtEpochMillis() > now + 10_000, 10_000, "the new token"); + Assert.assertEquals("NEXT", provider.getToken().toString()); + } + } + + @Test + public void testRefreshScheduleFollowsTheSpec() { + // C1: refresh times follow section 5.2 for L = 24 h, 60 min and 5 min, with and without refresh_at. + // u = 0 is the earliest jitter (0.45 L), u = 0.5 the middle (0.5 L), u = 0.999 near the latest (0.55 L). + // u = 0 is the earliest jitter (0.45 L), u = 0.5 the middle (0.5 L), u = 1 the latest (0.55 L). + assertRefreshDelay(DAY, 0.0, 0, 38_880_000L); // 10.8 h + assertRefreshDelay(DAY, 0.5, 0, 43_200_000L); // 12 h + assertRefreshDelay(DAY, 1.0, 0, 47_520_000L); // 13.2 h + assertRefreshDelay(HOUR, 0.0, 0, 1_620_000L); // 27 min + assertRefreshDelay(HOUR, 0.5, 0, 1_800_000L); // 30 min + assertRefreshDelay(HOUR, 1.0, 0, 1_980_000L); // 33 min + // L = 5 min: the margin is min(5 min, L/2) = 2.5 min, so the latest refresh is at 2.5 min + assertRefreshDelay(5 * MIN, 0.0, 0, 135_000L); + assertRefreshDelay(5 * MIN, 0.5, 0, 150_000L); + assertRefreshDelay(5 * MIN, 1.0, 0, 150_000L); + // a refresh hint earlier than the jittered half-life wins ... + assertRefreshDelay(HOUR, 0.5, T0 + 10 * MIN, 10 * MIN); + assertRefreshDelay(DAY, 0.5, T0 + 6 * HOUR, 6 * HOUR); + // ... but never earlier than min_refresh_interval + assertRefreshDelay(HOUR, 0.5, T0 + 5_000, 30_000L); + assertRefreshDelay(HOUR, 0.5, T0 - HOUR, 30_000L); + // a later hint does not delay the refresh + assertRefreshDelay(HOUR, 0.5, T0 + 50 * MIN, 30 * MIN); + assertRefreshDelay(5 * MIN, 0.5, T0 + 4 * MIN, 150_000L); + // a 50 s token: min_refresh_interval shrinks to L/2 = 25 s, the margin to 25 s + assertRefreshDelay(50_000L, 0.0, 0, 25_000L); + // the pure function agrees with the provider + Assert.assertEquals(T0 + 1_800_000L, + RefreshingTokenProvider.computeRefreshAtMillis(T0, T0 + HOUR, 0, 0.5, 5 * MIN, 30_000)); + } + + @Test + public void testRetryAfterIsHonouredAndCapped() { + // C5: retry_after is honoured and capped at retry_after_max. + assertFirstBackoff(10_000L, 10_000L); // longer than the backoff: honoured + assertFirstBackoff(10 * MIN, 5 * MIN); // capped at retry_after_max (5 min) + assertFirstBackoff(100L, 250L); // shorter than the backoff: the backoff wins + assertFirstBackoff(TokenUnavailableException.NO_RETRY_AFTER, 250L); + } + + @Test + public void testSameTokenConvergesAndStaysUsable() { + // C2: the platform keeps returning the same 24 h token. Each refresh lands at about half the remaining + // lifetime, so the token is fetched only a handful of times before it enters the hand-out floor, and + // it stays usable all the while. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + final long expiry = T0 + DAY; + ScriptedSource source = new ScriptedSource().thenToken("SAME", expiry); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + int fetchesWhileUsable = 1; + while (true) { + long remaining = expiry - clock.wallClockMillis(); + ManualScheduler.Task next = scheduler.peek(); + long expectedDelay = RefreshingTokenProvider.computeRefreshAtMillis( + 0, remaining, 0, 0.5, 5 * MIN, 30_000); + Assert.assertEquals("refresh at about half the remaining lifetime [remaining=" + remaining + ']', + expectedDelay, next.delayMillis()); + Assert.assertTrue(next.delayMillis() <= remaining / 2 + 1); + scheduler.advanceAndRunNext(); + if (clock.wallClockMillis() >= expiry - RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS) { + break; + } + Assert.assertEquals("SAME", provider.getToken().toString()); + fetchesWhileUsable++; + } + // 24 h halves to below the 60 s floor in 11 steps (86400 s -> ... -> 84.4 s) + Assert.assertEquals(11, fetchesWhileUsable); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + } + } + + @Test + public void testSentinelTokenIsNeverRendered() throws Exception { + // C20 for the cache: a sentinel token never appears in a string representation, an error, a failure + // record or a log line - including when the source returns it in a malformed result. + final String sentinel = "SENTINEL-" + UUID.randomUUID(); + ch.qos.logback.classic.Logger logger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(RefreshingTokenProvider.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = logger.getLevel(); + logger.setLevel(ch.qos.logback.classic.Level.ALL); + logger.addAppender(appender); + List rendered = new ArrayList<>(); + try { + ExpiringToken token = new ExpiringToken(sentinel, T0 + HOUR, T0 + MIN); + rendered.add(token.toString()); + try { + new ExpiringToken(sentinel + "\r\n", T0 + HOUR); + } catch (IllegalArgumentException e) { + rendered.add(e.getMessage()); + } + + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken(sentinel, T0 + 10 * MIN) + .thenToken(sentinel, T0 - 1) // malformed: already expired + .thenThrow(new IllegalStateException("token endpoint returned a response without access_token")); + RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).coldWaitMillis(0).build(); + try { + scheduler.runNext(); + rendered.add(provider.toString()); + scheduler.advanceAndRunNext(); + rendered.add(String.valueOf(provider.getLastFailure())); + scheduler.advanceAndRunNext(); + rendered.add(String.valueOf(provider.getLastFailure())); + rendered.add(provider.toString()); + clock.jumpWallMillis(HOUR); + try { + provider.getToken(); + Assert.fail("no usable token is held"); + } catch (TokenUnavailableException e) { + rendered.add(e.getMessage()); + rendered.add(e.toString()); + } + } finally { + provider.close(); + } + rendered.add(provider.toString()); + } finally { + logger.detachAppender(appender); + logger.setLevel(savedLevel); + appender.stop(); + } + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + rendered.add(event.getFormattedMessage()); + } + Assert.assertFalse("expected log lines to inspect", appender.list.isEmpty()); + for (String s : rendered) { + Assert.assertFalse("the token leaked into: " + s, s.contains(sentinel)); + } + Assert.assertTrue(rendered.get(0), rendered.get(0).contains("sha256:")); + } + + @Test + public void testSourceFailureTextIsSanitized() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow(TokenUnavailableException.retryable( + "AADSTS50000:\r\nX-Forged: 1\u202E" + repeat('z', 500))); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + String message = provider.getLastFailure().getMessage(); + Assert.assertTrue(message.length() <= CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertFalse(message.contains("\r") || message.contains("\n") || message.contains("\u202E")); + Assert.assertTrue(message, message.startsWith("AADSTS50000:X-Forged: 1zzz")); + } + } + + @Test + public void testSuccessResetsTheBackoff() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("down")) + .thenThrow(TokenUnavailableException.retryable("down")) + .thenThrow(TokenUnavailableException.retryable("down")) + .thenToken("UP", T0 + HOUR) + .thenThrow(TokenUnavailableException.retryable("down again")); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals(250, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(500, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(1_000, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals("UP", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + Assert.assertEquals("down", provider.getLastFailure().getMessage()); + scheduler.advanceAndRunNext(); // the proactive refresh fails + Assert.assertEquals(1, provider.getConsecutiveFailures()); + Assert.assertEquals("a success resets n, so the backoff starts over", 250, scheduler.peek().delayMillis()); + Assert.assertEquals("the current token keeps being served while usable", + "UP", provider.getToken().toString()); + } + } + + @Test(timeout = 30_000) + public void testUnclassifiedSourceExceptionIsRetryable() { + ScriptedSource source = new ScriptedSource().thenThrow(new IllegalStateException("socket reset")); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(200) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + try { + provider.getToken(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertTrue("an error that carries no classification is retryable", e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("IllegalStateException: socket reset")); + } + } + } + + @Test + public void testWarmPathDoesNotFetch() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("WARM", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + ManualScheduler.Task scheduled = scheduler.peek(); + for (int i = 0; i < 1_000; i++) { + Assert.assertEquals("WARM", provider.getToken().toString()); + } + Assert.assertEquals(1, source.calls()); + Assert.assertSame("a warm read must not touch the schedule", scheduled, scheduler.peek()); + } + } + + @Test + public void testWallClockOverdueRefreshStartsInTheBackground() { + // section 5.3 step 1: a usable token is returned at once, and when its refresh time has passed - here the + // wall clock moved past it while the monotonic schedule did not - a fetch starts in the background. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("T1", T0 + HOUR).thenToken("T2", T0 + 2 * HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + Assert.assertEquals(30 * MIN, scheduler.peek().delayMillis()); + clock.jumpWallMillis(31 * MIN); + Assert.assertEquals("the usable token is returned at once", "T1", provider.getToken().toString()); + Assert.assertEquals("the overdue refresh is pulled forward", 0, scheduler.peek().delayNanos); + scheduler.runNext(); + Assert.assertEquals("T2", provider.getToken().toString()); + } + } + + private static void assertBackoffDelays(double u, long[] expectedMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow(TokenUnavailableException.retryable("down")); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u).build()) { + scheduler.runNext(); + for (int n = 1; n <= expectedMillis.length; n++) { + Assert.assertEquals("backoff after failure " + n + " [u=" + u + ']', + expectedMillis[n - 1], scheduler.peek().delayMillis()); + Assert.assertEquals(n, provider.getConsecutiveFailures()); + scheduler.advanceAndRunNext(); + } + } + } + + private static void assertFirstBackoff(long retryAfterMillis, long expectedDelayMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow( + new TokenUnavailableException("throttled", true, retryAfterMillis)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals("retry_after=" + retryAfterMillis, expectedDelayMillis, scheduler.peek().delayMillis()); + Assert.assertEquals(retryAfterMillis, provider.getLastFailure().getRetryAfterMillis()); + } + } + + private static void assertRefreshDelay(long lifetimeMillis, double u, long refreshAt, long expectedDelayMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .then(() -> new ExpiringToken("TOKEN", T0 + lifetimeMillis, refreshAt)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u) + .handoutFloorMillis(0) + .build()) { + Assert.assertEquals("the prefetch runs at once", 0, scheduler.peek().delayNanos); + scheduler.runNext(); + Assert.assertEquals("refresh delay [L=" + lifetimeMillis + ", u=" + u + ", refreshAt=" + refreshAt + ']', + expectedDelayMillis, scheduler.peek().delayMillis()); + Assert.assertEquals(T0 + lifetimeMillis, provider.getTokenExpiresAtEpochMillis()); + } + } + + private static void assertRejected(String token) { + try { + new ExpiringToken(token, T0); + Assert.fail("expected the token to be rejected"); + } catch (IllegalArgumentException expected) { + // ok + } + } + + private static void assertReturnsPromptly(Runnable r) throws InterruptedException { + Thread t = start(r); + t.join(2_000); + Assert.assertFalse("the call must return without waiting", t.isAlive()); + } + + private static List burst(RefreshingTokenProvider provider, int callers, ScriptedSource source) + throws InterruptedException { + CountDownLatch ready = new CountDownLatch(callers); + List threads = new ArrayList<>(); + String[] results = new String[callers]; + for (int i = 0; i < callers; i++) { + final int idx = i; + Thread t = new Thread(() -> { + ready.countDown(); + results[idx] = provider.getToken().toString(); + }); + threads.add(t); + t.start(); + } + Assert.assertTrue(ready.await(10, TimeUnit.SECONDS)); + // Give every caller time to reach the wait before the single fetch completes. The assertion on the + // fetch count holds however the callers interleave; this only makes the burst a real burst. + final long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(2); + while (System.nanoTime() - deadline < 0 && !allParked(threads)) { + Thread.sleep(5); + } + source.openGate(); + List distinct = new ArrayList<>(); + for (int i = 0; i < callers; i++) { + threads.get(i).join(10_000); + Assert.assertFalse(threads.get(i).isAlive()); + if (!distinct.contains(results[i])) { + distinct.add(results[i]); + } + } + return distinct; + } + + private static boolean allParked(List threads) { + for (Thread t : threads) { + Thread.State s = t.getState(); + if (t.isAlive() && s != Thread.State.TIMED_WAITING && s != Thread.State.WAITING) { + return false; + } + } + return true; + } + + private static List listOf(String s) { + List l = new ArrayList<>(); + l.add(s); + return l; + } + + private static RefreshingTokenProvider.Builder provider( + ScriptedSource source, + FakeClock clock, + ManualScheduler scheduler, + double u + ) { + return RefreshingTokenProvider.builder(source) + .clock(clock) + .scheduler(scheduler) + .random(() -> u); + } + + private static List refresherThreads() { + List threads = new ArrayList<>(); + for (Thread t : Thread.getAllStackTraces().keySet()) { + if (t.getName().startsWith("qdb-token-refresh-") && t.isAlive()) { + threads.add(t); + } + } + return threads; + } + + private static String repeat(char c, int n) { + StringBuilder sb = new StringBuilder(n); + for (int i = 0; i < n; i++) { + sb.append(c); + } + return sb.toString(); + } + + private static Thread start(Runnable r) { + Thread t = new Thread(r, "token-test-caller"); + t.setDaemon(true); + t.start(); + return t; + } + + // Real monotonic time, with a wall clock the test can push forward. + private static final class SkewedClock implements RefreshingTokenProvider.Clock { + volatile long skewMillis; + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis() + skewMillis; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java new file mode 100644 index 000000000..357d80ad8 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java @@ -0,0 +1,100 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenSource; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.Function; + +/** + * A {@link TokenProviderFactory} for tests, installed in place of {@link java.util.ServiceLoader} discovery with + * {@link #install(TestTokenProviderFactory...)}. It counts the sources it creates and the fetches they make. + * Registering one named {@code azure} lets core tests drive the {@code azure} connect-string keys without the + * optional module. + */ +public final class TestTokenProviderFactory implements TokenProviderFactory { + public final AtomicInteger created = new AtomicInteger(); + public final AtomicInteger fetches = new AtomicInteger(); + public final List> params = new CopyOnWriteArrayList<>(); + private final String name; + private final Function, ExpiringToken> tokens; + + public TestTokenProviderFactory(String name) { + this(name, p -> new ExpiringToken("TOKEN-" + name, System.currentTimeMillis() + 3_600_000L)); + } + + public TestTokenProviderFactory(String name, Function, ExpiringToken> tokens) { + this.name = name; + this.tokens = tokens; + } + + /** + * Makes exactly these factories visible to {@code token_provider} resolution. + */ + public static void install(TestTokenProviderFactory... factories) { + List list = new ArrayList<>(); + Collections.addAll(list, factories); + TokenProviderRegistry.setFactoriesForTesting(list); + } + + /** + * Restores {@link java.util.ServiceLoader} discovery. + */ + public static void uninstall() { + TokenProviderRegistry.setFactoriesForTesting(null); + } + + @Override + public TokenSource createSource(Map params) { + created.incrementAndGet(); + this.params.add(params); + return () -> { + fetches.incrementAndGet(); + return tokens.apply(params); + }; + } + + @Override + public String name() { + return name; + } + + @Override + public void validate(Map params) { + String resource = params.get("azure_resource"); + if (resource != null && resource.startsWith("invalid:")) { + throw new IllegalArgumentException("azure_resource rejected by the factory"); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java new file mode 100644 index 000000000..bfe077344 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java @@ -0,0 +1,255 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.Map; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C18 (connect-string validation) and C19 (registry lifecycle) of the dynamic-credential + * specification (design/qwp-token-provider-spec.md, sections 7.2, 7.4 and 10). + */ +public class TokenProviderConfigTest { + private static final String WSS = "wss::addr=localhost:9000;"; + private TestTokenProviderFactory azure; + private TestTokenProviderFactory other; + + @Before + public void setUp() { + azure = new TestTokenProviderFactory("azure"); + other = new TestTokenProviderFactory("vault"); + TestTokenProviderFactory.install(azure, other); + } + + @After + public void tearDown() { + TestTokenProviderFactory.uninstall(); + } + + @Test + public void testAzureKeysAreNormalized() { + Map snap = Sender.builder(WSS + "token_provider=azure;azure_resource=api://qdb-app/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;").wsConfigSnapshotForTest(); + Assert.assertEquals("azure", snap.get("token_provider")); + Assert.assertEquals("a trailing /.default must be stripped", "api://qdb-app", snap.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", snap.get("azure_client_id")); + + // equivalent strings share a registry key; a different identity does not + Assert.assertEquals(spec("token_provider=azure;azure_resource=api://qdb-app/.default;").registryKey(), + spec("token_provider=azure;azure_resource=api://qdb-app;").registryKey()); + Assert.assertNotEquals(spec("token_provider=azure;azure_resource=api://qdb-app;").registryKey(), + spec("token_provider=azure;azure_resource=api://qdb-app;" + + "azure_client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee;").registryKey()); + } + + @Test + public void testBothClientsAcceptTheKeys() throws Exception { + assertMemoryLeak(() -> { + String cfg = WSS + "token_provider=azure;azure_resource=api://qdb-app;"; + Assert.assertEquals("azure", Sender.builder(cfg).wsConfigSnapshotForTest().get("token_provider")); + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + Map snap = client.configSnapshotForTest(); + Assert.assertEquals("azure", snap.get("token_provider")); + Assert.assertEquals("api://qdb-app", snap.get("azure_resource")); + } + }); + } + + @Test + public void testRegistryLifecycle() { + // C19: lease release, linger, and re-acquiring a lease during the linger period. + TokenProviderRegistry registry = new TokenProviderRegistry(300); + TokenProviderSpec spec = spec("token_provider=vault;"); + TokenProviderRegistry.Lease a = registry.acquire(spec); + TokenProviderRegistry.Lease b = registry.acquire(spec); + RefreshingTokenProvider provider = a.provider(); + Assert.assertSame("one provider per configuration", provider, b.provider()); + Assert.assertEquals(1, other.created.get()); + Assert.assertEquals(2, registry.leaseCount(spec)); + Assert.assertTrue(provider.awaitReady(10_000)); + + a.close(); + a.close(); // releasing twice is harmless + Assert.assertEquals(1, registry.leaseCount(spec)); + b.close(); + Assert.assertEquals(0, registry.leaseCount(spec)); + Assert.assertTrue("the provider lingers after its last lease", registry.isActive(spec)); + Assert.assertFalse(provider.isClosed()); + + // re-acquired during the linger: the same warm provider, and the pending close is cancelled + TokenProviderRegistry.Lease c = registry.acquire(spec); + Assert.assertSame(provider, c.provider()); + Assert.assertEquals(1, other.created.get()); + c.close(); + await(provider::isClosed, 10_000, "the provider to close once the linger elapses"); + Assert.assertFalse(registry.isActive(spec)); + + // the next lease starts a fresh provider + TokenProviderRegistry.Lease d = registry.acquire(spec); + Assert.assertNotSame(provider, d.provider()); + Assert.assertEquals(2, other.created.get()); + d.close(); + await(d.provider()::isClosed, 10_000, "the second provider to close"); + } + + @Test + public void testRegistryWithoutLingerClosesOnTheLastRelease() { + TokenProviderRegistry registry = new TokenProviderRegistry(0); + TokenProviderSpec spec = spec("token_provider=vault;"); + TokenProviderRegistry.Lease lease = registry.acquire(spec); + RefreshingTokenProvider provider = lease.provider(); + lease.close(); + Assert.assertTrue(provider.isClosed()); + Assert.assertFalse(registry.isActive(spec)); + } + + @Test + public void testRegistrySharesOnlyEquivalentConfigurations() { + TokenProviderRegistry registry = new TokenProviderRegistry(0); + TokenProviderRegistry.Lease a = registry.acquire(spec("token_provider=azure;azure_resource=api://one;")); + TokenProviderRegistry.Lease b = registry.acquire(spec("token_provider=azure;azure_resource=api://one/.default;")); + TokenProviderRegistry.Lease c = registry.acquire(spec("token_provider=azure;azure_resource=api://two;")); + try { + Assert.assertSame(a.provider(), b.provider()); + Assert.assertNotSame(a.provider(), c.provider()); + Assert.assertEquals(2, azure.created.get()); + Assert.assertTrue(a.provider().getName(), a.provider().getName().startsWith("azure")); + } finally { + a.close(); + b.close(); + c.close(); + } + } + + @Test + public void testValidationRules() throws Exception { + // C18: every rule of section 7.2, on both clients, naming the offending key + assertMemoryLeak(() -> { + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=x;token=abc;", + "token_provider cannot be combined with token"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=x;username=u;password=p;", + "token_provider cannot be combined with username"); + assertRejectedOnBoth(WSS + "token_provider=;", "token_provider must not be empty", "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "token_provider=kerberos;", "unsupported token_provider: kerberos", + "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "token_provider=azure_imds;", "azure_imds is reserved", + "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "azure_resource=api://x;", "azure_resource requires token_provider=azure"); + assertRejectedOnBoth(WSS + "azure_client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee;", + "azure_client_id requires token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=vault;azure_resource=api://x;", + "azure_resource is only valid with token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=azure;", "token_provider=azure requires azure_resource"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=/.default;", "azure_resource must not be empty"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_client_id=not-a-guid;", + "invalid azure_client_id"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=invalid:x;", "rejected by the factory"); + // decision D4: never over plain ws:: + assertRejectedOnBoth("ws::addr=localhost:9000;token_provider=azure;azure_resource=api://x;", + "token_provider requires the wss:: schema"); + // decision D9: never on a non-QWP schema + assertRejected(() -> Sender.fromConfig("http::addr=localhost:9000;token_provider=azure;"), + "token_provider is only supported with the wss:: schema"); + assertRejected(() -> Sender.fromConfig("https::addr=localhost:9000;azure_resource=api://x;"), + "azure_resource is only supported with the wss:: schema"); + + // an application-supplied provider is exclusive with token_provider + String cfg = WSS + "token_provider=azure;azure_resource=api://x;"; + assertRejected(() -> Sender.builder(cfg).httpTokenProvider(() -> "t"), + "application-supplied token provider cannot be combined with token_provider"); + assertRejected(() -> Sender.builder(cfg).httpToken("t"), "token cannot be combined with token_provider"); + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + assertRejected(() -> client.withBearerTokenProvider(() -> "t"), + "withBearerTokenProvider cannot be combined with token_provider"); + assertRejected(() -> client.withBearerToken("t"), "withBearerToken cannot be combined with token_provider"); + } + assertRejected(() -> QuestDB.builder().fromConfig(cfg).httpTokenProvider(() -> "t").build(), + "httpTokenProvider cannot be combined with token_provider"); + }); + } + + @Test + public void testValidationWithoutConnectingFetchesNoToken() throws Exception { + // section 7.2: validation that does not connect - a facade checking its configuration at build time, a + // parsed builder, a client that has not connected - must not fetch a token. + assertMemoryLeak(() -> { + String cfg = WSS + "token_provider=azure;azure_resource=api://validate-only;" + + "sender_pool_min=0;query_pool_min=0;"; + Sender.builder(cfg); + QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(cfg)), true); + QwpQueryClient.fromConfig(cfg).close(); + try (QuestDB ignored = QuestDB.connect(cfg)) { + Assert.assertEquals("no provider may be started", 0, azure.created.get()); + } + Assert.assertEquals(0, azure.created.get()); + Assert.assertEquals(0, azure.fetches.get()); + }); + } + + @Test + public void testWithoutTheModuleAzureIsRejectedWithAHint() { + TestTokenProviderFactory.install(); // nothing installed + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;", + "token_provider=azure requires the org.questdb:questdb-client-azure module", + "supported values: none installed"); + } + + private static void assertRejected(Runnable action, String... fragments) { + try { + action.run(); + Assert.fail("expected the configuration to be rejected with: " + fragments[0]); + } catch (IllegalArgumentException | IllegalStateException | LineSenderException e) { + for (String fragment : fragments) { + Assert.assertTrue("[" + e.getMessage() + "] does not contain [" + fragment + ']', + e.getMessage().contains(fragment)); + } + } + } + + private static void assertRejectedOnBoth(String cfg, String... fragments) { + assertRejected(() -> Sender.builder(cfg), fragments); + assertRejected(() -> QwpQueryClient.fromConfig(cfg).close(), fragments); + } + + private static TokenProviderSpec spec(String keys) { + return TokenProviderSpec.parse(new ConfigView(ConfigString.parse(WSS + keys)), true); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java new file mode 100644 index 000000000..41f3d1bc5 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java @@ -0,0 +1,269 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenSource; +import org.junit.Assert; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.BooleanSupplier; +import java.util.function.Supplier; + +/** + * Deterministic fixtures for the dynamic-credential conformance tests (design/qwp-token-provider-spec.md, section + * 10): a fake clock, a scheduler the test drives by hand, and a scripted token source. + */ +public final class TokenTestKit { + + private TokenTestKit() { + } + + public static void await(BooleanSupplier condition, long timeoutMillis, String what) { + final long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeoutMillis); + while (!condition.getAsBoolean()) { + if (System.nanoTime() - deadline > 0) { + Assert.fail("timed out after " + timeoutMillis + " ms waiting for " + what); + } + try { + Thread.sleep(2); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + Assert.fail("interrupted while waiting for " + what); + } + } + } + + /** + * A wall clock and a monotonic clock that only move when the test moves them. {@link #advanceMillis(long)} + * moves both; {@link #jumpWallMillis(long)} moves the wall clock alone, the way a suspended host or an NTP + * step does. + */ + public static final class FakeClock implements RefreshingTokenProvider.Clock { + private final AtomicLong mono = new AtomicLong(1_000_000_000L); + private final AtomicLong wall; + + public FakeClock(long wallStartMillis) { + this.wall = new AtomicLong(wallStartMillis); + } + + public void advanceMillis(long millis) { + wall.addAndGet(millis); + mono.addAndGet(TimeUnit.MILLISECONDS.toNanos(millis)); + } + + public void jumpWallMillis(long millis) { + wall.addAndGet(millis); + } + + @Override + public long monotonicNanos() { + return mono.get(); + } + + @Override + public long wallClockMillis() { + return wall.get(); + } + } + + /** + * Runs nothing on its own: the test picks the next task, optionally advances the fake clock to its due time, + * and runs it on the test thread. Tasks run outside the scheduler's monitor, because the provider calls + * {@link #schedule} while holding its own lock. + */ + public static final class ManualScheduler implements RefreshingTokenProvider.Scheduler { + private final FakeClock clock; + private final List tasks = new ArrayList<>(); + private volatile boolean shutdown; + + public ManualScheduler(FakeClock clock) { + this.clock = clock; + } + + /** + * Advances the clock to the next task's due time (if it lies ahead) and runs it. + * + * @return the task that ran + */ + public Task advanceAndRunNext() { + Task t = takeNext(); + Assert.assertNotNull("no fetch is scheduled", t); + long ahead = t.dueNanos - clock.monotonicNanos(); + if (ahead > 0) { + clock.advanceMillis(TimeUnit.NANOSECONDS.toMillis(ahead)); + } + t.runnable.run(); + return t; + } + + public boolean isShutdown() { + return shutdown; + } + + /** + * The earliest live task, without removing it, or null. + */ + public synchronized Task peek() { + Task best = null; + for (int i = 0, n = tasks.size(); i < n; i++) { + Task t = tasks.get(i); + if (!t.cancelled && (best == null || t.dueNanos - best.dueNanos < 0)) { + best = t; + } + } + return best; + } + + /** + * Runs the earliest live task now, without moving the clock. + */ + public Task runNext() { + Task t = takeNext(); + Assert.assertNotNull("no fetch is scheduled", t); + t.runnable.run(); + return t; + } + + @Override + public synchronized Task schedule(Runnable task, long delayNanos) { + Task t = new Task(task, clock.monotonicNanos() + delayNanos, delayNanos); + tasks.add(t); + return t; + } + + @Override + public void shutdown() { + shutdown = true; + } + + private synchronized Task takeNext() { + Task t = peek(); + if (t != null) { + tasks.remove(t); + } + return t; + } + + public static final class Task implements RefreshingTokenProvider.ScheduledTask { + public final long delayNanos; + final long dueNanos; + final Runnable runnable; + volatile boolean cancelled; + + Task(Runnable runnable, long dueNanos, long delayNanos) { + this.runnable = runnable; + this.dueNanos = dueNanos; + this.delayNanos = delayNanos; + } + + @Override + public void cancel() { + cancelled = true; + } + + public long delayMillis() { + return TimeUnit.NANOSECONDS.toMillis(delayNanos); + } + } + } + + /** + * A token source that replays a script of results. Each step either returns a token, throws, or computes + * its result from the clock at call time. When the script runs out the last step repeats. An optional gate + * blocks every fetch until the test opens it. + */ + public static final class ScriptedSource implements TokenSource { + private final AtomicInteger calls = new AtomicInteger(); + private final ArrayDeque> script = new ArrayDeque<>(); + private volatile CountDownLatch gate; + private Supplier last; + + public int calls() { + return calls.get(); + } + + @Override + public ExpiringToken fetchToken() { + calls.incrementAndGet(); + CountDownLatch g = gate; + if (g != null) { + try { + g.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("fetch interrupted"); + } + } + Supplier step; + synchronized (this) { + step = script.poll(); + if (step == null) { + step = last; + } else { + last = step; + } + } + Assert.assertNotNull("the source was called with nothing scripted", step); + return step.get(); + } + + public void openGate() { + CountDownLatch g = gate; + gate = null; + if (g != null) { + g.countDown(); + } + } + + public void setGate() { + gate = new CountDownLatch(1); + } + + public ScriptedSource then(Supplier step) { + synchronized (this) { + script.add(step); + } + return this; + } + + public ScriptedSource thenThrow(RuntimeException e) { + return then(() -> { + throw e; + }); + } + + public ScriptedSource thenToken(String token, long expiresAtEpochMillis) { + return then(() -> new ExpiringToken(token, expiresAtEpochMillis)); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java new file mode 100644 index 000000000..0fdae50d9 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java @@ -0,0 +1,94 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.http; + +import io.questdb.client.cutlass.http.BearerChallenge; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import org.junit.Assert; +import org.junit.Test; + +public class BearerChallengeTest { + + @Test + public void testAuthFailedExceptionClassification() { + QwpAuthFailedException plain = new QwpAuthFailedException(401, "h", 1); + Assert.assertTrue(plain.isTokenRefreshable()); + Assert.assertNull(plain.getBearerError()); + Assert.assertEquals("auth-rejected: WebSocket upgrade rejected with HTTP 401 for h:1", plain.getMessage()); + + QwpAuthFailedException invalid = new QwpAuthFailedException(401, "h", 1, "Bearer error=\"invalid_token\""); + Assert.assertTrue(invalid.isTokenRefreshable()); + Assert.assertEquals("auth-rejected: WebSocket upgrade rejected with HTTP 401 [error=invalid_token] for h:1", + invalid.getMessage()); + + QwpAuthFailedException scope = new QwpAuthFailedException(401, "h", 1, "Bearer error=\"insufficient_scope\""); + Assert.assertFalse(scope.isTokenRefreshable()); + + // decision D2: a 403 never qualifies, challenge or not + Assert.assertFalse(new QwpAuthFailedException(403, "h", 1).isTokenRefreshable()); + Assert.assertFalse(new QwpAuthFailedException(403, "h", 1, "Bearer error=\"invalid_token\"").isTokenRefreshable()); + + // a hostile error value is sanitized and capped before it reaches the message + QwpAuthFailedException hostile = new QwpAuthFailedException(401, "h", 1, + "Bearer error=\"x\r\nForged: 1\u202E" + new String(new char[200]).replace('\0', 'y') + "\""); + Assert.assertFalse(hostile.getMessage().contains("\r") || hostile.getMessage().contains("\u202E")); + Assert.assertTrue(hostile.getBearerError().length() <= 64); + } + + @Test + public void testBearerError() { + Assert.assertNull(BearerChallenge.bearerError(null)); + Assert.assertNull(BearerChallenge.bearerError("")); + Assert.assertNull(BearerChallenge.bearerError("Bearer")); + Assert.assertNull(BearerChallenge.bearerError("Bearer realm=\"questdb\"")); + Assert.assertEquals("invalid_token", + BearerChallenge.bearerError("Bearer realm=\"questdb\", error=\"invalid_token\", error_description=\"The access token expired\"")); + Assert.assertEquals("insufficient_scope", BearerChallenge.bearerError("Bearer error=insufficient_scope")); + Assert.assertEquals("invalid_token", BearerChallenge.bearerError("bearer ERROR=\"invalid_token\"")); + // a Bearer challenge after another scheme, with token68 and quoted commas in between + Assert.assertEquals("invalid_token", BearerChallenge.bearerError( + "Negotiate YIIB9QYGKwYBBQUCoIIB==, Basic realm=\"a, b\", Bearer error=\"invalid_token\"")); + // an error parameter of another scheme does not count + Assert.assertNull(BearerChallenge.bearerError("Basic error=\"invalid_token\"")); + Assert.assertNull(BearerChallenge.bearerError("Basic realm=\"x\", DPoP error=\"invalid_token\"")); + // escaped quotes inside a quoted value + Assert.assertEquals("a\"b", BearerChallenge.bearerError("Bearer error=\"a\\\"b\"")); + // only the first error counts + Assert.assertEquals("invalid_token", BearerChallenge.bearerError("Bearer error=invalid_token, error=other")); + // garbage does not throw + Assert.assertNull(BearerChallenge.bearerError("\u0000\"=,,=\"Bearer")); + } + + @Test + public void testRefreshDecision() { + // section 8.2: without a Bearer challenge carrying an error, the status code alone decides + Assert.assertTrue(BearerChallenge.allowsTokenRefresh(null)); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Basic realm=\"questdb\"")); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Bearer realm=\"questdb\"")); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Bearer error=\"invalid_token\"")); + Assert.assertFalse(BearerChallenge.allowsTokenRefresh("Bearer error=\"insufficient_scope\"")); + Assert.assertFalse(BearerChallenge.allowsTokenRefresh("Bearer error=\"invalid_request\"")); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java new file mode 100644 index 000000000..b8c8b487e --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -0,0 +1,431 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.ConnectionHealth; +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C21 (connection health, section 8.4) and C22 (the authentication-outage deadline, section + * 8.5) of the dynamic-credential specification (design/qwp-token-provider-spec.md). + */ +public class ConnectionHealthTest { + private static final String SENTINEL = "SENTINEL-TOKEN-7f3a91"; + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 60_000) + public void testDeadlineIsNotResetOrFiredByOtherFailureClasses() throws Exception { + // C22: rounds that fail for other reasons neither reset the clock nor fire the deadline. A credential + // outage, then role rejects and a transport outage that together outlast the deadline, then the + // credential outage again: the deadline fires on the first authentication-class round after them. + assertMemoryLeak(() -> { + final long deadlineMillis = 1_000; + AtomicReference mode = new AtomicReference<>("ok"); + ErrorCollector errors = new ErrorCollector(); + TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler()); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .authFailureMaxDurationMillis(deadlineMillis) + .errorHandler(errors) + .httpTokenProvider(() -> { + if ("fail".equals(mode.get())) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return "TOKEN"; + }) + .build()) { + mode.set("fail"); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + await(() -> errors.count("credential-unavailable") >= 1, 10_000, "the credential outage"); + long authOutageStart = System.nanoTime(); + // Other classes: every endpoint role-rejects (set before the provider recovers, so no upgrade can + // succeed in between and reset the clock), then the server goes away (transport). + server.setRejectWithRole("REPLICA"); + mode.set("ok"); + Thread.sleep(deadlineMillis * 2 / 3); + server.close(); + Thread.sleep(deadlineMillis * 2 / 3); + Assert.assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - authOutageStart) > deadlineMillis); + Assert.assertEquals("role rejects and transport failures must not fire the deadline", + 0, errors.terminalCount()); + // the credential outage returns: the clock was never reset, so the first such round fires + mode.set("fail"); + await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); + SenderError terminal = terminal(errors); + Assert.assertTrue(terminal.getServerMessage(), + terminal.getServerMessage().contains("credential-unavailable persisted for")); + Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); + } catch (LineSenderException expected) { + // close() rethrows the latched terminal + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("authentication outage deadline")); + } finally { + server.close(); + } + }); + } + + @Test(timeout = 60_000) + public void testDeadlineMakesTheSenderTerminalAndKeepsTheData() throws Exception { + // C22: set, the sender becomes terminal on an authentication-class round once the duration has passed; + // the error names the class and the elapsed time, is reported as terminal and thrown from later producer + // calls, and the unacknowledged rows stay on disk. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("deadline").getAbsolutePath(); + ErrorCollector errors = new ErrorCollector(); + AtomicReference rejectWith = new AtomicReference<>(0); + try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler())) { + server.setAuthorizationValidator(h -> rejectWith.get()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("deadline") + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .closeFlushTimeoutMillis(0) + .authFailureMaxDurationMillis(500) + .errorHandler(errors) + .httpTokenProvider(() -> SENTINEL) + .build(); + try { + rejectWith.set(401); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then dropped: every reconnect is now rejected + await(() -> server.authRejectCount() >= 1, 10_000, "the first 401"); + long start = System.nanoTime(); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); // buffered in store-and-forward + await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must not fire before the deadline, fired at " + elapsedMillis + " ms", + elapsedMillis >= 400); + SenderError terminal = terminal(errors); + Assert.assertEquals(SenderError.Category.SECURITY_ERROR, terminal.getCategory()); + Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("auth-rejected")); + Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("persisted for")); + Assert.assertFalse(terminal.getServerMessage().contains(SENTINEL)); + try { + sender.table("t").longColumn("v", 3).atNow(); + sender.flush(); + Assert.fail("later producer calls must throw the terminal"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("authentication outage deadline")); + } + Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); + } finally { + try { + sender.close(); + } catch (LineSenderException ignored) { + // close() may rethrow the terminal + } + } + } + Assert.assertEquals("the unacknowledged rows stay on disk for a later sender or an orphan drain", + 1, OrphanScanner.scan(sfDir, "someone-else").size()); + }); + } + + @Test(timeout = 60_000) + public void testDeadlineIsResetByASuccessfulUpgradeAndUnsetMeansNoDeadline() throws Exception { + // C22: a successful upgrade resets the clock; and without the key, a 401 outage is retried indefinitely. + assertMemoryLeak(() -> { + for (boolean armed : new boolean[]{true, false}) { + ErrorCollector errors = new ErrorCollector(); + AtomicReference rejectWith = new AtomicReference<>(0); + WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setAuthorizationValidator(h -> rejectWith.get()); + Sender.LineSenderBuilder b = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(() -> "TOKEN"); + if (armed) { + b.authFailureMaxDurationMillis(800); + } + try (Sender sender = b.build()) { + for (int outage = 0; outage < 2; outage++) { + rejectWith.set(401); + server.dropAllConnections(); + Thread.sleep(500); // under the deadline + rejectWith.set(0); // a successful upgrade ends the outage + final long acked = handler.frames.get(); + sender.table("t").longColumn("v", outage).atNow(); + sender.flush(); + await(() -> handler.frames.get() > acked, 10_000, "recovery"); + } + Assert.assertEquals("two outages of 500 ms each, the clock reset in between [armed=" + armed + ']', + 0, errors.terminalCount()); + if (!armed) { + rejectWith.set(401); + server.dropAllConnections(); + Thread.sleep(1_500); + Assert.assertEquals("no deadline: retried indefinitely", 0, errors.terminalCount()); + Assert.assertTrue(errors.count("auth-rejected") > 0); + rejectWith.set(0); + } + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testFacadeAggregatesEveryPooledConnection() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new ExecDoneHandler())) { + server.setSendServerInfo(true); + String cfg = "ws::addr=localhost:" + server.getPort() + ";sender_pool_min=2;query_pool_min=1;"; + try (QuestDB db = QuestDB.connect(cfg)) { + ConnectionHealth.Aggregate health = db.health(); + Assert.assertEquals(3, health.total()); + Assert.assertEquals(3, health.count(ConnectionHealth.State.CONNECTED)); + Assert.assertEquals(ConnectionHealth.NONE, health.getOldestOutageSinceEpochMillis()); + Assert.assertNull(health.getLastFailure()); + try (Sender s = db.borrowSender()) { + Assert.assertEquals(ConnectionHealth.State.CONNECTED, s.health().getState()); + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testQueryClientHealthFollowsConnectFailoverAndRecovery() throws Exception { + // C21 for egress: connecting, connected, reconnecting after a failed failover - with the class of the + // failure - and connected again on the next operation; closed at the end. + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(new ExecDoneHandler()); + TestWebSocketServer b = startServer(new ExecDoneHandler())) { + a.setSendServerInfo(true); + b.setSendServerInfo(true); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + a.getPort() + + ",localhost:" + b.getPort() + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> SENTINEL); + try { + ConnectionHealth h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, h.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + client.connect(); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + + b.setAuthorizationValidator(x -> 403); + a.close(); + client.execute("SELECT 1", new NoopHandler()); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertEquals(1, h.getFailedRounds()); + Assert.assertEquals(ConnectionHealth.FailureClass.AUTH_REJECTED, h.getLastFailure().getFailureClass()); + Assert.assertEquals(403, h.getLastFailure().getStatusCode()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertFalse(h.toString().contains(SENTINEL)); + + b.setAuthorizationValidator(null); + client.execute("SELECT 1", new NoopHandler()); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertEquals("last_failure is kept after recovery", + ConnectionHealth.FailureClass.AUTH_REJECTED, h.getLastFailure().getFailureClass()); + } finally { + client.close(); + } + Assert.assertEquals(ConnectionHealth.State.CLOSED, client.health().getState()); + } + }); + } + + @Test(timeout = 60_000) + public void testSenderHealthMovesThroughEveryState() throws Exception { + // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> + // connected. outage_since and failed_rounds reset on recovery; last_failure is kept; no credential appears. + assertMemoryLeak(() -> { + AtomicReference providerMode = new AtomicReference<>("fail"); + AtomicReference rejectWith = new AtomicReference<>(0); + WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setAuthorizationValidator(h -> rejectWith.get()); + long created = System.currentTimeMillis(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.ASYNC) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + if ("fail".equals(providerMode.get())) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return SENTINEL; + }) + .build(); + try { + // connecting, with credential-unavailable rounds + await(() -> sender.health().getFailedRounds() >= 2, 10_000, "failed rounds while connecting"); + ConnectionHealth h = sender.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, h.getState()); + Assert.assertTrue(h.getOutageSinceEpochMillis() >= created - 1_000); + Assert.assertEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + Assert.assertEquals(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, + h.getLastFailure().getFailureClass()); + Assert.assertTrue(h.getLastFailure().getMessage(), h.getLastFailure().getMessage().contains("IdP unreachable")); + + // connected + providerMode.set("ok"); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "connected"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + Assert.assertEquals("last_failure is kept after recovery", + ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, h.getLastFailure().getFailureClass()); + + // reconnecting, rejected with 401 + rejectWith.set(401); + server.dropAllConnections(); + await(() -> sender.health().getState() == ConnectionHealth.State.RECONNECTING + && sender.health().getLastFailure().getFailureClass() == ConnectionHealth.FailureClass.AUTH_REJECTED, + 10_000, "reconnecting with 401"); + h = sender.health(); + Assert.assertEquals(401, h.getLastFailure().getStatusCode()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + long outageSince = h.getOutageSinceEpochMillis(); + + // still reconnecting, now the provider fails + providerMode.set("fail"); + await(() -> sender.health().getLastFailure().getFailureClass() + == ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, 10_000, "credential-unavailable"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertEquals("the outage did not restart", outageSince, h.getOutageSinceEpochMillis()); + Assert.assertTrue(h.getFailedRounds() >= 2); + Assert.assertFalse("no credential may appear: " + h, h.toString().contains(SENTINEL)); + + // back to connected + providerMode.set("ok"); + rejectWith.set(0); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "recovered"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertEquals(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, h.getLastFailure().getFailureClass()); + Assert.assertFalse(h.toString().contains(SENTINEL)); + } finally { + sender.close(); + } + Assert.assertEquals(ConnectionHealth.State.CLOSED, sender.health().getState()); + } + }); + } + + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) + throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(handler); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private static SenderError terminal(ErrorCollector errors) { + for (SenderError e : errors.errors) { + if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { + return e; + } + } + throw new AssertionError("no terminal error"); + } + + private static final class ExecDoneHandler implements TestWebSocketServer.WebSocketServerHandler { + private final WebSocketDynamicCredentialTest.AckHandler ack = new WebSocketDynamicCredentialTest.AckHandler(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length > 0 && data[0] == QwpEgressMsgKind.QUERY_REQUEST) { + int bodyLen = 1 + 8 + 1 + 1; + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1).put((byte) 0).putShort((short) 0).putInt(bodyLen); + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(data, 1, 8); + bb.put((byte) 0).put((byte) 0); + try { + client.sendBinary(frame); + } catch (IOException ignored) { + // surfaces to the client as a transport error + } + } else { + ack.onBinaryMessage(client, data); + } + } + } + + private static final class NoopHandler implements QwpColumnBatchHandler { + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java new file mode 100644 index 000000000..addb4f52e --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java @@ -0,0 +1,349 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Query; +import io.questdb.client.QueryException; +import io.questdb.client.QuestDB; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpCredentialUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * The dynamic-credential specification (design/qwp-token-provider-spec.md) on the QWP egress client: the one + * retry after a 401 on connect and on failover reconnect (section 8.2, C10/C11/C23), failure-class naming in + * the reported error, and egress recovery - a client whose failover reconnect failed reconnects on its next + * operation rather than staying unusable (section 8.3). + */ +public class QwpQueryClientDynamicCredentialTest { + private static final long HOUR = 3_600_000L; + + @Test(timeout = 30_000) + public void testConnectCredentialUnavailableNamesTheClassAndContactsNoEndpoint() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(() -> { + throw TokenUnavailableException.retryable("IMDS unreachable"); + })) { + try { + client.connect(); + Assert.fail(); + } catch (QwpCredentialUnavailableException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("credential-unavailable: ")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("IMDS unreachable")); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getCause() instanceof TokenUnavailableException); + } + Assert.assertEquals("no endpoint may be contacted", 0, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectInsufficientScopeIsNotRetried() throws Exception { + // C23 on egress: a Bearer challenge with an error other than invalid_token gets no retry. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate("Bearer error=\"insufficient_scope\""); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals("insufficient_scope", e.getBearerError()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + Assert.assertEquals(1, source.calls()); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectPersistent401IsRetriedOnceThenFails() throws Exception { + // C11 on egress: the failure survives the one retry and fails the operation. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> 401); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(server)); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectStaleTokenIsRetriedOnceWithTheRefreshedToken() throws Exception { + // C10 on egress: one 401 for the stale token, then the same endpoint with the refreshed one. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + client.connect(); + Assert.assertTrue(client.isConnected()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(server)); + Assert.assertEquals(1, server.authRejectCount()); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNull(result.error.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailedFailoverReconnectRecoversOnTheNextExecute() throws Exception { + // Egress recovery (section 8.3): a failover reconnect that fails must not leave the client permanently + // unusable. Before the fix, every later execute() threw "QwpQueryClient not connected". + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> "TOKEN")) { + client.connect(); + ResultCollector first = new ResultCollector(); + client.execute("SELECT 1", first); + Assert.assertNull(first.error.get()); + + // A dies; B rejects the credential: the failover reconnect fails, naming the class + b.setAuthorizationValidator(h -> 401); + a.close(); + ResultCollector failed = new ResultCollector(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the operation must fail", failed.error.get()); + Assert.assertTrue(failed.error.get(), failed.error.get().contains("auth-rejected")); + Assert.assertFalse(client.isConnected()); + + // B accepts again: the next operation reconnects instead of failing forever + b.setAuthorizationValidator(null); + ResultCollector recovered = new ResultCollector(); + client.execute("SELECT 1", recovered); + Assert.assertNull("the next execute() must reconnect: " + recovered.error.get(), recovered.error.get()); + Assert.assertTrue(recovered.done.get() > 0); + Assert.assertTrue(client.isConnected()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailoverCredentialUnavailableNamesTheClass() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> { + if (pulls.incrementAndGet() > 1) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return "TOKEN"; + })) { + client.connect(); + a.close(); + ResultCollector failed = new ResultCollector(); + client.execute("SELECT 1", failed); + Assert.assertNotNull(failed.error.get()); + Assert.assertTrue(failed.error.get(), failed.error.get().contains("credential-unavailable: IdP unreachable")); + Assert.assertEquals("no endpoint contacted on the failed failover", 0, b.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailoverReconnectRetriesAStaleTokenOnce() throws Exception { + // C10 on the egress failover path: B rejects the stale token; the failover reconnect refreshes and + // retries B once, and the query completes. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(provider)) { + b.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + client.connect(); + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + a.close(); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNull(result.error.get()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(b)); + } + }); + } + + @Test(timeout = 60_000) + public void testPooledQueryClientRecoversAfterAFailedFailover() throws Exception { + // The suspected dead-pooled-worker bug (design/entra-id-qwp-auth.md, section 3): a pooled query client + // whose failover reconnect failed went back to the pool still disconnected, and with query_pool_min + // keeping it alive every later borrow failed with "QwpQueryClient not connected". + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer()) { + String cfg = "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";sender_pool_min=0;query_pool_min=1;query_pool_max=1;failover_backoff_initial_ms=0;"; + try (QuestDB db = QuestDB.connect(cfg, () -> "TOKEN")) { + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + } + b.setAuthorizationValidator(h -> 401); + a.close(); + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + Assert.fail("the failover reconnect must fail"); + } catch (QueryException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("auth-rejected")); + } + b.setAuthorizationValidator(null); + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + } catch (QueryException e) { + Assert.fail("the pooled client must reconnect on its next operation: " + e.getMessage()); + } + } + } + }); + } + + private static byte[] buildExecDone(byte[] queryRequest) { + int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1); // version + bb.put((byte) 0); // flags + bb.putShort((short) 0); // table_count + bb.putInt(bodyLen); // payload_length + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(queryRequest, 1, 8); // echo request_id verbatim + bb.put((byte) 0); // op_type + bb.put((byte) 0); // rows_affected = 0 + return frame; + } + + private static List headers(TestWebSocketServer server) throws InterruptedException { + List headers = new ArrayList<>(); + String h; + while ((h = server.pollAuthorizationHeader(200, TimeUnit.MILLISECONDS)) != null) { + headers.add(h); + } + return headers; + } + + private static TestWebSocketServer startServer() throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(new ExecDoneQueryServer()); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private static ScriptedSource twoTokens() { + return new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + } + + private static final class ExecDoneQueryServer implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length == 0 || data[0] != QwpEgressMsgKind.QUERY_REQUEST) { + return; + } + try { + client.sendBinary(buildExecDone(data)); + } catch (IOException e) { + // best-effort: a failed reply surfaces to the client as a transport error + } + } + } + + private static final class ResultCollector implements QwpColumnBatchHandler { + final AtomicInteger done = new AtomicInteger(); + final AtomicReference error = new AtomicReference<>(); + + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + done.incrementAndGet(); + } + + @Override + public void onError(byte status, String message) { + error.set(message); + } + + @Override + public void onExecDone(short opType, long rowsAffected) { + done.incrementAndGet(); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java index d0eb80db0..4d463fa7e 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java @@ -239,13 +239,14 @@ public void testExceptionPathStillClosesAndWalksAllEndpoints() throws Exception * needed. The connect walk dereferences only the fields wired below plus * primitives whose zero-defaults are valid here (field initializers do * not run under {@code Unsafe.allocateInstance}), plus the connect-walk - * lock, which buildAndConnect acquires unconditionally and is therefore - * wired here. + * lock and the connection-health tracker, which buildAndConnect uses + * unconditionally on a foreground walk and are therefore wired here. */ private static QwpWebSocketSender newBareSender() throws Exception { QwpWebSocketSender sender = (QwpWebSocketSender) Unsafe.getUnsafe() .allocateInstance(QwpWebSocketSender.class); setField(sender, "connectWalkLock", new java.util.concurrent.locks.ReentrantLock()); + setField(sender, "healthTracker", new io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker()); return sender; } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java new file mode 100644 index 000000000..6f1c897ef --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java @@ -0,0 +1,161 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.After; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance test C17 of the dynamic-credential specification (design/qwp-token-provider-spec.md, sections 7.4 + * and 10): senders and query clients built from one {@code token_provider} connect string share one provider, + * with one fetch per refresh, over real TLS connections. + */ +public class TokenProviderSharingTest { + private static final long HOUR = 3_600_000L; + + @After + public void tearDown() { + TestTokenProviderFactory.uninstall(); + } + + @Test(timeout = 60_000) + public void testFacadePoolsShareOneProvider() throws Exception { + assertMemoryLeak(() -> { + TestTokenProviderFactory azure = new TestTokenProviderFactory("azure"); + TestTokenProviderFactory.install(azure); + try (TestWebSocketServer server = startTlsServer()) { + String cfg = config(server, "c17-facade") + "sender_pool_min=2;query_pool_min=2;"; + TokenProviderSpec spec = spec(cfg); + try (QuestDB ignored = QuestDB.connect(cfg)) { + Assert.assertEquals("one provider for the whole facade", 1, azure.created.get()); + Assert.assertEquals("one fetch for every pooled connection", 1, azure.fetches.get()); + Assert.assertEquals("a lease per pooled connection", 4, TokenProviderRegistry.global().leaseCount(spec)); + for (int i = 0; i < 4; i++) { + Assert.assertEquals("Bearer TOKEN-azure", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + } + } + Assert.assertEquals("every pooled connection released its lease", + 0, TokenProviderRegistry.global().leaseCount(spec)); + Assert.assertTrue("the provider lingers for the next client", TokenProviderRegistry.global().isActive(spec)); + } + }); + } + + @Test(timeout = 60_000) + public void testSendersAndQueryClientsShareOneProviderWithOneFetchPerRefresh() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger issued = new AtomicInteger(); + TestTokenProviderFactory azure = new TestTokenProviderFactory("azure", + p -> new ExpiringToken("SHARED-" + issued.incrementAndGet(), System.currentTimeMillis() + HOUR)); + TestTokenProviderFactory.install(azure); + try (TestWebSocketServer server = startTlsServer()) { + String cfg = config(server, "c17-clients"); + TokenProviderSpec spec = spec(cfg); + List senders = new ArrayList<>(); + List queries = new ArrayList<>(); + try { + for (int i = 0; i < 3; i++) { + senders.add(Sender.fromConfig(cfg + "sender_id=s" + i + ';')); + } + for (int i = 0; i < 2; i++) { + QwpQueryClient client = QwpQueryClient.fromConfig(cfg); + queries.add(client); + client.connect(); + } + Assert.assertEquals("one provider for every client", 1, azure.created.get()); + Assert.assertEquals("one fetch for all five connections", 1, azure.fetches.get()); + Assert.assertEquals(5, TokenProviderRegistry.global().leaseCount(spec)); + for (int i = 0; i < 5; i++) { + Assert.assertEquals("Bearer SHARED-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + } + + // One refresh serves everyone: force one through the shared provider, then drop every + // connection. The senders reconnect with the new token without fetching again. + try (TokenProviderRegistry.Lease probe = TokenProviderRegistry.global().acquire(spec)) { + probe.provider().onTokenRejected("SHARED-1", 401); + Assert.assertEquals("SHARED-2", probe.provider().getToken().toString()); + } + Assert.assertEquals(2, azure.fetches.get()); + server.dropAllConnections(); + for (int i = 0; i < 3; i++) { + Assert.assertEquals("Bearer SHARED-2", server.pollAuthorizationHeader(10, TimeUnit.SECONDS)); + } + // the reconnects completed: every sender delivers on its new connection + for (Sender s : senders) { + s.table("t").longColumn("v", 1).atNow(); + s.flush(); + Assert.assertTrue("the sender must deliver after reconnecting", s.awaitAckedFsn(0, 10_000)); + } + Assert.assertEquals("reconnects reuse the cached token", 2, azure.fetches.get()); + } finally { + for (Sender s : senders) { + s.close(); + } + for (QwpQueryClient q : queries) { + q.close(); + } + } + Assert.assertEquals(0, TokenProviderRegistry.global().leaseCount(spec)); + Assert.assertTrue(TokenProviderRegistry.global().isActive(spec)); + } + }); + } + + private static String config(TestWebSocketServer server, String identity) { + // a resource unique to the test keeps the process-wide registry entry from being shared across tests + return "wss::addr=localhost:" + server.getPort() + ";tls_verify=unsafe_off;token_provider=azure;" + + "azure_resource=api://" + identity + '-' + System.nanoTime() + ';'; + } + + private static TokenProviderSpec spec(String cfg) { + return TokenProviderSpec.parse(new ConfigView(ConfigString.parse(cfg)), true); + } + + private static TestWebSocketServer startTlsServer() throws Exception { + TestWebSocketServer server = TestWebSocketServer.tls(new WebSocketDynamicCredentialTest.AckHandler()); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java new file mode 100644 index 000000000..cabf0e0db --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -0,0 +1,968 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.SenderConnectionEvent; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.test.cutlass.auth.TokenTestKit.FakeClock; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ManualScheduler; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C9-C16 and C23 of the dynamic-credential specification (design/qwp-token-provider-spec.md, + * section 10) for the WebSocket ingest {@code Sender}: a {@link RefreshingTokenProvider} wired into real + * connect, reconnect, failover, store-and-forward and orphan-drain paths, against a server that validates the + * presented token rather than rejecting blindly. + */ +public class WebSocketDynamicCredentialTest { + private static final long HOUR = 3_600_000L; + private static final long T0 = 1_800_000_000_000L; + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testAsyncInitializationRetriesProviderFailuresOfEitherClassification() throws Exception { + // C12, async rows: a credential-unavailable failure during async initialization is retried indefinitely + // whatever its classification, reported to the error handler, and recovers with the source. + for (boolean retryable : new boolean[]{true, false}) { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenThrow(new TokenUnavailableException("IdP says no", retryable)); + ErrorCollector errors = new ErrorCollector(); + AckHandler handler = new AckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).build(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.ASYNC) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + await(() -> errors.count("credential-unavailable") >= 2, 10_000, + "the async initial connect to report and retry the provider failure"); + Assert.assertEquals("retried, never terminal [retryable=" + retryable + ']', + 0, errors.terminalCount()); + source.then(() -> new ExpiringToken("RECOVERED", System.currentTimeMillis() + HOUR)); + await(() -> handler.frames.get() >= 1, 10_000, "the row to arrive once the source recovers"); + Assert.assertEquals("Bearer RECOVERED", lastHeader(server)); + } + }); + } + } + + @Test(timeout = 30_000) + public void testChallengeInsufficientScopeDoesNotRetry() throws Exception { + // C23: a 401 whose Bearer challenge carries an error other than invalid_token does not trigger the retry. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate("Bearer realm=\"questdb\", error=\"insufficient_scope\""); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 401 with error=insufficient_scope must not be retried"); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + Assert.assertEquals("insufficient_scope", e.getBearerError()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + Assert.assertEquals("exactly one upgrade", 1, server.upgradeRequestCount()); + Assert.assertEquals("no forced refresh", 1, source.calls()); + } + }); + } + + @Test(timeout = 30_000) + public void testChallengeInvalidTokenRetries() throws Exception { + // C23: a 401 with Bearer error="invalid_token" triggers the retry. + assertChallengeRetries("Bearer realm=\"questdb\", error=\"invalid_token\", error_description=\"expired\""); + } + + @Test(timeout = 30_000) + public void testChallengeOtherSchemeOnlyRetries() throws Exception { + // C23: without a Bearer challenge carrying an error, the status code alone decides. + assertChallengeRetries("Basic realm=\"questdb\""); + } + + @Test(timeout = 30_000) + public void testChallengePlain401Retries() throws Exception { + // C23: a plain 401, with no challenge at all, triggers the retry. + assertChallengeRetries(null); + } + + @Test(timeout = 60_000) + public void testFailoverAcrossARotationSendsTheNewTokenAndReplaysOnce() throws Exception { + // C14: connected to A with T1, the token rotates, A dies. B receives the NEW token, and the frames A + // never acknowledged are replayed to B exactly once. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + CollectingHandler silentA = new CollectingHandler(false); + CollectingHandler ackB = new CollectingHandler(true); + try (RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build(); + TestWebSocketServer b = startServer(ackB)) { + b.setAuthorizationValidator(h -> "Bearer T2".equals(h) ? 0 : 401); + TestWebSocketServer a = startServer(silentA); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + a.getPort()) + .address("localhost:" + b.getPort()) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + for (int i = 0; i < 5; i++) { + sender.table("t").longColumn("v", i).atNow(); + sender.flush(); + } + await(() -> silentA.frames.get() == 5, 10_000, "A to receive the unacknowledged frames"); + + // rotate: the provider now holds T2 + provider.onTokenRejected("T1", 401); + Assert.assertEquals("T2", provider.getToken().toString()); + + a.close(); // A dies; the reconnect fails over to B + await(() -> ackB.distinctPayloads.size() == 5, 15_000, "the unacked frames to replay at B"); + Assert.assertEquals("B must receive the new token", "Bearer T2", + b.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertEquals("B must never see the old token", 0, b.authRejectCount()); + Assert.assertEquals("each unacked frame replayed exactly once", 5, ackB.frames.get()); + } finally { + a.close(); + } + Assert.assertEquals(silentA.payloads, ackB.payloads); + } + }); + } + + @Test(timeout = 30_000) + public void testForbiddenNeverTriggersAForcedRefresh() throws Exception { + // D2: a 403 is an authorization decision; it gets no forced refresh and no retry. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 403 : 0); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 403 must fail startup"); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(403, e.getStatusCode()); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + Assert.assertEquals("a 403 must not force a refresh", 1, source.calls()); + } + }); + } + + @Test(timeout = 60_000) + public void testOrphanDrainAcrossARotationDrainsWithTheNewToken() throws Exception { + // C16: an orphan drain whose first upgrade presents the stale token gets one 401, refreshes, and drains + // the whole slot with the new token. No quarantine. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("orphan-rotation").getAbsolutePath(); + final int frames = 20; + try (TestWebSocketServer silent = startServer(new CollectingHandler(false))) { + String ghostCfg = "ws::addr=localhost:" + silent.getPort() + ";sf_dir=" + sfDir + + ";sender_id=ghost;close_flush_timeout_millis=0;"; + try (Sender ghost = Sender.fromConfig(ghostCfg)) { + for (int i = 0; i < frames; i++) { + ghost.table("t").longColumn("v", i).atNow(); + ghost.flush(); + } + } + } + Assert.assertEquals(1, OrphanScanner.scan(sfDir, "primary").size()); + + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + CollectingHandler ack = new CollectingHandler(true); + AtomicInteger t1Uses = new AtomicInteger(); + try (RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build(); + TestWebSocketServer server = startServer(ack)) { + // T1 is good for exactly one upgrade - the foreground's - and is revoked after it. The drainer's + // first upgrade presents the cached T1 and must recover through the one-retry rule. + server.setAuthorizationValidator(h -> { + if ("Bearer T2".equals(h)) { + return 0; + } + return "Bearer T1".equals(h) && t1Uses.incrementAndGet() == 1 ? 0 : 401; + }); + try (Sender ignored = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("primary") + .drainOrphans(true) + .httpTokenProvider(provider) + .build()) { + await(() -> ack.distinctPayloads.size() == frames, 15_000, "the orphan slot to drain"); + } + Assert.assertEquals("the drainer met the stale token exactly once", 1, server.authRejectCount()); + Assert.assertEquals("the drain used the new token", 2, source.calls()); + } + Assert.assertFalse("a slot that drained must not be quarantined", + Files.exists(new java.io.File(sfDir, "ghost/" + OrphanScanner.FAILED_SENTINEL_NAME).toPath())); + }); + } + + @Test(timeout = 30_000) + public void testPersistent401AfterTheRetryFailsStartup() throws Exception { + // C11, initialization rows: a 401 that survives the one retry fails startup, in OFF and in SYNC mode + // (SYNC does not spend its reconnect budget on it). + for (Sender.InitialConnectMode mode : new Sender.InitialConnectMode[]{ + Sender.InitialConnectMode.OFF, Sender.InitialConnectMode.SYNC}) { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> 401); + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(mode) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a persistent 401 must fail startup [mode=" + mode + ']'); + } catch (QwpAuthFailedException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("startup must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("the original attempt plus one retry with the refreshed token [mode=" + mode + ']', + 2, server.authRejectCount()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), drainHeaders(server)); + } + }); + } + } + + @Test(timeout = 60_000) + public void testPersistent401WhileEstablishedIsRetriedAndReported() throws Exception { + // C11, established row: a 401 that survives the retry is retried indefinitely and reported as a + // retriable security error that names the failure class (decision D1). + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource().then( + () -> new ExpiringToken("T-" + System.nanoTime(), System.currentTimeMillis() + HOUR)); + ErrorCollector errors = new ErrorCollector(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + AtomicInteger accepted = new AtomicInteger(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build()) { + server.setAuthorizationValidator(h -> accepted.getAndIncrement() == 0 ? 0 : 401); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + await(() -> errors.count("auth-rejected") >= 3, 15_000, "repeated auth-rejected reports"); + Assert.assertEquals("an established sender never goes terminal on a 401", 0, errors.terminalCount()); + for (SenderError e : errors.errors) { + if (e.getServerMessage().contains("auth-rejected")) { + Assert.assertEquals(SenderError.Category.SECURITY_ERROR, e.getCategory()); + Assert.assertEquals(SenderError.Policy.RETRIABLE, e.getAppliedPolicy()); + } + } + server.setAuthorizationValidator(null); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 15_000, "recovery once the server accepts again"); + } + } + }); + } + + @Test(timeout = 60_000) + public void testProviderOutageWhileEstablishedIsRetriedReportedAndRecovers() throws Exception { + // C13: the source fails while a connected sender's token expires. The reconnect cannot get a credential; + // it is retried indefinitely and reported, and recovers when the source does. + assertMemoryLeak(() -> { + SkewedClock clock = new SkewedClock(); + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + ErrorCollector errors = new ErrorCollector(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).clock(clock).build(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + // the IdP goes down, and the held token expires (the clock jumps past it) + source.thenThrow(TokenUnavailableException.retryable("IdP unreachable")); + clock.skewMillis = 2 * HOUR; + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed on the live connection, which the server then drops + await(() -> errors.count("credential-unavailable") >= 3, 15_000, + "repeated credential-unavailable reports"); + Assert.assertEquals(0, errors.terminalCount()); + for (SenderError e : errors.errors) { + if (e.getServerMessage().contains("credential-unavailable")) { + Assert.assertEquals(SenderError.Policy.RETRIABLE, e.getAppliedPolicy()); + Assert.assertTrue(e.getServerMessage(), e.getServerMessage().contains("IdP unreachable")); + } + } + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); // buffered in store-and-forward meanwhile + source.then(() -> new ExpiringToken("T2", clock.wallClockMillis() + HOUR)); + await(() -> handler.frames.get() >= 2, 15_000, "the buffered row to arrive after recovery"); + Assert.assertEquals("Bearer T2", lastHeader(server)); + } + }); + } + + @Test(timeout = 30_000) + public void testReconnectAfterExpiryCarriesTheRefreshedToken() throws Exception { + // C9: the server rejects expired tokens against a clock shared with the provider. The proactive refresh + // rotated the token before the old one expired, so the reconnect carries the new one and sees no 401. + assertMemoryLeak(() -> { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1." + (T0 + HOUR), T0 + HOUR) + .thenToken("T2." + (T0 + 2 * HOUR), T0 + 2 * HOUR); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .clock(clock).scheduler(scheduler).random(() -> 0.5).build()) { + scheduler.runNext(); // the prefetch: T1 + server.setAuthorizationValidator(h -> expiredAt(h) <= clock.wallClockMillis() ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build()) { + Assert.assertTrue(server.pollAuthorizationHeader(5, TimeUnit.SECONDS).startsWith("Bearer T1.")); + scheduler.advanceAndRunNext(); // the proactive refresh at the half-life: T2 + clock.advanceMillis(HOUR / 2 + 60_000); // T1 is now expired at the server + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + String reconnect = server.pollAuthorizationHeader(5, TimeUnit.SECONDS); + Assert.assertNotNull("the sender must reconnect", reconnect); + Assert.assertTrue(reconnect, reconnect.startsWith("Bearer T2.")); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 10_000, "the second row"); + } + Assert.assertEquals("no 401 may be observed", 0, server.authRejectCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testStaleTokenGetsExactlyOne401ThenAnImmediateRetry() throws Exception { + // C10: the reconnect presents a token the server has revoked. Exactly one 401, then an immediate retry of + // the same endpoint with the refreshed token: no backoff, no failed-attempt event, no AUTH_FAILED. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + List events = new CopyOnWriteArrayList<>(); + Set revoked = Collections.synchronizedSet(new HashSet<>()); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> revoked.contains(h) ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.OFF) + // a retry through the reconnect loop would wait at least this long + .reconnectInitialBackoffMillis(5_000) + .reconnectMaxBackoffMillis(5_000) + .connectionListener(events::add) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + revoked.add("Bearer T1"); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + Assert.assertEquals("the reconnect first presents the cached token", + "Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long rejectedAt = System.nanoTime(); + Assert.assertEquals("then retries with the refreshed one", + "Bearer T2", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long retryMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - rejectedAt); + Assert.assertTrue("the retry must be immediate, not after backoff; took " + retryMillis + " ms", + retryMillis < 4_000); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 10_000, "the second row"); + await(() -> kinds(events).contains(SenderConnectionEvent.Kind.RECONNECTED), 5_000, "RECONNECTED"); + } + Assert.assertEquals("exactly one 401", 1, server.authRejectCount()); + List kinds = kinds(events); + Assert.assertFalse("only the round's final outcome is reported: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.AUTH_FAILED)); + Assert.assertFalse("the retried 401 is not an endpoint failure: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.ENDPOINT_ATTEMPT_FAILED)); + } + }); + } + + @Test(timeout = 60_000) + public void testSentinelTokensNeverLeakFromTheClient() throws Exception { + // C20 at the client level: across a stale-token 401 and its retry (C10), a persistent 401 (C11), a cold + // provider failure (C4) and a malformed source result, the tokens never appear in any log line (every + // logger, every level), exception, SenderError, connection event, health snapshot or toString(). + final String stale = "SENTINEL-STALE-" + System.nanoTime(); + final String fresh = "SENTINEL-FRESH-" + System.nanoTime(); + ch.qos.logback.classic.Logger root = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = root.getLevel(); + root.setLevel(ch.qos.logback.classic.Level.ALL); + root.addAppender(appender); + List rendered = new CopyOnWriteArrayList<>(); + try { + assertMemoryLeak(() -> { + ErrorCollector errors = new ErrorCollector(); + List events = new CopyOnWriteArrayList<>(); + // C10: the stale token is rejected once, the fresh one is accepted + ScriptedSource source = new ScriptedSource() + .thenToken(stale, System.currentTimeMillis() + HOUR) + .thenToken(fresh, System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> h.contains(stale) ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .errorHandler(errors) + .connectionListener(events::add) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + rendered.add(sender.health().toString()); + rendered.add(sender.toString()); + } + rendered.add(provider.toString()); + + // C11: every token is rejected + server.setAuthorizationValidator(h -> 401); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (RuntimeException e) { + renderThrowable(e, rendered); + } + } + // C4 and a malformed result: the source fails, then returns an already-expired token + ScriptedSource failing = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("IdP unreachable")) + .thenToken(stale, 1L); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(failing).build()) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (RuntimeException e) { + renderThrowable(e, rendered); + } + rendered.add(provider.toString()); + rendered.add(String.valueOf(provider.getLastFailure())); + } + for (SenderError e : errors.errors) { + rendered.add(e.toString()); + rendered.add(String.valueOf(e.getServerMessage())); + } + for (SenderConnectionEvent e : events) { + rendered.add(e.toString()); + if (e.getCause() != null) { + renderThrowable(e.getCause(), rendered); + } + } + }); + } finally { + root.detachAppender(appender); + root.setLevel(savedLevel); + appender.stop(); + } + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + rendered.add(event.getFormattedMessage()); + for (ch.qos.logback.classic.spi.IThrowableProxy t = event.getThrowableProxy(); t != null; t = t.getCause()) { + rendered.add(t.getMessage()); + } + } + Assert.assertTrue("the scenarios must have produced output to inspect", rendered.size() > 20); + for (String s : rendered) { + if (s != null) { + Assert.assertFalse("a token leaked into: " + s, s.contains(stale) || s.contains(fresh)); + } + } + } + + @Test(timeout = 30_000) + public void testStaticCredentialNeverGetsTheRetry() throws Exception { + // section 8.2: a static credential never gets the retry - presenting the same bytes again cannot help. + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new AckHandler())) { + server.setAuthorizationValidator(h -> 401); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpToken("static") + .build() + .close(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 60_000) + public void testStoreAndForwardAcrossARotationDeliversEveryRow() throws Exception { + // C15: the server rejects T1 with 401 while the IdP still hands out T1; the producer keeps writing into + // store-and-forward. Once the IdP rotates, every row is delivered: no quarantine, no data loss, no terminal. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("sf-rotation").getAbsolutePath(); + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + ErrorCollector errors = new ErrorCollector(); + CollectingHandler ack = new CollectingHandler(true); + AtomicInteger connections = new AtomicInteger(); + try (TestWebSocketServer server = startServer(ack); + RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build()) { + // T1 is accepted for the first connection only + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) && connections.getAndIncrement() > 0 ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("rot") + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(50) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 0).atNow(); + sender.flush(); + await(() -> ack.distinctPayloads.size() >= 1, 10_000, "the first row"); + server.dropAllConnections(); + for (int i = 1; i < 20; i++) { + sender.table("t").longColumn("v", i).atNow(); + sender.flush(); + } + await(() -> errors.count("auth-rejected") >= 2, 15_000, "the 401 episode to be reported"); + source.thenToken("T2", System.currentTimeMillis() + HOUR); // the IdP rotates + await(() -> ack.distinctPayloads.size() >= 20, 15_000, "every row to arrive"); + Assert.assertTrue(sender.awaitAckedFsn(19, 10_000)); + } + Assert.assertEquals(0, errors.terminalCount()); + Assert.assertEquals("no data-loss report", 0, errors.count(SenderError.Category.DATA_LOSS)); + } + Assert.assertFalse("no quarantine", Files.exists(new java.io.File(sfDir, "rot/" + OrphanScanner.FAILED_SENTINEL_NAME).toPath())); + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationFailsFastOnAPermanentProviderFailure() throws Exception { + // C12 / D8: a permanent credential-unavailable failure fails SYNC startup fast with the provider's error. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.permanent("no managed identity is assigned")); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertFalse(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no managed identity is assigned")); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("no endpoint may be contacted", 0, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationFailsFastOnAnUnclassifiedProviderException() throws Exception { + // D8: an exception from an application-supplied provider that is not a token-unavailable error is + // permanent, even when it says "retry". + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler())) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + throw new IllegalStateException("temporarily unavailable, please retry"); + }) + .build() + .close(); + Assert.fail(); + } catch (IllegalStateException e) { + Assert.assertTrue(e.getMessage().contains("temporarily unavailable")); + } + Assert.assertEquals(1, pulls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationRetriesARetryableProviderFailureWithinTheBudget() throws Exception { + // C12 / D6: a retryable credential-unavailable failure is retried within reconnect_max_duration_millis. + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + if (pulls.incrementAndGet() < 4) { + throw TokenUnavailableException.retryable("IMDS returned HTTP 429", 0); + } + return "FINALLY"; + }) + .build()) { + Assert.assertEquals("Bearer FINALLY", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertEquals(4, pulls.get()); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationRetryableProviderFailureExhaustsTheBudgetNamingTheClass() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler())) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(300) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + throw TokenUnavailableException.retryable("IdP timed out"); + }) + .build() + .close(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.fail("a retryable failure must consume the budget, not fail fast: " + e.getMessage()); + } catch (io.questdb.client.cutlass.line.LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("credential-unavailable: IdP timed out")); + } + Assert.assertTrue("retried within the budget, pulls=" + pulls.get(), pulls.get() > 2); + Assert.assertEquals(0, server.upgradeRequestCount()); + } + }); + } + + private static void renderThrowable(Throwable t, List into) { + for (Throwable c = t; c != null; c = c.getCause()) { + into.add(c.toString()); + for (Throwable s : c.getSuppressed()) { + renderThrowable(s, into); + } + } + } + + private static List drainHeaders(TestWebSocketServer server) throws InterruptedException { + List headers = new ArrayList<>(); + String h; + while ((h = server.pollAuthorizationHeader(200, TimeUnit.MILLISECONDS)) != null) { + headers.add(h); + } + return headers; + } + + // Tokens shaped T., as the specification's conformance harness suggests. + private static long expiredAt(String header) { + int dot = header.lastIndexOf('.'); + return dot < 0 ? Long.MIN_VALUE : Long.parseLong(header.substring(dot + 1)); + } + + private static RefreshingTokenProvider.Builder fastProvider(ScriptedSource source) { + return RefreshingTokenProvider.builder(source) + .coldWaitMillis(200) + .backoffInitialMillis(10) + .backoffMaxMillis(20) + .forcedWaitMillis(5_000); + } + + private static List kinds(List events) { + List kinds = new ArrayList<>(); + for (SenderConnectionEvent e : events) { + kinds.add(e.getKind()); + } + return kinds; + } + + private static String lastHeader(TestWebSocketServer server) throws InterruptedException { + String last = null; + String h; + while ((h = server.pollAuthorizationHeader(100, TimeUnit.MILLISECONDS)) != null) { + last = h; + } + return last; + } + + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) + throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(handler); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private void assertChallengeRetries(String challenge) throws Exception { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate(challenge); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + } + Assert.assertEquals("challenge=" + challenge, Arrays.asList("Bearer T1", "Bearer T2"), drainHeaders(server)); + Assert.assertEquals(1, server.authRejectCount()); + } + }); + } + + static byte[] buildAck(long seq) { + byte[] buf = new byte[1 + 8 + 2]; + ByteBuffer bb = ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(seq); + bb.putShort((short) 0); + return buf; + } + + /** + * ACKs every frame, per connection: wire sequences restart at 0 on each connection. + */ + static class AckHandler implements TestWebSocketServer.WebSocketServerHandler { + final AtomicLong frames = new AtomicLong(); + private final java.util.Map seqs = + Collections.synchronizedMap(new java.util.IdentityHashMap<>()); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + frames.incrementAndGet(); + long seq = seqs.computeIfAbsent(client, c -> new AtomicLong()).getAndIncrement(); + try { + client.sendBinary(buildAck(seq)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } + + /** + * Records every frame; ACKs them per connection when {@code ack} is set. + */ + static final class CollectingHandler extends AckHandler { + final Set distinctPayloads = Collections.synchronizedSet(new HashSet<>()); + final List payloads = new CopyOnWriteArrayList<>(); + private final boolean ack; + + CollectingHandler(boolean ack) { + this.ack = ack; + } + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + String p = Arrays.toString(data); + distinctPayloads.add(p); + payloads.add(p); + if (ack) { + super.onBinaryMessage(client, data); + } else { + frames.incrementAndGet(); + } + } + } + + /** + * ACKs every frame; closes the first connection right after its first ACK, forcing a reconnect. + */ + static final class DropAfterFirstAckHandler extends AckHandler { + private volatile boolean dropped; + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + super.onBinaryMessage(client, data); + if (!dropped) { + dropped = true; + try { + Thread.sleep(50); // let the ACK flush before the socket goes + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + client.close(); + } + } + } + + static final class ErrorCollector implements io.questdb.client.SenderErrorHandler { + final List errors = new CopyOnWriteArrayList<>(); + + int count(String messageFragment) { + int n = 0; + for (SenderError e : errors) { + if (e.getServerMessage() != null && e.getServerMessage().contains(messageFragment)) { + n++; + } + } + return n; + } + + int count(SenderError.Category category) { + int n = 0; + for (SenderError e : errors) { + if (e.getCategory() == category) { + n++; + } + } + return n; + } + + @Override + public void onError(SenderError error) { + errors.add(error); + } + + int terminalCount() { + int n = 0; + for (SenderError e : errors) { + if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { + n++; + } + } + return n; + } + } + + // Real monotonic time, with a wall clock the test can push forward. + static final class SkewedClock implements RefreshingTokenProvider.Clock { + volatile long skewMillis; + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis() + skewMillis; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java new file mode 100644 index 000000000..d36fb79b4 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java @@ -0,0 +1,112 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.websocket; + +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLServerSocketFactory; +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.security.KeyStore; +import java.util.concurrent.TimeUnit; + +/** + * A self-signed server identity for TLS test servers, generated once per JVM with the running JDK's + * {@code keytool} - no key material is committed to the repository. Clients connect with + * {@code tls_verify=unsafe_off}. + */ +public final class TestTls { + private static final char[] PASSWORD = "test-only".toCharArray(); + private static SSLServerSocketFactory factory; + + private TestTls() { + } + + public static synchronized SSLServerSocketFactory serverSocketFactory() throws IOException { + if (factory == null) { + factory = create(); + } + return factory; + } + + private static SSLServerSocketFactory create() throws IOException { + File keystore = File.createTempFile("qdb-test-server", ".p12"); + if (!keystore.delete()) { + throw new IOException("could not prepare " + keystore); + } + keystore.deleteOnExit(); + String keytool = System.getProperty("java.home") + File.separator + "bin" + File.separator + "keytool"; + ProcessBuilder pb = new ProcessBuilder( + keytool, "-genkeypair", + "-alias", "server", + "-keyalg", "RSA", + "-keysize", "2048", + "-validity", "3650", + "-dname", "CN=localhost", + "-ext", "SAN=dns:localhost,ip:127.0.0.1", + "-storetype", "PKCS12", + "-keystore", keystore.getAbsolutePath(), + "-storepass", new String(PASSWORD), + "-keypass", new String(PASSWORD) + ); + pb.redirectErrorStream(true); + Process process = pb.start(); + byte[] output = readAll(process.getInputStream()); + try { + if (!process.waitFor(60, TimeUnit.SECONDS) || process.exitValue() != 0) { + throw new IOException("keytool failed: " + new String(output, StandardCharsets.UTF_8)); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("interrupted while running keytool", e); + } + try (InputStream in = new FileInputStream(keystore)) { + KeyStore ks = KeyStore.getInstance("PKCS12"); + ks.load(in, PASSWORD); + KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); + kmf.init(ks, PASSWORD); + SSLContext context = SSLContext.getInstance("TLS"); + context.init(kmf.getKeyManagers(), null, null); + return context.getServerSocketFactory(); + } catch (IOException e) { + throw e; + } catch (Exception e) { + throw new IOException("could not load the test keystore", e); + } + } + + private static byte[] readAll(InputStream in) throws IOException { + java.io.ByteArrayOutputStream out = new java.io.ByteArrayOutputStream(); + byte[] buf = new byte[4096]; + int n; + while ((n = in.read(buf)) > 0) { + out.write(buf, 0, n); + } + return out.toByteArray(); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java index 908e3bd56..85d6dea87 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java @@ -49,6 +49,7 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.ToIntFunction; /** * A simple WebSocket server for client integration testing. @@ -57,6 +58,10 @@ public class TestWebSocketServer implements Closeable { private static final Logger LOG = LoggerFactory.getLogger(TestWebSocketServer.class); private static final String WEBSOCKET_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + // Number of upgrades the authorization validator rejected, over the server's lifetime. + private final AtomicInteger authRejectCount = new AtomicInteger(); + // Number of well-formed upgrade requests received, accepted or rejected, over the server's lifetime. + private final AtomicInteger upgradeRequestCount = new AtomicInteger(); // Authorization header value captured from each well-formed upgrade request ("" when absent), in // arrival order. Tests poll this to assert the token a provider supplied at each (re)handshake. private final BlockingQueue capturedAuthHeaders = new LinkedBlockingQueue<>(); @@ -84,6 +89,12 @@ public class TestWebSocketServer implements Closeable { // connected even after they have all disconnected. private final AtomicInteger totalHandshakes = new AtomicInteger(); private Thread acceptThread; + // When non-null, decides every upgrade from its Authorization header value ("" when absent): a positive + // return is the HTTP status to reject with (e.g. 401, 403), zero or less accepts. Lets a test model a + // server that validates tokens - expired, revoked or rotated - rather than one that rejects blindly. + private volatile ToIntFunction authorizationValidator; + // WWW-Authenticate value sent with a 401 reject (validator or setRejectWithStatus), or null for none. + private volatile String rejectWwwAuthenticate; // X-QuestDB-Role value to emit on handshake responses. null = omit the // header (legacy behavior for tests written before role-aware failover). // The server emits the header on both the 101 success path and (when @@ -168,6 +179,12 @@ public TestWebSocketServer(WebSocketServerHandler handler, public TestWebSocketServer(WebSocketServerHandler handler, boolean emitDurableAckHeader, String advertisedRole, int requestedPort) throws IOException { + this(handler, emitDurableAckHeader, advertisedRole, requestedPort, false); + } + + private TestWebSocketServer(WebSocketServerHandler handler, + boolean emitDurableAckHeader, String advertisedRole, + int requestedPort, boolean tls) throws IOException { this.handler = handler; this.emitDurableAckHeader = emitDurableAckHeader; this.advertisedRole = advertisedRole; @@ -177,11 +194,21 @@ public TestWebSocketServer(WebSocketServerHandler handler, // which another process could grab a pre-selected port before start() // binds it. Pinning to loopback keeps client "localhost" connections // routed here rather than to a wildcard listener on the same port. - serverSocket = new ServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()); + serverSocket = tls + ? TestTls.serverSocketFactory().createServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()) + : new ServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()); serverSocket.setSoTimeout(100); this.port = serverSocket.getLocalPort(); } + /** + * A server that speaks WebSocket over TLS with a self-signed certificate (see {@link TestTls}). Connect + * with {@code wss::...;tls_verify=unsafe_off;}. + */ + public static TestWebSocketServer tls(WebSocketServerHandler handler) throws IOException { + return new TestWebSocketServer(handler, false, null, 0, true); + } + public boolean awaitRoleReject(long timeout, TimeUnit unit) throws InterruptedException { return roleRejectLatch.await(timeout, unit); } @@ -234,6 +261,46 @@ public int handshakeCount() { return totalHandshakes.get(); } + /** + * Number of upgrades the authorization validator rejected over the server's lifetime. + */ + public int authRejectCount() { + return authRejectCount.get(); + } + + /** + * Closes every live client connection while the listener keeps accepting, so connected clients see a + * dropped connection and reconnect. + */ + public void dropAllConnections() { + for (ClientHandler client : clients) { + client.close(); + } + } + + /** + * Number of well-formed upgrade requests received over the server's lifetime, whatever the answer. + */ + public int upgradeRequestCount() { + return upgradeRequestCount.get(); + } + + /** + * Installs a validator that decides every upgrade from its {@code Authorization} header value ({@code ""} + * when absent): a positive return is the HTTP status to reject with, zero or less accepts. Pass null to + * remove it. Runs before the role and status rejects. + */ + public void setAuthorizationValidator(ToIntFunction validator) { + this.authorizationValidator = validator; + } + + /** + * {@code WWW-Authenticate} value to send with a {@code 401} reject, or null for none. + */ + public void setRejectWwwAuthenticate(String value) { + this.rejectWwwAuthenticate = value; + } + /** * Number of WebSocket connections currently live from the server's view. * Drops back to zero once every client has closed its socket. @@ -603,6 +670,7 @@ private boolean performHandshake() throws IOException { if (key == null) { return false; } + upgradeRequestCount.incrementAndGet(); capturedAuthHeaders.add(authorization); // Read-path reject: drop the egress upgrade before the 101 so the @@ -612,18 +680,23 @@ private boolean performHandshake() throws IOException { return false; } + // Token-validating reject path: the validator inspects the presented credential. + ToIntFunction validator = authorizationValidator; + if (validator != null) { + int status = validator.applyAsInt(authorization); + if (status > 0) { + writeStatusReject(status, status == 401 ? "Unauthorized" : status == 403 ? "Forbidden" : "Rejected"); + authRejectCount.incrementAndGet(); + return false; + } + } + // Arbitrary-status reject path: tests use setRejectWithStatus // to drive the failover loop's terminal-vs-transient // classification (failover.md §6). int customStatus = rejectingStatusCode; if (customStatus > 0) { - String reason = rejectingStatusReason != null ? rejectingStatusReason : ""; - String sb = "HTTP/1.1 " + customStatus + ' ' + reason + "\r\n" + - "Connection: close\r\n" + - "Content-Length: 0\r\n" + - "\r\n"; - out.write(sb.getBytes(StandardCharsets.US_ASCII)); - out.flush(); + writeStatusReject(customStatus, rejectingStatusReason != null ? rejectingStatusReason : ""); statusRejectCount.incrementAndGet(); return false; } @@ -672,6 +745,20 @@ private boolean performHandshake() throws IOException { return true; } + private void writeStatusReject(int status, String reason) throws IOException { + StringBuilder sb = new StringBuilder() + .append("HTTP/1.1 ").append(status).append(' ').append(reason).append("\r\n") + .append("Connection: close\r\n") + .append("Content-Length: 0\r\n"); + String challenge = rejectWwwAuthenticate; + if (status == 401 && challenge != null) { + sb.append("WWW-Authenticate: ").append(challenge).append("\r\n"); + } + sb.append("\r\n"); + out.write(sb.toString().getBytes(StandardCharsets.US_ASCII)); + out.flush(); + } + private synchronized void writeFrame(int opcode, byte[] payload, int length) throws IOException { // first byte: FIN + opcode out.write(0x80 | (opcode & 0x0F)); @@ -702,6 +789,12 @@ void start() { readThread = new Thread(() -> { try { + if (socket instanceof javax.net.ssl.SSLSocket) { + // finish the TLS handshake under a generous timeout before the short polling + // timeout below applies to every read + socket.setSoTimeout(10_000); + ((javax.net.ssl.SSLSocket) socket).startHandshake(); + } socket.setSoTimeout(100); in = socket.getInputStream(); diff --git a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java index 10b1e3c03..52758d3ea 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java @@ -28,6 +28,7 @@ import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import io.questdb.client.impl.ConfigSchema; import io.questdb.client.impl.Side; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import io.questdb.client.test.tools.TestUtils; import org.junit.Assert; import org.junit.Test; @@ -92,6 +93,19 @@ public void testEveryEgressKeyIsHonored() throws Exception { Assert.assertEquals("pw", tls.get("tls_roots_password")); markHonored("tls_verify", "tls_roots", "tls_roots_password"); + // token_provider and its azure keys (wss only; the factory is resolved at parse time, so install one). + TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); + try { + Map tp = snapshot("wss::addr=h:9000;token_provider=azure;" + + "azure_resource=api://app/.default;azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + Assert.assertEquals("azure", tp.get("token_provider")); + Assert.assertEquals("api://app", tp.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + } finally { + TestTokenProviderFactory.uninstall(); + } + markHonored("token_provider", "azure_resource", "azure_client_id"); + // Drift guard: every egress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so // deleting one trips this -- unlike a hand-maintained list, it cannot diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index 3c257827a..6e6d2358d 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -26,6 +26,7 @@ import io.questdb.client.Sender; import io.questdb.client.impl.ConfigSchema; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import io.questdb.client.impl.Side; import org.junit.Assert; import org.junit.Test; @@ -82,6 +83,7 @@ public void testEveryIngressKeyIsHonored() { assertHonored("connection_listener_inbox_capacity=64", "connection_listener_inbox_capacity", 64); assertHonored("token=ey.abc", "token", "ey.abc"); assertHonored("auth_timeout_ms=4321", "auth_timeout_ms", 4321L); + assertHonored("auth_failure_max_duration_millis=600000", "auth_failure_max_duration_millis", 600000L); assertHonored("connect_timeout=7000", "connect_timeout", 7000); // username/password together (both-or-neither), and the user/pass aliases. @@ -104,6 +106,19 @@ public void testEveryIngressKeyIsHonored() { Assert.assertEquals("pw", tls.get("tls_roots_password")); markHonored("tls_roots", "tls_roots_password"); + // token_provider and its azure keys (wss only; the factory is resolved at parse time, so install one). + TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); + try { + Map tp = snapshot("wss::addr=h:9000;token_provider=azure;azure_resource=api://app/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + Assert.assertEquals("azure", tp.get("token_provider")); + Assert.assertEquals("api://app", tp.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + } finally { + TestTokenProviderFactory.uninstall(); + } + markHonored("token_provider", "azure_resource", "azure_client_id"); + // Drift guard: every ingress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so // deleting one trips this -- unlike a hand-maintained list, it cannot diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 03985fce8..31c4c1e53 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -1,6 +1,6 @@ # Entra ID app-only auth over QWP: findings and proposed design -Status: **draft for review**. No production code has been written. This file is not committed. +Status: **implemented** on `feat/qwp-entra-token-provider` - all four steps of the Java plan (§10). §11 maps each step to the code and to the conformance tests, and lists where the code departs from this document's earlier design. **Read these first:** @@ -552,5 +552,53 @@ Build in this order. Each step is one PR and must pass the conformance tests lis **Separate tickets, outside this feature:** -- **Dead pooled egress worker** (§3). Start with a test that reproduces it. +- **Dead pooled egress worker** (§3). Start with a test that reproduces it. *Done after all - the spec makes recovery a MUST (§8.3, "Egress recovery"); see §11.* - **Treat a 503 at the upgrade as transient in every phase** (§3, spec Appendix C). This must land before any server change to return 503. + +## 11. Implementation status + +Every step of §10 is implemented. Line references in §1-§7 predate it. + +**Step 1 - token cache** (spec §3-§5, Appendix B). `io.questdb.client.cutlass.auth`: + +- `ExpiringToken`, `TokenSource`, `TokenUnavailableException`, and `RefreshingTokenProvider` with its builder (every §5.1 parameter, plus clock, scheduler and jitter seams for tests). +- `CredentialRedaction`: the §9 token rendering (length + 8-hex SHA-256 prefix) and the sanitizing of library text (display-unsafe characters stripped, 256-character cap). +- The warm path of `getToken()` is a volatile read with no lock and no I/O. A refresh whose wall-clock time has passed but whose monotonic schedule has not (a host that slept) starts in the background. +- A cold caller starts one fetch and then waits; it does not start another each time a fetch completes. A source that keeps returning a token inside the hand-out floor is therefore not fetched in a tight loop. While waiting, the "fail immediately" rule of §5.3 is re-evaluated after each failed fetch. +- Tests: `RefreshingTokenProviderTest` (C1-C8, the cache half of C20). + +**Step 2 - client integration** (spec §6, §8.1-§8.3). + +- `HttpTokenProvider.onTokenRejected` (default no-op). +- `QwpWebSocketSender.tokenProviderAuthHeader` is the named dynamic credential that replaced the lambda in `Sender.buildWebSocketAuthHeader`. +- The one retry after a refreshable 401 is in `QwpWebSocketSender.connectWalk` (shared by the foreground and the orphan drainers) and in `QwpQueryClient.connect` and `reconnectViaTracker`. It records no health penalty and fires no event. +- `WWW-Authenticate` is captured by `WebSocketClient` and parsed by `cutlass.http.BearerChallenge`. `QwpAuthFailedException.isTokenRefreshable()` applies the §8.2 challenge rule; its message now starts with `auth-rejected`. +- SYNC startup (`CursorWebSocketSendLoop.connectWithRetry`) retries a credential failure only when `QwpCredentialUnavailableException.isRetryable()` (D6/D8) and the thread is not interrupted. +- Egress errors name the failure class. A provider failure on `connect()` is a `QwpCredentialUnavailableException` with a `credential-unavailable:` message. +- Egress recovery: after a failover reconnect fails, the next `execute()` reconnects instead of throwing "not connected". `QwpQueryClientDynamicCredentialTest.testPooledQueryClientRecoversAfterAFailedFailover` reproduced the §3 dead-pooled-worker bug before the fix (it failed with exactly that message). +- Tests: `WebSocketDynamicCredentialTest` (C9-C16, C23), `QwpQueryClientDynamicCredentialTest`, `BearerChallengeTest`. `TestWebSocketServer` gained an authorization validator, a `WWW-Authenticate` challenge, `dropAllConnections()`, and a TLS mode. The TLS mode uses a self-signed identity generated per JVM with `keytool` (`TestTls`), because `token_provider` is `wss::`-only. + +**Step 3 - connect string, registry, Azure module** (spec §7). + +- `ConfigSchema` registers `token_provider`, `azure_resource` and `azure_client_id` (COMMON). +- `TokenProviderSpec.parse` enforces §7.2 on both clients and resolves the factory without fetching anything. It strips `/.default`, checks and lower-cases the client-ID GUID, and builds the registry key. +- `TokenProviderFactory` is the `ServiceLoader` SPI (`uses` in `module-info.java`). `TokenProviderRegistry` hands out ref-counted leases, with a 60 s linger, a daemon timer that exits when idle, and a test seam that replaces discovery. +- The non-QWP `Sender` schemas reject the keys (D9). The builders and `QuestDBBuilder` reject mixing them with an application-supplied provider. +- Leases: `Sender.build()` acquires one and hands it to `QwpWebSocketSender.setCredentialLease` (released on close, or released by `build()` if it fails). `QwpQueryClient` acquires one on its first `connect()` and releases it on close. +- New reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (not `io.questdb`: the core artifact is `org.questdb:questdb-client`): + - `AzureTokenProviderFactory` uses `DefaultAzureCredential`; `azure_client_id` sets both the managed-identity and workload-identity client ID. + - `AzureTokenSource` bounds each request at 30 s and classifies per §7.5: `CredentialUnavailableException`, HTTP 400/401 and known AADSTS configuration codes are permanent; everything else is retryable, with `Retry-After` honoured. + - It never attaches a library exception: its response references the raw request. + - Java 8 bytecode. It has no `module-info`; `Automatic-Module-Name: io.questdb.client.azure`, and an automatic module provides its `META-INF/services`. + - Its release profiles mirror core's. +- Tests: `TokenProviderConfigTest` (C18, C19), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network). + +**Step 4 - health and deadline** (spec §8.4, §8.5). + +- `io.questdb.client.ConnectionHealth` (with `State`, `FailureClass`, `Failure`, `Aggregate`) and `QwpConnectionHealthTracker`, which publishes immutable snapshots through a volatile field. +- The ingest walk reports rounds and upgrades; the I/O loop reports connection loss and terminal failure. Drainers never report. +- Accessors: `Sender.health()` (default throws `UnsupportedOperationException`), `QwpQueryClient.health()`, `QuestDB.health()` (aggregate of every pooled connection). +- `auth_failure_max_duration_millis` (INGRESS key) and `Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)`. The clock lives in `CursorWebSocketSendLoop`, applies to FOREGROUND loops only, and is tracked even before the deadline is armed. The builder arms it right after connecting, so an `async` start is measured from its first failure. +- Tests: `ConnectionHealthTest` (C21, C22). + +**Not done:** the spec documents (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`, the connect-string reference) live in the documentation repository. The Rust core and Python parity is tracked there too. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 9acd0e197..07e4e7c67 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -520,10 +520,10 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | Refreshing provider | `RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable`, created with `RefreshingTokenProvider.builder(TokenSource)`. Methods: `getToken()`, `onTokenRejected(CharSequence, int)`, `awaitReady(long)`, `close()`. The builder exposes the §5.1 parameters, plus clock and scheduler seams for tests. | | token-unavailable error | `TokenUnavailableException extends LineSenderException`, with `isRetryable()` and `getRetryAfterMillis()` (-1 means none) | | `on_rejected` | `onTokenRejected(CharSequence token, int httpStatus)`, a default no-op method on the existing `io.questdb.client.HttpTokenProvider` | -| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`) | -| Registry | `TokenProviderRegistry`: process-wide, hands out ref-counted leases | -| `azure` provider | The new reactor module `azure/`, artifact `io.questdb:questdb-client-azure`, package `io.questdb.client.azure`, class `AzureTokenProviderFactory`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | -| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as an aggregate by `QuestDB.health()`. | +| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`). `TokenProviderSpec.parse` applies the §7.2 rules to a connect string and resolves the factory without fetching a token. | +| Registry | `TokenProviderRegistry`: process-wide (`global()`), hands out ref-counted `Lease`s, closes a provider `registry_linger` after its last lease | +| `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | +| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | | Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | **Python notes:** diff --git a/pom.xml b/pom.xml index 0b9b6b335..85e2be820 100644 --- a/pom.xml +++ b/pom.xml @@ -86,6 +86,7 @@ core + azure examples From 6fcc5df8f3c4296863bbc378cd80aa3bcbdf27fe Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Fri, 2 Oct 2026 19:30:57 +0100 Subject: [PATCH 3/3] Classify Azure credential errors by mode The spec classified Azure Identity's "no credential available in the chain" as permanent (section 7.5), while section 4 calls network failures and IMDS 404/410 retryable. Inside DefaultAzureCredential an unreachable IMDS produces exactly that message: the chain probes IMDS once, with a short timeout and no retries, so a transient IMDS outage failed a SYNC startup fast. Spec v0.4 follows Azure Identity's own split between fail-fast discovery and a resilient single credential: - 7.1, 7.2: a new key, azure_credential (default, managed_identity, workload_identity, environment), with its validation rules. - 7.5: library errors are classified by how the credential was selected. With one credential, configuration errors are permanent and endpoint or network failures retryable. In the discovery chain, "no credential available" is retryable and the client warns once. Errors should carry the library's innermost reason. - 4: a source that discovers its credential by probing must not treat "nothing found" as permanent on that evidence alone. - 10: conformance test C24. 12: decision D10, which records why an SMBIOS host check was rejected. Appendix B: the Java binding. Appendix D: precedents from Azure Identity, MongoDB, Google, AWS and Apache Druid. The Java implementation still follows v0.3. --- design/qwp-token-provider-spec.md | 51 +++++++++++++++++++++++-------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 07e4e7c67..f9f312500 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -1,9 +1,10 @@ # QWP dynamic bearer credentials: cross-language specification -**Status:** v0.3, ready for implementation. All decisions are resolved (§12). +**Status:** v0.4, ready for implementation. All decisions are resolved (§12). **Changes:** +- **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), and library errors are classified by how the credential was selected (§7.5, D10). This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. - **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). - **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. @@ -86,7 +87,8 @@ Rules for `expires_at`: Classification: - **Retryable:** network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. -- **Permanent:** configuration that is missing or wrong. Examples: no credential configured, identity not found, invalid client. +- **Permanent:** configuration that is missing or wrong. Examples: the credential the source was told to use is not configured, identity not found, invalid client. +- **Discovery:** a source that finds its credential by probing the environment MUST NOT treat "nothing found" as permanent on that evidence alone. A brief outage of a probed endpoint looks the same (§7.5) **[D10]**. - **When unsure:** treat the failure as retryable. - An error that carries no classification MUST be treated as retryable. @@ -236,6 +238,7 @@ None of these MAY reveal the token itself. | `token_provider` | `azure`. The name `azure_imds` is reserved **[D5]**. | no | Selects a provider. | | `azure_resource` | `api://` or `` | when `token_provider=azure` | The application ID URI or the client ID of the QuestDB app registration. A trailing `/.default` MUST be stripped. The requested scope is `/.default`. | | `azure_client_id` | a GUID | no | The client ID of a user-assigned managed identity or of a workload identity. | +| `azure_credential` | `default`, `managed_identity`, `workload_identity`, `environment` | no | Which Azure credential `azure` uses (§7.5). `default`, the default, is the library's default credential: unless the platform's own selector (`AZURE_TOKEN_CREDENTIALS`) names one credential, it is a discovery chain, meant for development. Each other value selects that one credential and overrides the platform's selector. This deterministic mode is recommended in production **[D10]**. | ### 7.2 Validation @@ -245,6 +248,8 @@ A client MUST reject the configuration, with an error that names the offending k - `token_provider` is empty, unknown, or not supported by this client. The error MUST list the supported values. A client MUST NOT silently connect without credentials; - a provider-specific key is present but the selected provider does not accept it. For example, `azure_resource` without `token_provider=azure`; - a provider key that is required is missing; +- `azure_credential` has a value that §7.1 does not list. The error MUST list the supported values; +- `azure_client_id` is combined with `azure_credential=environment`. That credential takes its client ID from the platform's configuration; - `token_provider` is used with any schema other than `wss::`: - plain `ws::` is rejected because the token would cross the network in cleartext **[D4]**; - non-QWP schemas, such as ILP's `http::`, are out of scope **[D9]**. @@ -269,23 +274,29 @@ A client MUST reject the configuration, with an error that names the offending k ### 7.5 The `azure` provider -- **How tokens are obtained:** through the language's Azure Identity default credential chain, for the scope `/.default`. Depending on the environment, the chain finds: - - an environment service principal, - - a workload identity, - - a managed identity, - - developer tools. +- **How tokens are obtained:** through the language's Azure Identity library, for the scope `/.default`. `azure_credential` (§7.1) decides which credential is used: + - `default`: the library's default credential. Unless the platform's selector names one credential, this is a **discovery chain**. It tries an environment service principal, a workload identity, a managed identity and developer tools, in that order, and uses the first that works. + - `managed_identity`, `workload_identity` or `environment`: that credential alone, called **deterministic mode**. A platform selector that names one credential has the same effect. +- **Why the mode matters:** to keep development machines fast, a discovery chain checks for the instance metadata service (IMDS) with one short request and no retries, and moves on when nothing answers. So it reports "no credential available" both on a machine without a managed identity and on an Azure host whose IMDS is briefly unreachable. In deterministic mode the library skips that check and retries the managed-identity endpoint with backoff, so the two cases are reported differently. - **`azure_client_id`:** when set, it selects both the managed identity and the workload identity. - **Expiry:** `expires_at`, and `refresh_at` when the library exposes it, come from the library's token result. -- **Classification of library errors:** - - **Permanent:** "no credential available in the chain", and authentication failures that Entra reports (such as an invalid client or an application that does not exist). - - **Retryable:** network errors, timeouts, throttling, 5xx responses, and anything else. +- **Classification of library errors [D10]:** + - **Deterministic mode:** + - **Permanent:** configuration that the selected credential reports as missing or wrong. That covers missing settings it requires (for example, no federated token file for a workload identity), an identity that is not found or not assigned to the host, and Entra rejecting the client, secret, assertion, application or tenant. + - **Retryable:** network errors, including a managed-identity endpoint that cannot be reached; timeouts; throttling; `404`, `410` and `5xx` responses from a managed-identity endpoint; and anything else. + - **Discovery chain:** + - "No credential available in the chain" is **retryable** (§4). + - A failure that one credential in the chain reports for itself is classified as in deterministic mode. + - A client SHOULD log once, when the provider starts, that a discovery chain does not retry a managed-identity outage, and recommend `azure_credential` for production. +- **Error text:** libraries often report every managed-identity failure with one generic message and keep the reason, such as "Identity not found", in a nested cause. The error SHOULD carry that innermost reason, sanitized as §9 requires. - **Unsupported platforms:** a client whose platform has no Azure Identity library MAY leave `azure` unsupported, and must then reject it as described in §7.2. *Informative bindings:* -- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. +- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. It maps `azure_credential` onto the library's own selector (Appendix B). - **Python:** `azure-identity` as an optional extra. - **Rust and C:** not required. +- **Platform support:** the Azure Identity libraries read `AZURE_TOKEN_CREDENTIALS`. When it selects `ManagedIdentityCredential`, they skip the IMDS check and retry with backoff from .NET 1.16.0, Java 1.18.1, Python 1.25.1, JavaScript 4.13.0, Go 1.13.0 and C++ 1.13.2. ## 8. Failure handling @@ -437,6 +448,7 @@ Every client that implements this specification SHOULD pass these scenarios. The | C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | | C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | | C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | +| C24 | Azure credential selection | With `azure_credential=managed_identity`, an unreachable managed-identity endpoint is retryable, so `sync` initialization retries it within its budget; an identity that is not assigned is permanent, so initialization fails fast. With `default`, a chain that finds nothing is retryable, and the warning in §7.5 is logged once. | ## 11. Relation to existing specifications @@ -461,6 +473,7 @@ All decisions below are resolved. Changing one after the Java implementation mer | D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | | D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | | D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | +| D10 | How the `azure` provider classifies "no credential available", and how to make it resilient | `azure_credential` selects one credential, so the library itself reports an unreachable endpoint (retryable) apart from a misconfigured credential (permanent). In a discovery chain, "no credential available" is retryable, because an IMDS outage looks the same (§7.5). Only `sync` initialization depends on the difference (§8.3), so the cost is a wait of at most `reconnect_max_duration_millis`. This follows Azure Identity's split between fail-fast discovery and a resilient single credential (Appendix D). A local host check, as Google's library makes, was not adopted: Azure's SMBIOS asset tag identifies only the public cloud, not sovereign clouds or Azure Local. | **Still to verify in a real Entra tenant.** These checks don't block implementation, but they should be done before the spec freezes: @@ -468,6 +481,7 @@ All decisions below are resolved. Changing one after the Java implementation mer - The v1 and v2 signing-key endpoints serve the same keys. - Whether IMDS returns the same token when asked again. This affects only the convergence note in §5.2. - How long `DefaultAzureCredential` takes on a cold start. This sizes `cold_wait`. +- On a VM, with `azure_credential=managed_identity`: an unreachable IMDS is reported as retryable, and an identity that is not assigned as permanent. So far this was checked only against a stub endpoint, with Azure Identity for Java 1.18.4. ## Appendix A. QuestDB Enterprise and Entra configuration (informative) @@ -525,6 +539,7 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | | Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | | Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | +| Credential selection | Key `azure_credential`, handled by `AzureTokenProviderFactory`. It sets `AZURE_TOKEN_CREDENTIALS` (`ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential`) in the configuration it passes to `DefaultAzureCredentialBuilder.configuration(...)`, never in the process environment. That needs `azure-identity` 1.18.1 or later. `AzureTokenSource` learns the mode from the factory; for a credential an application passes in itself, a `DefaultAzureCredential` counts as a discovery chain and any other credential as deterministic. | **Python notes:** @@ -551,7 +566,7 @@ These changes would let clients refresh only when a new token can help, and trea Other server-side follow-ups, including security hardening, are tracked privately with the QuestDB Enterprise team (see `SECURITY.md`). -## Appendix D. Precedents for D1 and D2 (informative) +## Appendix D. Precedents for D1, D2 and D10 (informative) | Source | Behaviour | Bearing on this spec | |---|---|---| @@ -562,6 +577,10 @@ Other server-side follow-ups, including security hardening, are tracked privatel | Azure SDK, `BearerTokenAuthenticationPolicy` | Retries only on a `401` that carries a Continuous Access Evaluation claims challenge. A plain `401` goes back to the caller. | A stricter variant of D2. | | Kafka: KIP-152 and KAFKA-6516 | Authentication failures are non-retriable at the API. The client is not closed, though, and keeps reconnecting in the background; a request to stop that was closed Won't Fix. | D1. | | Kafka: KAFKA-10840 (open) | When credentials expire, a consumer keeps failing authentication in the background and the application cannot see it. A proposed fix notes that Kafka Connect tasks report RUNNING meanwhile. | Why §8.4 exists. | +| Azure Identity: credential chains and the managed-identity retry strategy | `DefaultAzureCredential` runs managed identity in a "fail fast" mode, meant for the development inner loop: one IMDS probe with a short timeout and no retries. A "resilient" mode skips the probe and retries with exponential backoff; `AZURE_TOKEN_CREDENTIALS=ManagedIdentityCredential`, or the managed-identity credential used directly, enables it. Microsoft recommends a deterministic credential in production. | D10: a discovery chain's "no credential" is no evidence of misconfiguration; `azure_credential` selects the resilient mode. | +| MongoDB driver auth spec (MONGODB-OIDC) | The application names its environment (`ENVIRONMENT:azure`, `gcp` or `k8s`), and the driver calls that metadata endpoint directly, with no discovery. | D10: deterministic selection. | +| Google auth library, `ComputeEngineCredentials` | Detection pings the metadata server three times with a short timeout, "for developer desktop scenarios", then checks the SMBIOS product name, which needs no network. So a slow metadata server on a real VM is not taken for "not on GCE". A `503` from the metadata server is retryable. | D10: a probe that gets no answer is not proof; the local check is not adopted (§12). | +| AWS SDK for Java v2, issue #3939; Apache Druid | On EC2, the default chain sometimes reports "Unable to load credentials from any of the providers in the chain", which AWS attributes to IMDS latency. Druid now treats that error as recoverable within its retry budget. | D10: "nothing found" in a chain is retryable. | Links: @@ -573,3 +592,11 @@ Links: - KIP-152: https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures - KAFKA-6516: https://issues.apache.org/jira/browse/KAFKA-6516 - KAFKA-10840: https://issues.apache.org/jira/browse/KAFKA-10840 (proposed fix: https://github.com/apache/kafka/pull/16418) +- Azure Identity best practices (deterministic credentials, the managed-identity retry strategy): https://learn.microsoft.com/en-us/dotnet/azure/sdk/authentication/best-practices +- Azure Identity for Java, credential chains and `AZURE_TOKEN_CREDENTIALS`: https://learn.microsoft.com/en-us/azure/developer/java/sdk/authentication/credential-chains +- Azure SDK design note on IMDS probing: https://gist.github.com/ahsonkhan/d6c4d3a9780bb058a729b76844923ef1 +- Azure SDK release, October 2025 (resilient managed identity in C++, Go, Java, JavaScript and Python): https://devblogs.microsoft.com/azure-sdk/azure-sdk-release-october-2025/ +- Identify an Azure VM from the guest (SMBIOS asset tag, public cloud only): https://learn.microsoft.com/en-us/azure/virtual-machines/identify-azure-vm-from-guest +- Google `ComputeEngineCredentials`: https://github.com/googleapis/google-auth-library-java/blob/main/oauth2_http/java/com/google/auth/oauth2/ComputeEngineCredentials.java +- AWS SDK for Java v2, issue #3939: https://github.com/aws/aws-sdk-java-v2/issues/3939 +- Apache Druid, retry transient AWS credential resolution failures (#19558): https://www.mail-archive.com/commits@druid.apache.org/msg115181.html