From 4a6db57eb0f4cd0c2e2d82c1800fbdaeb77c904f Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 09:59:53 -0700 Subject: [PATCH 01/12] feat: select a wake-up adapter automatically A commit in one process did not wake a worker process unless somebody configured an adapter, so delivery waited out the polling interval. The adapters existed and were measured. Nothing chose between them. `wakeUp` now takes a name or an adapter, as `config.cache_store` does in Rails, and defaults to `"automatic"`. Selection prefers a configured Redis URL, then PostgreSQL notifications, then polling. An unknown name throws rather than polls quietly. Selection proves the PostgreSQL path before it chooses it. It listens on a probe channel, sends one `NOTIFY` from a second connection, and waits for it to arrive. A setting that survives one round trip proves nothing, because a transaction pooler can hand the same backend to two consecutive statements, so only a delivered notification counts. `runtime.wakeUpCapability()` reports what was installed, whether it crosses processes, its measured floor, and why. The doctor reports the same record, and the polling-only warning now fires on what was installed rather than on whether a setting was set. An adapter that declares its own capability keeps it, so a configured `InProcessWakeUpAdapter` still warns. Selection runs once per runtime. The probe opens connections and waits, so callers that race for the first use share one promise rather than run one probe each. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 23 +++ docs/api.md | 18 ++ docs/configuration.md | 44 +++-- docs/operations.md | 23 ++- docs/parity.md | 13 +- src/broadcast-worker.ts | 2 +- src/configuration.ts | 28 ++- src/database/types.ts | 3 + src/doctor.ts | 21 +++ src/effect-recovery-coordinator.ts | 4 +- src/effect-worker.ts | 2 +- src/index.ts | 11 ++ src/reminder-scheduler.ts | 2 +- src/repository.ts | 7 +- src/runtime.ts | 56 +++++- src/wake-up-selection.ts | 212 ++++++++++++++++++++++ src/wake-up.ts | 27 +++ src/wake-up/postgresql.ts | 20 +- src/wake-up/redis.ts | 8 + src/worker.ts | 2 +- test/effect-recovery.test.ts | 18 +- test/polling-loop.test.ts | 40 +++- test/postgresql.test.ts | 35 ++++ test/wake-up-selection.test.ts | 281 +++++++++++++++++++++++++++++ 24 files changed, 849 insertions(+), 51 deletions(-) create mode 100644 src/wake-up-selection.ts create mode 100644 test/wake-up-selection.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 15d6144..c84e513 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,29 @@ ## Unreleased +- Select a wake-up adapter automatically. `wakeUp` now takes a name or an + adapter, as `config.cache_store` does in Rails, and defaults to + `"automatic"`. Selection prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL + notifications, then polling. `"in_process"` opts out, and an unknown name + throws rather than polls quietly. +- PostgreSQL applications that configure nothing now use notifications. They + gain cross-process wake-up, one dedicated listening client, and one `NOTIFY` + per commit. Set `wakeUp: "in_process"` to keep polling. +- Prove the PostgreSQL notification path before selecting it, because `LISTEN` + does not survive a transaction pooler such as PgBouncer. Selection listens on + a probe channel, sends one `NOTIFY` from a second connection, and waits up to + two seconds for it to arrive. A probe that does not deliver falls back to + polling and warns once. +- Report the resolved choice. `runtime.wakeUpCapability()` names the adapter, + whether it crosses processes, its measured floor, and why it was chosen. The + doctor reports it as a `wakeUp` check, and the polling-only warning now fires + on what was installed rather than on whether a setting was set. +- Keep the capability a configured adapter reports about itself. A configured + `InProcessWakeUpAdapter` now reports `in_process` and warns, rather than + claim that it crosses processes. +- Select once per runtime. `runtime.wakeUpAdapter()` memoises the selection, so + callers that race for the first use share one probe rather than run one each. + - Add reminder reading. `reminder()` returns one armed alarm as a `ScheduledReminder`, and `reminders()` lists every key of one operation. Both apply the intents staged so far in the turn, so a read agrees with what the diff --git a/docs/api.md b/docs/api.md index 204aefe..8c4d5d3 100644 --- a/docs/api.md +++ b/docs/api.md @@ -664,6 +664,24 @@ wait. `WakeUpWatch.wait()` returns `true` for a notification and `false` for a timeout or cancellation. A legacy `void` result remains accepted and preserves the fast polling cadence. +`configuration.wakeUp` takes a name or an adapter. `WakeUpSetting` is that +union, `WakeUpName` is one of `WAKE_UP_NAMES`, and the default is +`"automatic"`. Selection prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL +notifications, then polling. An unknown name throws rather than polls. + +`selectWakeUp(options)` runs that choice and returns a `SelectedWakeUp`, which +pairs the adapter with a `WakeUpCapability`. `WakeUpSelectionOptions` names the +inputs: the setting, the database, the idle polling interval, a logger, and an +optional Redis URL and probe timeout. `WakeUpCapability` reports +`WakeUpAdapterName`, whether the adapter crosses processes, the measured floor +in milliseconds, and the reason. `runtime.wakeUpAdapter()` and +`runtime.wakeUpCapability()` return the choice, which is made once per runtime. + +An adapter may declare its own `capability`, and selection keeps what it +declares rather than assume. `NotificationWakeUpAdapter` adds +`channelFor(role)`, which a database that offers a notification channel +provides through `database.wakeUp(options)`. + ### Errors The root exports `SolidObjectsError` and its supported subclasses: diff --git a/docs/configuration.md b/docs/configuration.md index 6c24112..4b408f0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -49,19 +49,19 @@ as an instruction to divide the actor. See ## Runtime roles and supervision -| Option | Default | Contract | -| ------------------------------------------- | -----------------------: | -------------------------------------------------------- | -| `workerCount` | `1` | Non-negative actor workers. | -| `effectWorkerCount` | `1` | Non-negative effect workers. | -| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. | -| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. | -| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. | -| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. | -| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. | -| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. | -| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. | -| `wakeUp` | `InProcessWakeUpAdapter` | Adapter implementing `watch`, `notify`, and `close`. | -| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. | +| Option | Default | Contract | +| ------------------------------------------- | --------------: | ----------------------------------------------------------------- | +| `workerCount` | `1` | Non-negative actor workers. | +| `effectWorkerCount` | `1` | Non-negative effect workers. | +| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. | +| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. | +| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. | +| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. | +| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. | +| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. | +| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. | +| `wakeUp` | `"automatic"` | A name or an adapter implementing `watch`, `notify`, and `close`. | +| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. | Counts may be zero, but the complete configuration must leave at least one runtime role enabled. Broadcast workers are started only when `broadcast` or @@ -158,6 +158,24 @@ short capped backoff only when no synchronous deadline is active. | `applicationName` | `"solid-objects"` | | `onPoolError` | structured `console.error` | +`wakeUp` takes a name or an adapter, as `config.cache_store` does in Rails. +`"automatic"` is the default. It prefers `SOLID_OBJECTS_REDIS_URL`, then +PostgreSQL notifications, then polling. `"in_process"` opts out, `"postgresql"` +and `"redis"` force one, and an unknown name throws rather than polls quietly. + +Selection runs once per runtime, on first use. `runtime.wakeUpCapability()` +reports what it chose, whether that choice crosses processes, its measured +floor in milliseconds, and why. The doctor reports the same record, and the +polling-only warning fires on what was installed rather than on whether a +setting was set. An adapter that declares its own `capability` keeps it, so a +configured `InProcessWakeUpAdapter` still warns. + +Automatic selection proves the PostgreSQL path before it chooses it. It listens +on a probe channel, sends one `NOTIFY` from a second connection, and waits up to +two seconds for it to arrive. A probe that does not deliver falls back to +polling and warns once, because `LISTEN` does not survive a transaction pooler +such as PgBouncer. + `database.wakeUp(options)` creates a dedicated notification adapter using the same connection string. Its options are `channelPrefix = "solid_objects"`, `applicationName = "solid-objects-wake-up"`, and `onListenerError`. Use a diff --git a/docs/operations.md b/docs/operations.md index e17def4..bfb00d1 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -14,16 +14,25 @@ work and wake-up notifications reset the role to the fast interval. Actor workers clamp the ceiling to `leaseRenewalIntervalMilliseconds` while they may hold cached activations. -The default generation-based wake-up adapter interrupts waits for new actor +The runtime selects a wake-up adapter on first use. It prefers +`SOLID_OBJECTS_REDIS_URL`, then PostgreSQL notifications, then polling. +The generation-based in-process adapter interrupts waits for new actor messages, effects, reminders, and broadcasts in the same Node process. It does -not cross a process boundary. When live processes share the database without a -configured adapter, the runtime logs -`solid_objects.polling_only_cross_process_wake_up` once. Use PostgreSQL -notifications or optional Redis Pub/Sub when separate processes need prompt -delivery. Without one of them, newly committed work can wait for the current -idle polling interval. The runtime isolates notification errors and logs them by +not cross a process boundary. When live processes share the database and the +installed adapter does not cross processes, the runtime logs +`solid_objects.polling_only_cross_process_wake_up` once. Without a +cross-process adapter, newly committed work can wait for the current idle +polling interval. The runtime isolates notification errors and logs them by role and error class. The committed work does not fail. +`runtime.wakeUpCapability()` and the `wakeUp` doctor check report the adapter +that is installed, whether it crosses processes, its measured floor, and why it +was chosen. On PostgreSQL, selection first proves the path: it listens on a +probe channel, notifies it from a second connection, and waits for the +notification. A probe that does not arrive logs +`solid_objects.wake_up.pooled_session` once and falls back to polling, because +`LISTEN` does not survive a transaction pooler such as PgBouncer. + The warning excludes process rows with the current hostname and host process ID. It can therefore appear during a rolling deployment or restart overlap when an older and newer process briefly share the same database. A process that stopped diff --git a/docs/parity.md b/docs/parity.md index 7410553..cd1254d 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -106,11 +106,14 @@ Ruby field names; this does not change runtime delivery semantics. | PostgreSQL wake-up | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback. | | Redis wake-up | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback. | -Every wake-up adapter above is opt-in. Neither runtime selects one -automatically. An application that configures nothing keeps polling. Each -runtime warns once when live processes share a database without a configured -cross-process adapter. This limit is intentional in both runtimes. It is not a -gap between them. +Both runtimes select a wake-up adapter automatically. `wakeUp` takes a name or +an adapter and defaults to `"automatic"`, which prefers a configured Redis URL, +then PostgreSQL notifications, then polling. Selection proves the PostgreSQL +path with a probe notification, because `LISTEN` does not survive a transaction +pooler. Each runtime reports what it installed, whether that crosses processes, +its measured floor, and why, and warns once when live processes share a +database and the installed adapter cannot reach them. MySQL still polls in both +runtimes, because MySQL has no notification channel. ## Realtime and browser behavior diff --git a/src/broadcast-worker.ts b/src/broadcast-worker.ts index 670ee47..f31892e 100644 --- a/src/broadcast-worker.ts +++ b/src/broadcast-worker.ts @@ -54,7 +54,7 @@ export class BroadcastWorker { await this.ensureRegistered() await this.runtime.warnIfPollingIsOnlyCrossProcessWakeUp() while (!signal.aborted && !this.stopping) { - const wakeUp = await this.runtime.settings.wakeUp.watch("broadcasts") + const wakeUp = await this.runtime.watchWakeUp("broadcasts") const processed = await this.runOnce() if (processed > 0) { this.pollingBackoff.reset("work") diff --git a/src/configuration.ts b/src/configuration.ts index ed27fe9..9bae301 100644 --- a/src/configuration.ts +++ b/src/configuration.ts @@ -1,7 +1,7 @@ import { InvalidActor } from "./errors.js" import type { Database } from "./database/types.js" import type { DeepReadonly, JsonObject, JsonValue, Logger, LongRunningComponent } from "./types.js" -import { InProcessWakeUpAdapter, type WakeUpAdapter } from "./wake-up.js" +import { WAKE_UP_NAMES, type WakeUpName, type WakeUpSetting } from "./wake-up.js" export interface AuthorizationInput { actorType: string @@ -79,7 +79,7 @@ export interface SolidObjectsConfiguration { authorizeSubscription?: (input: SubscriptionAuthorizationInput) => boolean | Promise instrumentation?: (event: InstrumentationEvent) => void broadcast?: (event: BroadcastEvent) => Promise - wakeUp?: WakeUpAdapter + wakeUp?: WakeUpSetting } export interface BroadcastEvent { @@ -97,8 +97,7 @@ export interface RuntimeSettings extends Required< logger: Logger broadcast?: (event: BroadcastEvent) => Promise instrumentation?: (event: InstrumentationEvent) => void - wakeUp: WakeUpAdapter - wakeUpConfigured: boolean + wakeUp: WakeUpSetting authorizationPoliciesConfigured: Readonly> } @@ -156,8 +155,7 @@ export function buildSettings(configuration: SolidObjectsConfiguration): Runtime processRetentionMilliseconds: configuration.processRetentionMilliseconds ?? 7 * 86_400_000, pruneBatchSize: configuration.pruneBatchSize ?? 1_000, logger: configuration.logger ?? consoleLogger, - wakeUp: configuration.wakeUp ?? new InProcessWakeUpAdapter(), - wakeUpConfigured: configuration.wakeUp !== undefined, + wakeUp: configuration.wakeUp ?? "automatic", authorizeMessage: configuration.authorizeMessage ?? (() => false), authorizeQuery: configuration.authorizeQuery ?? (() => false), authorizeDestroy: configuration.authorizeDestroy ?? (() => false), @@ -193,12 +191,22 @@ export function validateComponent(component: LongRunningComponent): void { } } -function validateSettings(settings: RuntimeSettings): void { - for (const name of ["watch", "notify", "close"] as const) { - if (typeof settings.wakeUp[name] !== "function") { - throw new TypeError(`wakeUp must implement ${name}`) +function validateWakeUp(setting: WakeUpSetting): void { + if (typeof setting !== "string") { + for (const name of ["watch", "notify", "close"] as const) { + if (typeof setting[name] !== "function") throw new TypeError(`wakeUp must implement ${name}`) } + return } + if (WAKE_UP_NAMES.includes(setting as WakeUpName)) return + + throw new TypeError( + `unknown wakeUp ${JSON.stringify(setting)}, expected one of ${WAKE_UP_NAMES.join(", ")} or an adapter`, + ) +} + +function validateSettings(settings: RuntimeSettings): void { + validateWakeUp(settings.wakeUp) if (!/^[a-z][a-z0-9_]*$/.test(settings.tableNamePrefix)) { throw new TypeError("tableNamePrefix must contain lowercase letters, digits, and underscores") } diff --git a/src/database/types.ts b/src/database/types.ts index e8cb383..f061432 100644 --- a/src/database/types.ts +++ b/src/database/types.ts @@ -1,3 +1,5 @@ +import type { NotificationWakeUpAdapter } from "../wake-up.js" + export type DatabaseFamily = "sqlite" | "postgresql" | "mysql" export interface RunResult { @@ -20,6 +22,7 @@ export interface Database { readonly family: DatabaseFamily readonly schemaIdentity: string transactionActive?(): boolean + wakeUp?(options?: { channelPrefix?: string }): NotificationWakeUpAdapter connection(callback: (connection: DatabaseConnection) => Promise): Promise transaction( callback: (connection: DatabaseConnection) => Promise, diff --git a/src/doctor.ts b/src/doctor.ts index ad65018..19cfc9c 100644 --- a/src/doctor.ts +++ b/src/doctor.ts @@ -117,6 +117,7 @@ export class Doctor { schema, await this.checkAuthorization(), await this.checkDatabase(), + await this.checkWakeUp(), ] checks.push( schema.status === "fail" @@ -145,6 +146,26 @@ export class Doctor { }) } + private async checkWakeUp(): Promise { + try { + const capability = await this.runtime.wakeUpCapability() + const floor = capability.measuredFloorMilliseconds + const summary = `${capability.adapter}: ${capability.reason}${ + floor === undefined ? "" : `, floor ${floor} ms` + }` + if (capability.crossesProcesses) { + return check({ name: "wakeUp", status: "pass", message: summary }) + } + return check({ + name: "wakeUp", + status: "warn", + message: `${summary}; a commit in one process cannot wake another`, + }) + } catch (error) { + return failedCheck("wakeUp", error) + } + } + private checkConfiguration(): DoctorCheck { return check({ name: "configuration", diff --git a/src/effect-recovery-coordinator.ts b/src/effect-recovery-coordinator.ts index b6a4574..8146f23 100644 --- a/src/effect-recovery-coordinator.ts +++ b/src/effect-recovery-coordinator.ts @@ -9,6 +9,7 @@ import { } from "./effect-recovery.js" import type { EffectRow, EnqueueInput, MessageRow, ProcessRow } from "./records.js" import { jsonObject, normalizeJson } from "./serialization.js" +import type { WakeUpAdapter } from "./wake-up.js" import { notifyWakeUp } from "./wake-up-notification.js" interface RecoveryBinding { @@ -30,6 +31,7 @@ export class EffectRecoveryCoordinator { constructor( private readonly options: { settings: RuntimeSettings + wakeUpAdapter: () => Promise enqueue: (connection: DatabaseConnection, input: EnqueueInput) => Promise }, ) {} @@ -87,7 +89,7 @@ export class EffectRecoveryCoordinator { }) if (retired) notifyWakeUp({ - adapter: this.options.settings.wakeUp, + adapter: await this.options.wakeUpAdapter(), logger: this.options.settings.logger, role: "actors", }) diff --git a/src/effect-worker.ts b/src/effect-worker.ts index 68f4653..3446d4d 100644 --- a/src/effect-worker.ts +++ b/src/effect-worker.ts @@ -54,7 +54,7 @@ export class EffectWorker { await this.ensureRegistered() await this.runtime.warnIfPollingIsOnlyCrossProcessWakeUp() while (!signal.aborted && !this.stopping) { - const wakeUp = await this.runtime.settings.wakeUp.watch("effects") + const wakeUp = await this.runtime.watchWakeUp("effects") const processed = await this.runOnce() if (processed > 0) { this.pollingBackoff.reset("work") diff --git a/src/index.ts b/src/index.ts index 769b8b6..2e0ec5a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -51,11 +51,22 @@ export { export { BroadcastWorker } from "./broadcast-worker.js" export { InProcessWakeUpAdapter, + WAKE_UP_NAMES, + type NotificationWakeUpAdapter, type WakeUpAdapter, + type WakeUpAdapterName, + type WakeUpCapability, + type WakeUpName, type WakeUpRole, + type WakeUpSetting, type WakeUpWaitOptions, type WakeUpWatch, } from "./wake-up.js" +export { + selectWakeUp, + type SelectedWakeUp, + type WakeUpSelectionOptions, +} from "./wake-up-selection.js" export { parseSubscriptionRequest, RealtimeManager, diff --git a/src/reminder-scheduler.ts b/src/reminder-scheduler.ts index 96b7fe8..cfdcaa2 100644 --- a/src/reminder-scheduler.ts +++ b/src/reminder-scheduler.ts @@ -75,7 +75,7 @@ export class ReminderScheduler { await this.ensureRegistered() await this.runtime.warnIfPollingIsOnlyCrossProcessWakeUp() while (!signal.aborted && !this.stopping) { - const wakeUp = await this.runtime.settings.wakeUp.watch("reminders") + const wakeUp = await this.runtime.watchWakeUp("reminders") const processed = await this.runOnce() if (processed > 0) { this.pollingBackoff.reset("work") diff --git a/src/repository.ts b/src/repository.ts index 0e7296d..4840d28 100644 --- a/src/repository.ts +++ b/src/repository.ts @@ -36,6 +36,7 @@ import type { } from "./types.js" import { VERSION } from "./version.js" import { EffectRecoveryCoordinator } from "./effect-recovery-coordinator.js" +import type { WakeUpAdapter } from "./wake-up.js" export interface SyncDiagnosticsRecord { message: MessageRow @@ -67,7 +68,10 @@ interface MessageClaimFenceRow { } export class Repository { - constructor(private readonly settings: RuntimeSettings) {} + constructor( + private readonly settings: RuntimeSettings, + private readonly options: { wakeUpAdapter: () => Promise }, + ) {} table(name: string): string { return `${this.settings.tableNamePrefix}${name}` @@ -2056,6 +2060,7 @@ export class Repository { private effectRecoveryCoordinator(): EffectRecoveryCoordinator { return new EffectRecoveryCoordinator({ settings: this.settings, + wakeUpAdapter: this.options.wakeUpAdapter, enqueue: (connection, input) => this.enqueueInTransaction(connection, input), }) } diff --git a/src/runtime.ts b/src/runtime.ts index 8af5fee..c333829 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -122,8 +122,9 @@ import type { import { SolidObjectsTestHelper } from "./test-helper.js" import { waitFor, Worker } from "./worker.js" import { EffectWorker } from "./effect-worker.js" -import type { WakeUpRole } from "./wake-up.js" +import type { WakeUpAdapter, WakeUpCapability, WakeUpRole, WakeUpWatch } from "./wake-up.js" import { notifyWakeUp } from "./wake-up-notification.js" +import { selectWakeUp, type SelectedWakeUp } from "./wake-up-selection.js" import { withDatabaseDeadline } from "./database/deadline.js" import type { DatabaseConnection } from "./database/types.js" import { evaluateActorTurn, readActorObservables } from "./turn.js" @@ -201,10 +202,13 @@ export class SolidObjectsRuntime { private running = false private pollingOnlyWakeUpWarningEmitted = false private pollingOnlyWakeUpWarningCheck: Promise | undefined + private wakeUpSelection: Promise | undefined constructor(configuration: SolidObjectsConfiguration) { this.settings = buildSettings(configuration) - this.repository = new Repository(this.settings) + this.repository = new Repository(this.settings, { + wakeUpAdapter: () => this.wakeUpAdapter(), + }) this.deadLetters = new DeadLetterManager(this) this.reconciliation = new ReconciliationManager(this) this.retention = new RetentionManager(this) @@ -1344,13 +1348,25 @@ export class SolidObjectsRuntime { throw new Error("abort runtime.run() and wait for it before closing the runtime") await this.callerWorker?.stop() this.realtime.close() - await this.settings.wakeUp.close() + await this.closeWakeUp() await this.settings.database.close() clearDefaultRuntime(this) } + async wakeUpAdapter(): Promise { + return (await this.resolveWakeUp()).adapter + } + + async wakeUpCapability(): Promise { + return (await this.resolveWakeUp()).capability + } + + async watchWakeUp(role: WakeUpRole): Promise { + return await (await this.wakeUpAdapter()).watch(role) + } + async warnIfPollingIsOnlyCrossProcessWakeUp(): Promise { - if (this.settings.wakeUpConfigured || this.pollingOnlyWakeUpWarningEmitted) return + if (this.pollingOnlyWakeUpWarningEmitted) return if (this.pollingOnlyWakeUpWarningCheck) return this.pollingOnlyWakeUpWarningCheck const check = this.checkPollingOnlyCrossProcessWakeUp() this.pollingOnlyWakeUpWarningCheck = check @@ -1364,6 +1380,7 @@ export class SolidObjectsRuntime { } private async checkPollingOnlyCrossProcessWakeUp(): Promise { + if ((await this.wakeUpCapability()).crossesProcesses) return if (!(await this.repository.hasLiveProcessOutsideCurrentHostProcess())) return if (this.pollingOnlyWakeUpWarningEmitted) return this.pollingOnlyWakeUpWarningEmitted = true @@ -1383,6 +1400,7 @@ export class SolidObjectsRuntime { await this.callerWorker?.stop() this.callerWorker = undefined this.realtime.close() + await this.closeWakeUp() await this.repository.resetForTesting() } @@ -1963,8 +1981,36 @@ export class SolidObjectsRuntime { }) } + private resolveWakeUp(): Promise { + this.wakeUpSelection ??= selectWakeUp({ + setting: this.settings.wakeUp, + database: this.settings.database, + idlePollingIntervalMilliseconds: this.settings.idlePollingIntervalMilliseconds, + logger: this.settings.logger, + }) + return this.wakeUpSelection + } + + private async closeWakeUp(): Promise { + const selection = this.wakeUpSelection + this.wakeUpSelection = undefined + if (!selection) return + const selected = await selection.catch(() => undefined) + await selected?.adapter.close() + } + private wakeUp(role: WakeUpRole): void { - notifyWakeUp({ adapter: this.settings.wakeUp, logger: this.settings.logger, role }) + void this.resolveWakeUp() + .then((selected) => + notifyWakeUp({ adapter: selected.adapter, logger: this.settings.logger, role }), + ) + .catch((error: unknown) => { + this.settings.logger.error({ + event: "solid_objects.wake_up.failed", + role, + errorName: error instanceof Error ? error.name : "Error", + }) + }) } private async authorize(options: { diff --git a/src/wake-up-selection.ts b/src/wake-up-selection.ts new file mode 100644 index 0000000..387ff4b --- /dev/null +++ b/src/wake-up-selection.ts @@ -0,0 +1,212 @@ +import type { Database } from "./database/types.js" +import type { Logger } from "./types.js" +import { + InProcessWakeUpAdapter, + WAKE_UP_NAMES, + type NotificationWakeUpAdapter, + type WakeUpAdapter, + type WakeUpCapability, + type WakeUpSetting, +} from "./wake-up.js" + +export const REDIS_URL_VARIABLE = "SOLID_OBJECTS_REDIS_URL" +export const POSTGRESQL_FLOOR_MILLISECONDS = 2.9 +export const REDIS_FLOOR_MILLISECONDS = 5.7 +export const PROBE_TIMEOUT_MILLISECONDS = 2_000 +export const PROBE_CHANNEL_PREFIX = "solid_objects_probe" + +const PROBE_ROLE = "actors" as const + +export interface SelectedWakeUp { + readonly adapter: WakeUpAdapter + readonly capability: WakeUpCapability +} + +export interface WakeUpSelectionOptions { + setting: WakeUpSetting + database: Database + idlePollingIntervalMilliseconds: number + logger: Logger + redisUrl?: string + probeTimeoutMilliseconds?: number +} + +export async function selectWakeUp(options: WakeUpSelectionOptions): Promise { + const { setting } = options + if (typeof setting !== "string") return configured(setting) + if (setting === "automatic") return await automatic(options) + return await named({ ...options, name: setting }) +} + +function configured(adapter: WakeUpAdapter): SelectedWakeUp { + if (adapter.capability) return { adapter, capability: adapter.capability } + return { + adapter, + capability: { + adapter: "configured", + crossesProcesses: true, + reason: "an adapter was configured, so selection did not run", + }, + } +} + +async function named(options: WakeUpSelectionOptions & { name: string }): Promise { + if (options.name === "in_process") { + return { + adapter: new InProcessWakeUpAdapter(), + capability: { + adapter: "in_process", + crossesProcesses: false, + reason: "in-process signalling was requested", + }, + } + } + if (options.name === "postgresql") { + return { + adapter: notificationAdapter(options.database), + capability: { + adapter: "postgresql_notify", + crossesProcesses: true, + measuredFloorMilliseconds: POSTGRESQL_FLOOR_MILLISECONDS, + reason: "PostgreSQL LISTEN was requested", + }, + } + } + if (options.name === "redis") { + return await redisSelection({ + url: requiredRedisUrl(options), + reason: "Redis was requested", + }) + } + throw new TypeError( + `unknown wakeUp ${JSON.stringify(options.name)}, expected one of ${WAKE_UP_NAMES.join(", ")} or an adapter`, + ) +} + +async function automatic(options: WakeUpSelectionOptions): Promise { + const url = redisUrl(options) + if (url !== undefined) { + return await redisSelection({ + url, + reason: `${REDIS_URL_VARIABLE} is set, so Redis carries the signal between processes`, + }) + } + if (options.database.family !== "postgresql" || !options.database.wakeUp) { + return polling({ + options, + reason: `${options.database.family} has no notification channel and ${REDIS_URL_VARIABLE} is not set`, + }) + } + return await postgresqlSelection(options) +} + +async function postgresqlSelection(options: WakeUpSelectionOptions): Promise { + if (!(await notificationsDeliver(options))) return pooled(options) + return { + adapter: notificationAdapter(options.database), + capability: { + adapter: "postgresql_notify", + crossesProcesses: true, + measuredFloorMilliseconds: POSTGRESQL_FLOOR_MILLISECONDS, + reason: + "a probe notification arrived, so PostgreSQL LISTEN carries the signal between processes", + }, + } +} + +async function notificationsDeliver(options: WakeUpSelectionOptions): Promise { + const probe = notificationAdapter(options.database, { channelPrefix: PROBE_CHANNEL_PREFIX }) + try { + const watch = await probe.watch(PROBE_ROLE) + await options.database.connection((connection) => + connection.run("SELECT pg_notify(?, ?)", [probe.channelFor(PROBE_ROLE), PROBE_ROLE]), + ) + const timeoutMilliseconds = options.probeTimeoutMilliseconds ?? PROBE_TIMEOUT_MILLISECONDS + return (await watch.wait({ timeoutMilliseconds })) === true + } catch { + return false + } finally { + await closeQuietly(probe) + } +} + +function pooled(options: WakeUpSelectionOptions): SelectedWakeUp { + options.logger.warn({ + event: "solid_objects.wake_up.pooled_session", + reason: + "PostgreSQL notifications were not selected because a probe notification did not arrive", + }) + return polling({ + options, + reason: + "a probe notification did not arrive, so LISTEN cannot carry the signal between processes; " + + "a transaction pooler such as PgBouncer is the usual cause", + }) +} + +function polling(input: { options: WakeUpSelectionOptions; reason: string }): SelectedWakeUp { + return { + adapter: new InProcessWakeUpAdapter(), + capability: { + adapter: "polling", + crossesProcesses: false, + measuredFloorMilliseconds: input.options.idlePollingIntervalMilliseconds, + reason: input.reason, + }, + } +} + +async function redisSelection(input: { url: string; reason: string }): Promise { + const { RedisWakeUpAdapter } = await importRedis() + return { + adapter: new RedisWakeUpAdapter({ url: input.url }), + capability: { + adapter: "redis", + crossesProcesses: true, + measuredFloorMilliseconds: REDIS_FLOOR_MILLISECONDS, + reason: input.reason, + }, + } +} + +async function importRedis(): Promise { + try { + return await import("./wake-up/redis.js") + } catch (error) { + throw new TypeError("the redis package is required for the Redis wake-up adapter", { + cause: error, + }) + } +} + +function notificationAdapter( + database: Database, + options: { channelPrefix?: string } = {}, +): NotificationWakeUpAdapter { + if (!database.wakeUp) { + throw new TypeError(`the ${database.family} database does not provide a notification channel`) + } + return database.wakeUp(options) +} + +function redisUrl(options: WakeUpSelectionOptions): string | undefined { + const value = options.redisUrl ?? process.env[REDIS_URL_VARIABLE] + if (value === undefined || value.length === 0) return undefined + return value +} + +function requiredRedisUrl(options: WakeUpSelectionOptions): string { + const url = redisUrl(options) + if (url === undefined) { + throw new TypeError(`wakeUp "redis" requires ${REDIS_URL_VARIABLE} or an explicit adapter`) + } + return url +} + +async function closeQuietly(adapter: WakeUpAdapter): Promise { + try { + await adapter.close() + } catch { + return + } +} diff --git a/src/wake-up.ts b/src/wake-up.ts index fc451f5..8f2ae04 100644 --- a/src/wake-up.ts +++ b/src/wake-up.ts @@ -9,13 +9,40 @@ export interface WakeUpWatch { wait(options: WakeUpWaitOptions): Promise } +export type WakeUpAdapterName = + "in_process" | "polling" | "postgresql_notify" | "redis" | "configured" + +export interface WakeUpCapability { + readonly adapter: WakeUpAdapterName + readonly crossesProcesses: boolean + readonly measuredFloorMilliseconds?: number + readonly reason: string +} + +export const WAKE_UP_NAMES = ["automatic", "in_process", "postgresql", "redis"] as const + +export type WakeUpName = (typeof WAKE_UP_NAMES)[number] + export interface WakeUpAdapter { watch(role: WakeUpRole): WakeUpWatch | Promise notify(role: WakeUpRole): void | Promise close(): void | Promise + readonly capability?: WakeUpCapability } +export interface NotificationWakeUpAdapter extends WakeUpAdapter { + channelFor(role: WakeUpRole): string +} + +export type WakeUpSetting = WakeUpName | WakeUpAdapter + export class InProcessWakeUpAdapter implements WakeUpAdapter { + readonly capability: WakeUpCapability = { + adapter: "in_process", + crossesProcesses: false, + reason: "in-process signalling, which a commit in another process cannot reach", + } + private readonly generations = new Map() private readonly waiters = new Map void>>() private closed = false diff --git a/src/wake-up/postgresql.ts b/src/wake-up/postgresql.ts index 14f297b..5e29cc9 100644 --- a/src/wake-up/postgresql.ts +++ b/src/wake-up/postgresql.ts @@ -1,5 +1,10 @@ import { Client, type ClientConfig, type Notification } from "pg" -import type { WakeUpAdapter, WakeUpRole, WakeUpWatch } from "../wake-up.js" +import type { + NotificationWakeUpAdapter, + WakeUpCapability, + WakeUpRole, + WakeUpWatch, +} from "../wake-up.js" const ROLES = [ "actors", @@ -20,7 +25,14 @@ export interface PostgreSQLWakeUpOptions { onListenerError?: (failure: PostgreSQLWakeUpFailure) => void } -export class PostgreSQLWakeUpAdapter implements WakeUpAdapter { +export class PostgreSQLWakeUpAdapter implements NotificationWakeUpAdapter { + readonly capability: WakeUpCapability = { + adapter: "postgresql_notify", + crossesProcesses: true, + measuredFloorMilliseconds: 2.9, + reason: "PostgreSQL LISTEN carries the signal between processes", + } + private readonly clientConfiguration: ClientConfig private readonly channels = new Map() private readonly rolesByChannel = new Map() @@ -65,6 +77,10 @@ export class PostgreSQLWakeUpAdapter implements WakeUpAdapter { })) } + channelFor(role: WakeUpRole): string { + return this.channel(role) + } + async watch(role: WakeUpRole): Promise { await this.ensureListening(role) const generation = this.generation(role) diff --git a/src/wake-up/redis.ts b/src/wake-up/redis.ts index 82ddd6d..d89e8f7 100644 --- a/src/wake-up/redis.ts +++ b/src/wake-up/redis.ts @@ -2,6 +2,7 @@ import { createClient, type RedisClientType } from "redis" import { InProcessWakeUpAdapter, type WakeUpAdapter, + type WakeUpCapability, type WakeUpRole, type WakeUpWatch, } from "../wake-up.js" @@ -26,6 +27,13 @@ export interface RedisWakeUpOptions { } export class RedisWakeUpAdapter implements WakeUpAdapter { + readonly capability: WakeUpCapability = { + adapter: "redis", + crossesProcesses: true, + measuredFloorMilliseconds: 5.7, + reason: "Redis carries the signal between processes", + } + private readonly local = new InProcessWakeUpAdapter() private readonly publisher: RedisClientType private readonly subscriber: RedisClientType diff --git a/src/worker.ts b/src/worker.ts index c1e5e0a..fbc68e8 100644 --- a/src/worker.ts +++ b/src/worker.ts @@ -141,7 +141,7 @@ export class Worker { await this.ensureRegistered() await this.runtime.warnIfPollingIsOnlyCrossProcessWakeUp() while (!signal.aborted && !this.stopping) { - const wakeUp = await this.runtime.settings.wakeUp.watch("actors") + const wakeUp = await this.runtime.watchWakeUp("actors") const processed = await this.runOnce() if (processed > 0) { this.pollingBackoff.reset("work") diff --git a/test/effect-recovery.test.ts b/test/effect-recovery.test.ts index 23202c0..e8a5c55 100644 --- a/test/effect-recovery.test.ts +++ b/test/effect-recovery.test.ts @@ -134,7 +134,7 @@ it("retires abandoned processing effects before the scheduler can reclaim them", ), ) await runtime.repository.registerProcess("replacement", "effect") - const wakeUp = await runtime.settings.wakeUp.watch("actors") + const wakeUp = await runtime.watchWakeUp("actors") expect(await runtime.repository.claimEffect("replacement")).toBeUndefined() expect(await wakeUp.wait({ timeoutMilliseconds: 0 })).toBe(true) const notifications = await runtime.settings.database.connection((connection) => @@ -237,6 +237,7 @@ it("rolls back retirement if the second mailbox insert fails", async () => { const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => { if (input.operation === "inspect") throw new Error("injected second insert failure") @@ -267,6 +268,7 @@ it("reports missing from the owned binding without exposing another actor", asyn await runtime!.ref(ReportExport, "export").check() expect(await messages("inspect")).toEqual([{ effectId: effect.id, outcome: "missing" }]) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -286,6 +288,7 @@ it("rolls back both callbacks when only one mailbox slot remains", async () => { await ageOwner(70_000) runtime!.settings.maxMailboxLength = 1 const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -308,6 +311,7 @@ it("surfaces an owner query failure without deciding abandonment", async () => { const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -362,6 +366,7 @@ it("requires recovery opt-in for a timeout and both bindings for an explicit che connection.get(`SELECT * FROM ${runtime!.repository.table("effects")}`), ) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -450,6 +455,7 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -480,6 +486,7 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -511,6 +518,7 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -620,10 +628,14 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( database: runtime.settings.database, table: "effects", }) - const claimant = new Repository({ ...runtime.settings, database: pausedDatabase }) + const claimant = new Repository( + { ...runtime.settings, database: pausedDatabase }, + { wakeUpAdapter: () => runtime!.wakeUpAdapter() }, + ) const claim = claimant.claimEffect("owner") const originLocked = deferred() const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -659,6 +671,7 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( const effect = await storedEffect(handle!.id) await runtime.repository.registerProcess("owner", "effect") const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) @@ -683,6 +696,7 @@ it.skipIf(!process.env.SOLID_OBJECTS_DATABASE_URL?.startsWith("postgresql:"))( const effect = await processingEffect() await ageOwner(70_000) const coordinator = new EffectRecoveryCoordinator({ + wakeUpAdapter: () => runtime!.wakeUpAdapter(), settings: runtime!.settings, enqueue: (connection, input) => runtime!.repository.enqueueInTransaction(connection, input), }) diff --git a/test/polling-loop.test.ts b/test/polling-loop.test.ts index 91f7d86..2b3918d 100644 --- a/test/polling-loop.test.ts +++ b/test/polling-loop.test.ts @@ -2,7 +2,12 @@ import { afterEach, describe, expect, it, vi } from "vitest" import { sqlite } from "../src/database/sqlite.js" import { createRuntime, type SolidObjectsRuntime } from "../src/runtime.js" import type { InstrumentationEvent } from "../src/configuration.js" -import type { WakeUpAdapter, WakeUpRole, WakeUpWatch } from "../src/wake-up.js" +import { + InProcessWakeUpAdapter, + type WakeUpAdapter, + type WakeUpRole, + type WakeUpWatch, +} from "../src/wake-up.js" let runtime: SolidObjectsRuntime | undefined @@ -365,6 +370,39 @@ describe("idle polling", () => { expect(logger.warn).not.toHaveBeenCalled() }) + + it("warns when the configured adapter reports that it stays in one process", async () => { + const logger = { + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + } + runtime = createRuntime({ + database: sqlite({ path: ":memory:" }), + workerCount: 1, + effectWorkerCount: 0, + reminderSchedulerCount: 0, + retentionIntervalMilliseconds: 0, + deadProcessCleanupIntervalMilliseconds: 0, + logger, + wakeUp: new InProcessWakeUpAdapter(), + }) + await runtime.install() + await runtime.repository.registerProcess("other-process", "worker") + await runtime.settings.database.connection((connection) => + connection.run( + `UPDATE ${runtime?.repository.table("processes")} SET host_process_id = ? WHERE id = ?`, + [process.pid + 1, "other-process"], + ), + ) + + await runtime.warnIfPollingIsOnlyCrossProcessWakeUp() + + expect(logger.warn).toHaveBeenCalledWith( + expect.objectContaining({ event: "solid_objects.polling_only_cross_process_wake_up" }), + ) + }) }) class ImmediateTimeoutWakeUpAdapter implements WakeUpAdapter { diff --git a/test/postgresql.test.ts b/test/postgresql.test.ts index 4eb1b96..19b6f90 100644 --- a/test/postgresql.test.ts +++ b/test/postgresql.test.ts @@ -729,6 +729,41 @@ describePostgreSQL("PostgreSQL adapter", () => { expect(dashboardResponse.status).toBe(200) expect(await dashboardResponse.text()).toContain("flow") }) + + it("selects PostgreSQL notifications after a probe notification arrives", async () => { + if (!connectionString) throw new Error("PostgreSQL connection string is required") + database = postgresql({ connectionString }) + runtime = configure({ + database, + tableNamePrefix: "postgresql_test_", + authorizeMessage: () => true, + logger: quietLogger, + }) + + const capability = await runtime.wakeUpCapability() + const adapter = await runtime.wakeUpAdapter() + + expect(capability.adapter).toBe("postgresql_notify") + expect(capability.crossesProcesses).toBe(true) + expect(capability.reason).toMatch(/probe notification arrived/i) + expect(adapter).toBeInstanceOf(PostgreSQLWakeUpAdapter) + }) + + it("wakes a listener from a second connection, which a pooled session could not", async () => { + if (!connectionString) throw new Error("PostgreSQL connection string is required") + database = postgresql({ connectionString }) + const listener = database.wakeUp({ channelPrefix: "postgresql_probe_test" }) + try { + const watch = await listener.watch("actors") + await database.connection((connection) => + connection.run("SELECT pg_notify(?, ?)", [listener.channelFor("actors"), "actors"]), + ) + + expect(await watch.wait({ timeoutMilliseconds: 2_000 })).toBe(true) + } finally { + await listener.close() + } + }) }) function dashboardContext() { diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts new file mode 100644 index 0000000..2078f4c --- /dev/null +++ b/test/wake-up-selection.test.ts @@ -0,0 +1,281 @@ +import { afterEach, describe, expect, it } from "vitest" +import { sqlite } from "../src/database/sqlite.js" +import type { + Database, + DatabaseConnection, + DatabaseFamily, + DatabaseTransactionOptions, + RunResult, +} from "../src/database/types.js" +import { configure, type SolidObjectsRuntime } from "../src/runtime.js" +import { + InProcessWakeUpAdapter, + type NotificationWakeUpAdapter, + type WakeUpAdapter, + type WakeUpRole, + type WakeUpWatch, +} from "../src/wake-up.js" +import { selectWakeUp } from "../src/wake-up-selection.js" + +const silentLogger = { + debug: () => undefined, + info: () => undefined, + warn: () => undefined, + error: () => undefined, +} + +class ProbeWakeUpAdapter implements NotificationWakeUpAdapter { + closed = false + readonly watchedRoles: WakeUpRole[] = [] + + constructor(private readonly options: { delivers: boolean; channelPrefix: string }) {} + + channelFor(role: WakeUpRole): string { + return `${this.options.channelPrefix}_${role}` + } + + watch(role: WakeUpRole): WakeUpWatch { + this.watchedRoles.push(role) + return { wait: () => Promise.resolve(this.options.delivers) } + } + + notify(): void {} + + close(): void { + this.closed = true + } +} + +class NotifyingDatabase implements Database { + readonly family: DatabaseFamily = "postgresql" + readonly schemaIdentity: string + readonly adapters: ProbeWakeUpAdapter[] = [] + readonly notifiedChannels: string[] = [] + + constructor(private readonly options: { database: Database; delivers: boolean }) { + this.schemaIdentity = options.database.schemaIdentity + } + + wakeUp(options: { channelPrefix?: string } = {}): NotificationWakeUpAdapter { + const adapter = new ProbeWakeUpAdapter({ + delivers: this.options.delivers, + channelPrefix: options.channelPrefix ?? "solid_objects", + }) + this.adapters.push(adapter) + return adapter + } + + connection( + callback: (connection: DatabaseConnection) => Promise, + ): Promise { + return this.options.database.connection((connection) => callback(this.intercept(connection))) + } + + transaction( + callback: (connection: DatabaseConnection) => Promise, + options?: DatabaseTransactionOptions, + ): Promise { + return this.options.database.transaction(callback, options) + } + + close(): Promise { + return this.options.database.close() + } + + private intercept(connection: DatabaseConnection): DatabaseConnection { + return { + run: (sql: string, parameters: readonly unknown[] = []): Promise => { + if (!sql.includes("pg_notify")) return connection.run(sql, parameters) + this.notifiedChannels.push(String(parameters[0])) + return Promise.resolve({ changes: 0 }) + }, + get: (sql, parameters) => connection.get(sql, parameters), + all: (sql, parameters) => connection.all(sql, parameters), + nowMilliseconds: () => connection.nowMilliseconds(), + } + } +} + +class CustomWakeUpAdapter implements WakeUpAdapter { + watch(): WakeUpWatch { + return { wait: () => Promise.resolve(false) } + } + + notify(): void {} + + close(): void {} +} + +let runtime: SolidObjectsRuntime | undefined + +afterEach(async () => { + await runtime?.close() + runtime = undefined + delete process.env["SOLID_OBJECTS_REDIS_URL"] +}) + +function selectionOptions(options: { database: Database; setting?: unknown }) { + return { + setting: (options.setting ?? "automatic") as never, + database: options.database, + idlePollingIntervalMilliseconds: 1_000, + logger: silentLogger, + } +} + +describe("wake-up selection", () => { + it("keeps an explicitly configured adapter", async () => { + const configured = new InProcessWakeUpAdapter() + const database = sqlite({ path: ":memory:" }) + + const selected = await selectWakeUp(selectionOptions({ database, setting: configured })) + + expect(selected.adapter).toBe(configured) + await database.close() + }) + + it("keeps the capability a configured adapter reports about itself", async () => { + const database = sqlite({ path: ":memory:" }) + + const selected = await selectWakeUp( + selectionOptions({ database, setting: new InProcessWakeUpAdapter() }), + ) + + expect(selected.capability.adapter).toBe("in_process") + expect(selected.capability.crossesProcesses).toBe(false) + await database.close() + }) + + it("records a configured adapter that reports no capability as configured", async () => { + const database = sqlite({ path: ":memory:" }) + + const selected = await selectWakeUp( + selectionOptions({ database, setting: new CustomWakeUpAdapter() }), + ) + + expect(selected.capability.adapter).toBe("configured") + expect(selected.capability.crossesProcesses).toBe(true) + await database.close() + }) + + it("refuses an unknown name rather than polling quietly", async () => { + const database = sqlite({ path: ":memory:" }) + + await expect( + selectWakeUp(selectionOptions({ database, setting: "carrier_pigeon" })), + ).rejects.toThrow(/carrier_pigeon/) + await database.close() + }) + + it("polls on a database without a notification channel", async () => { + const database = sqlite({ path: ":memory:" }) + + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("polling") + expect(selected.capability.crossesProcesses).toBe(false) + expect(selected.capability.measuredFloorMilliseconds).toBe(1_000) + expect(selected.capability.reason).toMatch(/no notification channel/i) + await database.close() + }) + + it("selects PostgreSQL notifications when a probe notification arrives", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("postgresql_notify") + expect(selected.capability.crossesProcesses).toBe(true) + expect(selected.capability.reason).toMatch(/probe notification arrived/i) + await database.close() + }) + + it("notifies the probe channel from the database rather than from the listener", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + + await selectWakeUp(selectionOptions({ database })) + + expect(database.notifiedChannels).toHaveLength(1) + expect(database.notifiedChannels[0]).not.toBe("solid_objects_actors") + expect(database.adapters[0]?.closed).toBe(true) + await database.close() + }) + + it("polls when a probe notification does not arrive", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: false }) + + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("polling") + expect(selected.capability.crossesProcesses).toBe(false) + expect(selected.capability.reason).toMatch(/pooler/i) + await database.close() + }) + + it("warns once about a pooled PostgreSQL session", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: false }) + const warnings: unknown[] = [] + const logger = { ...silentLogger, warn: (entry: unknown) => warnings.push(entry) } + + await selectWakeUp({ ...selectionOptions({ database }), logger }) + + expect(warnings).toHaveLength(1) + await database.close() + }) +}) + +describe("runtime wake-up selection", () => { + it("selects once when callers race for the adapter", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + runtime = configure({ database, authorizeMessage: () => true }) + + const selections = await Promise.all(Array.from({ length: 8 }, () => runtime?.wakeUpAdapter())) + + expect(database.adapters).toHaveLength(2) + expect(new Set(selections).size).toBe(1) + }) + + it("reports the capability of the adapter that is installed", async () => { + runtime = configure({ + database: sqlite({ path: ":memory:" }), + authorizeMessage: () => true, + }) + + const capability = await runtime.wakeUpCapability() + + expect(capability.adapter).toBe("polling") + expect(capability.crossesProcesses).toBe(false) + }) + + it("warns in the doctor about a configured in-process adapter", async () => { + runtime = configure({ + database: sqlite({ path: ":memory:" }), + authorizeAdministration: () => true, + wakeUp: new InProcessWakeUpAdapter(), + }) + await runtime.install() + + const report = await runtime.doctor.run({ roundTrip: "skip" }) + const wakeUp = report.checks.find(({ name }) => name === "wakeUp") + + expect(wakeUp?.status).toBe("warn") + expect(wakeUp?.message).toMatch(/in_process/) + }) + + it("passes the doctor when the installed adapter crosses processes", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + runtime = configure({ database, authorizeAdministration: () => true }) + + const report = await runtime.doctor.run({ roundTrip: "skip" }) + const wakeUp = report.checks.find(({ name }) => name === "wakeUp") + + expect(wakeUp?.status).toBe("pass") + expect(wakeUp?.message).toMatch(/postgresql_notify/) + }) +}) From c0d82f180e98d5774bbc1192dfdaa4be9e1a3755 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:01:50 -0700 Subject: [PATCH 02/12] fix: keep the wake-up choice after the runtime closes `close()` cleared the memoised selection, so a late notify would select again and open a PostgreSQL client that nothing would ever close. The selection now survives `close()`. Every adapter ignores a notify after its own close, so a late caller is a no-op rather than a new connection. `resetForTesting()` still discards it, because a reset must select again. Co-Authored-By: Claude Opus 5 (1M context) --- src/runtime.ts | 8 ++++++-- test/wake-up-selection.test.ts | 11 +++++++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/src/runtime.ts b/src/runtime.ts index c333829..690805e 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -1400,7 +1400,7 @@ export class SolidObjectsRuntime { await this.callerWorker?.stop() this.callerWorker = undefined this.realtime.close() - await this.closeWakeUp() + await this.discardWakeUp() await this.repository.resetForTesting() } @@ -1993,12 +1993,16 @@ export class SolidObjectsRuntime { private async closeWakeUp(): Promise { const selection = this.wakeUpSelection - this.wakeUpSelection = undefined if (!selection) return const selected = await selection.catch(() => undefined) await selected?.adapter.close() } + private async discardWakeUp(): Promise { + await this.closeWakeUp() + this.wakeUpSelection = undefined + } + private wakeUp(role: WakeUpRole): void { void this.resolveWakeUp() .then((selected) => diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 2078f4c..11f4088 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -240,6 +240,17 @@ describe("runtime wake-up selection", () => { expect(new Set(selections).size).toBe(1) }) + it("does not select again after the runtime closes", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + const closing = configure({ database, authorizeMessage: () => true }) + const selected = await closing.wakeUpAdapter() + await closing.close() + + expect(await closing.wakeUpAdapter()).toBe(selected) + expect(database.adapters).toHaveLength(2) + }) + it("reports the capability of the adapter that is installed", async () => { runtime = configure({ database: sqlite({ path: ":memory:" }), From 9f7b0cf95af4954bb3bac6a26c33945672f91090 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:02:11 -0700 Subject: [PATCH 03/12] docs: record the two wake-up parity differences Co-Authored-By: Claude Opus 5 (1M context) --- docs/parity.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/parity.md b/docs/parity.md index cd1254d..b4a2939 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -115,6 +115,12 @@ its measured floor, and why, and warns once when live processes share a database and the installed adapter cannot reach them. MySQL still polls in both runtimes, because MySQL has no notification channel. +Two details differ. `wakeUp: "redis"` without `SOLID_OBJECTS_REDIS_URL` throws +here, because the `redis` client needs a URL, while the Ruby gem falls back to +its own default. The pooled-session warning is emitted once per runtime here and +once per process in Ruby, because this runtime supports several runtimes in one +process. + ## Realtime and browser behavior | Capability | Status | TypeScript shape or remaining work | From 0356606ab21b46cda7eb34b81dad793c548ec767 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:09:41 -0700 Subject: [PATCH 04/12] fix: read the Redis variable without the process global Selection reached for `process.env` directly. Node defines that global and a browser does not, so the browser runtime threw before it could choose polling, and every wake-up stopped. Two browser tests caught it: the outbox drained nothing and live signals never updated. The read now goes through `globalThis.process?.env`, which is undefined in a browser rather than a throw. A test stubs the global away and holds that: without the guard it fails with "Cannot read properties of undefined (reading 'env')". Co-Authored-By: Claude Opus 5 (1M context) --- src/wake-up-selection.ts | 6 +++++- test/wake-up-selection.test.ts | 15 ++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/src/wake-up-selection.ts b/src/wake-up-selection.ts index 387ff4b..aaa2c4d 100644 --- a/src/wake-up-selection.ts +++ b/src/wake-up-selection.ts @@ -190,11 +190,15 @@ function notificationAdapter( } function redisUrl(options: WakeUpSelectionOptions): string | undefined { - const value = options.redisUrl ?? process.env[REDIS_URL_VARIABLE] + const value = options.redisUrl ?? environmentRedisUrl() if (value === undefined || value.length === 0) return undefined return value } +function environmentRedisUrl(): string | undefined { + return globalThis.process?.env?.[REDIS_URL_VARIABLE] +} + function requiredRedisUrl(options: WakeUpSelectionOptions): string { const url = redisUrl(options) if (url === undefined) { diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 11f4088..2e8a12e 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -1,4 +1,4 @@ -import { afterEach, describe, expect, it } from "vitest" +import { afterEach, describe, expect, it, vi } from "vitest" import { sqlite } from "../src/database/sqlite.js" import type { Database, @@ -158,6 +158,19 @@ describe("wake-up selection", () => { await database.close() }) + it("selects where the runtime defines no process global, as a browser does not", async () => { + const database = sqlite({ path: ":memory:" }) + vi.stubGlobal("process", undefined) + try { + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("polling") + } finally { + vi.unstubAllGlobals() + await database.close() + } + }) + it("refuses an unknown name rather than polling quietly", async () => { const database = sqlite({ path: ":memory:" }) From 8e4e554c5c22fc6f7c2e6707d2edf25c52aa9929 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:17:59 -0700 Subject: [PATCH 05/12] refactor: flatten the wake-up setting validation `validateWakeUp` nested the adapter check inside its non-string branch. The adapter check moves to its own function, so each branch is a guard clause. The selection test helper typed its setting as `unknown` and cast it away with `as never`, which removed the type from every caller. It takes a `WakeUpSetting` now, and the one test that supplies an invalid name casts at that call alone. Co-Authored-By: Claude Opus 5 (1M context) --- src/configuration.ts | 21 +++++++++++++-------- test/wake-up-selection.test.ts | 10 +++++++--- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/src/configuration.ts b/src/configuration.ts index 9bae301..b55ab38 100644 --- a/src/configuration.ts +++ b/src/configuration.ts @@ -1,7 +1,9 @@ import { InvalidActor } from "./errors.js" import type { Database } from "./database/types.js" import type { DeepReadonly, JsonObject, JsonValue, Logger, LongRunningComponent } from "./types.js" -import { WAKE_UP_NAMES, type WakeUpName, type WakeUpSetting } from "./wake-up.js" +import { WAKE_UP_NAMES, type WakeUpAdapter, type WakeUpSetting } from "./wake-up.js" + +const wakeUpNames: readonly string[] = WAKE_UP_NAMES export interface AuthorizationInput { actorType: string @@ -192,19 +194,22 @@ export function validateComponent(component: LongRunningComponent): void { } function validateWakeUp(setting: WakeUpSetting): void { - if (typeof setting !== "string") { - for (const name of ["watch", "notify", "close"] as const) { - if (typeof setting[name] !== "function") throw new TypeError(`wakeUp must implement ${name}`) - } - return - } - if (WAKE_UP_NAMES.includes(setting as WakeUpName)) return + if (typeof setting !== "string") return validateWakeUpAdapter(setting) + if (wakeUpNames.includes(setting)) return throw new TypeError( `unknown wakeUp ${JSON.stringify(setting)}, expected one of ${WAKE_UP_NAMES.join(", ")} or an adapter`, ) } +function validateWakeUpAdapter(adapter: WakeUpAdapter): void { + for (const name of ["watch", "notify", "close"] as const) { + if (typeof adapter[name] === "function") continue + + throw new TypeError(`wakeUp must implement ${name}`) + } +} + function validateSettings(settings: RuntimeSettings): void { validateWakeUp(settings.wakeUp) if (!/^[a-z][a-z0-9_]*$/.test(settings.tableNamePrefix)) { diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 2e8a12e..c47027d 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -13,6 +13,7 @@ import { type NotificationWakeUpAdapter, type WakeUpAdapter, type WakeUpRole, + type WakeUpSetting, type WakeUpWatch, } from "../src/wake-up.js" import { selectWakeUp } from "../src/wake-up-selection.js" @@ -114,9 +115,9 @@ afterEach(async () => { delete process.env["SOLID_OBJECTS_REDIS_URL"] }) -function selectionOptions(options: { database: Database; setting?: unknown }) { +function selectionOptions(options: { database: Database; setting?: WakeUpSetting }) { return { - setting: (options.setting ?? "automatic") as never, + setting: options.setting ?? "automatic", database: options.database, idlePollingIntervalMilliseconds: 1_000, logger: silentLogger, @@ -175,7 +176,10 @@ describe("wake-up selection", () => { const database = sqlite({ path: ":memory:" }) await expect( - selectWakeUp(selectionOptions({ database, setting: "carrier_pigeon" })), + selectWakeUp({ + ...selectionOptions({ database }), + setting: "carrier_pigeon" as WakeUpSetting, + }), ).rejects.toThrow(/carrier_pigeon/) await database.close() }) From 46f15f13d3554193f463192e36626eaa7f882777 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:28:00 -0700 Subject: [PATCH 06/12] refactor: name the declared capability apart from the installed one `capability` meant two things: what an adapter reports about itself and what selection installed. Polling shows the difference, because the adapter is an `InProcessWakeUpAdapter` while the installed capability is `polling` with a floor. An adapter declares `defaultCapability` now, and `runtime.wakeUpCapability()` stays the installed record. Ruby uses the same two names. `polling` also takes the interval it needs rather than the whole selection options. Co-Authored-By: Claude Opus 5 (1M context) --- docs/api.md | 5 +++-- docs/configuration.md | 2 +- src/wake-up-selection.ts | 13 ++++++++----- src/wake-up.ts | 4 ++-- src/wake-up/postgresql.ts | 2 +- src/wake-up/redis.ts | 2 +- test/wake-up-selection.test.ts | 1 + 7 files changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/api.md b/docs/api.md index 8c4d5d3..582e7a6 100644 --- a/docs/api.md +++ b/docs/api.md @@ -677,8 +677,9 @@ optional Redis URL and probe timeout. `WakeUpCapability` reports in milliseconds, and the reason. `runtime.wakeUpAdapter()` and `runtime.wakeUpCapability()` return the choice, which is made once per runtime. -An adapter may declare its own `capability`, and selection keeps what it -declares rather than assume. `NotificationWakeUpAdapter` adds +An adapter may declare its own `defaultCapability`, which is what it reports +about itself. Selection keeps what a configured adapter declares rather than +assume, and `runtime.wakeUpCapability()` reports what selection installed. `NotificationWakeUpAdapter` adds `channelFor(role)`, which a database that offers a notification channel provides through `database.wakeUp(options)`. diff --git a/docs/configuration.md b/docs/configuration.md index 4b408f0..ff1077d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -167,7 +167,7 @@ Selection runs once per runtime, on first use. `runtime.wakeUpCapability()` reports what it chose, whether that choice crosses processes, its measured floor in milliseconds, and why. The doctor reports the same record, and the polling-only warning fires on what was installed rather than on whether a -setting was set. An adapter that declares its own `capability` keeps it, so a +setting was set. An adapter that declares its own `defaultCapability` keeps it, so a configured `InProcessWakeUpAdapter` still warns. Automatic selection proves the PostgreSQL path before it chooses it. It listens diff --git a/src/wake-up-selection.ts b/src/wake-up-selection.ts index aaa2c4d..edfe16d 100644 --- a/src/wake-up-selection.ts +++ b/src/wake-up-selection.ts @@ -39,7 +39,7 @@ export async function selectWakeUp(options: WakeUpSelectionOptions): Promise notify(role: WakeUpRole): void | Promise close(): void | Promise - readonly capability?: WakeUpCapability + readonly defaultCapability?: WakeUpCapability } export interface NotificationWakeUpAdapter extends WakeUpAdapter { @@ -37,7 +37,7 @@ export interface NotificationWakeUpAdapter extends WakeUpAdapter { export type WakeUpSetting = WakeUpName | WakeUpAdapter export class InProcessWakeUpAdapter implements WakeUpAdapter { - readonly capability: WakeUpCapability = { + readonly defaultCapability: WakeUpCapability = { adapter: "in_process", crossesProcesses: false, reason: "in-process signalling, which a commit in another process cannot reach", diff --git a/src/wake-up/postgresql.ts b/src/wake-up/postgresql.ts index 5e29cc9..5e6232c 100644 --- a/src/wake-up/postgresql.ts +++ b/src/wake-up/postgresql.ts @@ -26,7 +26,7 @@ export interface PostgreSQLWakeUpOptions { } export class PostgreSQLWakeUpAdapter implements NotificationWakeUpAdapter { - readonly capability: WakeUpCapability = { + readonly defaultCapability: WakeUpCapability = { adapter: "postgresql_notify", crossesProcesses: true, measuredFloorMilliseconds: 2.9, diff --git a/src/wake-up/redis.ts b/src/wake-up/redis.ts index d89e8f7..15da70f 100644 --- a/src/wake-up/redis.ts +++ b/src/wake-up/redis.ts @@ -27,7 +27,7 @@ export interface RedisWakeUpOptions { } export class RedisWakeUpAdapter implements WakeUpAdapter { - readonly capability: WakeUpCapability = { + readonly defaultCapability: WakeUpCapability = { adapter: "redis", crossesProcesses: true, measuredFloorMilliseconds: 5.7, diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index c47027d..d8b5388 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -193,6 +193,7 @@ describe("wake-up selection", () => { expect(selected.capability.crossesProcesses).toBe(false) expect(selected.capability.measuredFloorMilliseconds).toBe(1_000) expect(selected.capability.reason).toMatch(/no notification channel/i) + expect(selected.adapter.defaultCapability?.adapter).toBe("in_process") await database.close() }) From a6b2bd30f160c28eb65a7b88d4b77c74eb32b0a9 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:30:46 -0700 Subject: [PATCH 07/12] fix: report a wake-up selection failure once A selection that cannot run is memoised, so the failure repeated on every commit. `wakeUp: "redis"` without a URL, or `"postgresql"` on a database that offers no notification channel, wrote one error per message. It writes one error per runtime now, under its own event name, because a failure to select is not the same as a failure to notify. The notify path is a try/catch rather than a promise chain, so the caught error carries its own type and no annotation widens it. Co-Authored-By: Claude Opus 5 (1M context) --- src/runtime.ts | 29 ++++++++++++++++----------- test/wake-up-selection.test.ts | 36 ++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 11 deletions(-) diff --git a/src/runtime.ts b/src/runtime.ts index 690805e..259869d 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -203,6 +203,7 @@ export class SolidObjectsRuntime { private pollingOnlyWakeUpWarningEmitted = false private pollingOnlyWakeUpWarningCheck: Promise | undefined private wakeUpSelection: Promise | undefined + private wakeUpSelectionFailureLogged = false constructor(configuration: SolidObjectsConfiguration) { this.settings = buildSettings(configuration) @@ -2001,20 +2002,26 @@ export class SolidObjectsRuntime { private async discardWakeUp(): Promise { await this.closeWakeUp() this.wakeUpSelection = undefined + this.wakeUpSelectionFailureLogged = false } private wakeUp(role: WakeUpRole): void { - void this.resolveWakeUp() - .then((selected) => - notifyWakeUp({ adapter: selected.adapter, logger: this.settings.logger, role }), - ) - .catch((error: unknown) => { - this.settings.logger.error({ - event: "solid_objects.wake_up.failed", - role, - errorName: error instanceof Error ? error.name : "Error", - }) - }) + void this.notifyWhenSelected(role) + } + + private async notifyWhenSelected(role: WakeUpRole): Promise { + try { + const selected = await this.resolveWakeUp() + notifyWakeUp({ adapter: selected.adapter, logger: this.settings.logger, role }) + } catch (error) { + this.reportWakeUpSelectionFailure(error instanceof Error ? error.name : "Error") + } + } + + private reportWakeUpSelectionFailure(errorName: string): void { + if (this.wakeUpSelectionFailureLogged) return + this.wakeUpSelectionFailureLogged = true + this.settings.logger.error({ event: "solid_objects.wake_up.selection_failed", errorName }) } private async authorize(options: { diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index d8b5388..70ae590 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -1,4 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest" +import { Actor } from "../src/actor.js" import { sqlite } from "../src/database/sqlite.js" import type { Database, @@ -107,6 +108,16 @@ class CustomWakeUpAdapter implements WakeUpAdapter { close(): void {} } +class SelectionCounter extends Actor { + static override readonly actorType = "WakeUpSelectionCounter" + + count = 0 + + increment(): void { + this.count += 1 + } +} + let runtime: SolidObjectsRuntime | undefined afterEach(async () => { @@ -269,6 +280,31 @@ describe("runtime wake-up selection", () => { expect(database.adapters).toHaveLength(2) }) + it("reports a selection that cannot run once rather than on every commit", async () => { + const errors: { event?: string }[] = [] + const logger = { ...silentLogger, error: (entry: { event?: string }) => errors.push(entry) } + runtime = configure({ + database: sqlite({ path: ":memory:" }), + wakeUp: "postgresql", + logger, + authorizeMessage: () => true, + }) + runtime.register(SelectionCounter) + await runtime.install() + + await runtime.ref(SelectionCounter, "one").send.increment() + await runtime.ref(SelectionCounter, "one").send.increment() + await vi.waitFor(() => + expect(errors.some(({ event }) => event === "solid_objects.wake_up.selection_failed")).toBe( + true, + ), + ) + + expect( + errors.filter(({ event }) => event === "solid_objects.wake_up.selection_failed"), + ).toHaveLength(1) + }) + it("reports the capability of the adapter that is installed", async () => { runtime = configure({ database: sqlite({ path: ":memory:" }), From 5b52d271c0e4f2c43c1f03a12c60a838e9337b13 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:36:33 -0700 Subject: [PATCH 08/12] fix: poll rather than pretend when an adapter cannot be built A requested adapter that the environment cannot provide claimed to cross processes. `wakeUp: "postgresql"` on a database with no notification channel threw, and a Redis URL without the `redis` package threw, so the failure surfaced as a repeated error after each commit rather than as a choice the operator could read. Each case now polls, logs `solid_objects.wake_up.unavailable` once, and records the reason in the capability, so the doctor warns. That is what the capability record exists for. Only a name that does not exist is still refused, because a typo cannot be honoured at all. Test types no longer annotate values as `unknown`. The intercepting connection takes its parameter types from `DatabaseConnection`, and the warning collectors name the shape they read. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 6 ++ docs/configuration.md | 7 ++ docs/parity.md | 12 +-- src/wake-up-selection.ts | 134 +++++++++++++++++++++------------ test/wake-up-selection.test.ts | 60 +++++++++++++-- 5 files changed, 157 insertions(+), 62 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c84e513..8feec93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,12 @@ - Keep the capability a configured adapter reports about itself. A configured `InProcessWakeUpAdapter` now reports `in_process` and warns, rather than claim that it crosses processes. +- Poll rather than pretend when a requested adapter cannot be built. + `wakeUp: "postgresql"` on a database with no notification channel, + `wakeUp: "redis"` without `SOLID_OBJECTS_REDIS_URL`, and a Redis URL without + the `redis` package each log `solid_objects.wake_up.unavailable` once and + report the reason in the capability, so the doctor warns rather than claim a + cross-process wake-up that cannot happen. - Select once per runtime. `runtime.wakeUpAdapter()` memoises the selection, so callers that race for the first use share one probe rather than run one each. diff --git a/docs/configuration.md b/docs/configuration.md index ff1077d..f36a519 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -163,6 +163,13 @@ short capped backoff only when no synchronous deadline is active. PostgreSQL notifications, then polling. `"in_process"` opts out, `"postgresql"` and `"redis"` force one, and an unknown name throws rather than polls quietly. +A requested adapter that the environment cannot provide polls instead and says +so. `"postgresql"` on a database with no notification channel, `"redis"` without +`SOLID_OBJECTS_REDIS_URL`, and a Redis URL without the `redis` package each log +`solid_objects.wake_up.unavailable` once and report the reason in the +capability. Only a name that does not exist is refused, because a typo cannot be +honoured at all. + Selection runs once per runtime, on first use. `runtime.wakeUpCapability()` reports what it chose, whether that choice crosses processes, its measured floor in milliseconds, and why. The doctor reports the same record, and the diff --git a/docs/parity.md b/docs/parity.md index b4a2939..46dc318 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -115,11 +115,13 @@ its measured floor, and why, and warns once when live processes share a database and the installed adapter cannot reach them. MySQL still polls in both runtimes, because MySQL has no notification channel. -Two details differ. `wakeUp: "redis"` without `SOLID_OBJECTS_REDIS_URL` throws -here, because the `redis` client needs a URL, while the Ruby gem falls back to -its own default. The pooled-session warning is emitted once per runtime here and -once per process in Ruby, because this runtime supports several runtimes in one -process. +A requested adapter that the environment cannot provide polls instead and says +so in both runtimes, rather than claim a cross-process wake-up that cannot +happen. Only a name that does not exist is refused. + +One detail differs. The pooled-session warning is emitted once per runtime here +and once per process in Ruby, because this runtime supports several runtimes in +one process. ## Realtime and browser behavior diff --git a/src/wake-up-selection.ts b/src/wake-up-selection.ts index edfe16d..404d2fc 100644 --- a/src/wake-up-selection.ts +++ b/src/wake-up-selection.ts @@ -17,6 +17,10 @@ export const PROBE_CHANNEL_PREFIX = "solid_objects_probe" const PROBE_ROLE = "actors" as const +type NotificationAdapterFactory = (options?: { + channelPrefix?: string +}) => NotificationWakeUpAdapter + export interface SelectedWakeUp { readonly adapter: WakeUpAdapter readonly capability: WakeUpCapability @@ -61,49 +65,72 @@ async function named(options: WakeUpSelectionOptions & { name: string }): Promis }, } } - if (options.name === "postgresql") { - return { - adapter: notificationAdapter(options.database), - capability: { - adapter: "postgresql_notify", - crossesProcesses: true, - measuredFloorMilliseconds: POSTGRESQL_FLOOR_MILLISECONDS, - reason: "PostgreSQL LISTEN was requested", - }, - } - } - if (options.name === "redis") { - return await redisSelection({ - url: requiredRedisUrl(options), - reason: "Redis was requested", - }) - } + if (options.name === "postgresql") return requestedPostgresql(options) + if (options.name === "redis") return await requestedRedis(options) + throw new TypeError( `unknown wakeUp ${JSON.stringify(options.name)}, expected one of ${WAKE_UP_NAMES.join(", ")} or an adapter`, ) } +function requestedPostgresql(options: WakeUpSelectionOptions): SelectedWakeUp { + const wakeUp = notificationFactory(options.database) + if (!wakeUp) { + return unavailable({ + options, + reason: + `wakeUp "postgresql" needs a database with a notification channel, ` + + `and ${options.database.family} provides none`, + }) + } + return { + adapter: wakeUp(), + capability: { + adapter: "postgresql_notify", + crossesProcesses: true, + measuredFloorMilliseconds: POSTGRESQL_FLOOR_MILLISECONDS, + reason: "PostgreSQL LISTEN was requested", + }, + } +} + +async function requestedRedis(options: WakeUpSelectionOptions): Promise { + const url = redisUrl(options) + if (url === undefined) { + return unavailable({ + options, + reason: `wakeUp "redis" needs ${REDIS_URL_VARIABLE}, which is not set`, + }) + } + return await redisSelection({ options, url, reason: "Redis was requested" }) +} + async function automatic(options: WakeUpSelectionOptions): Promise { const url = redisUrl(options) if (url !== undefined) { return await redisSelection({ + options, url, reason: `${REDIS_URL_VARIABLE} is set, so Redis carries the signal between processes`, }) } - if (options.database.family !== "postgresql" || !options.database.wakeUp) { + const wakeUp = notificationFactory(options.database) + if (options.database.family !== "postgresql" || !wakeUp) { return polling({ idlePollingIntervalMilliseconds: options.idlePollingIntervalMilliseconds, reason: `${options.database.family} has no notification channel and ${REDIS_URL_VARIABLE} is not set`, }) } - return await postgresqlSelection(options) + return await postgresqlSelection({ options, wakeUp }) } -async function postgresqlSelection(options: WakeUpSelectionOptions): Promise { - if (!(await notificationsDeliver(options))) return pooled(options) +async function postgresqlSelection(input: { + options: WakeUpSelectionOptions + wakeUp: NotificationAdapterFactory +}): Promise { + if (!(await notificationsDeliver(input))) return pooled(input.options) return { - adapter: notificationAdapter(options.database), + adapter: input.wakeUp(), capability: { adapter: "postgresql_notify", crossesProcesses: true, @@ -114,14 +141,17 @@ async function postgresqlSelection(options: WakeUpSelectionOptions): Promise { - const probe = notificationAdapter(options.database, { channelPrefix: PROBE_CHANNEL_PREFIX }) +async function notificationsDeliver(input: { + options: WakeUpSelectionOptions + wakeUp: NotificationAdapterFactory +}): Promise { + const probe = input.wakeUp({ channelPrefix: PROBE_CHANNEL_PREFIX }) try { const watch = await probe.watch(PROBE_ROLE) - await options.database.connection((connection) => + await input.options.database.connection((connection) => connection.run("SELECT pg_notify(?, ?)", [probe.channelFor(PROBE_ROLE), PROBE_ROLE]), ) - const timeoutMilliseconds = options.probeTimeoutMilliseconds ?? PROBE_TIMEOUT_MILLISECONDS + const timeoutMilliseconds = input.options.probeTimeoutMilliseconds ?? PROBE_TIMEOUT_MILLISECONDS return (await watch.wait({ timeoutMilliseconds })) === true } catch { return false @@ -144,6 +174,14 @@ function pooled(options: WakeUpSelectionOptions): SelectedWakeUp { }) } +function unavailable(input: { options: WakeUpSelectionOptions; reason: string }): SelectedWakeUp { + input.options.logger.warn({ event: "solid_objects.wake_up.unavailable", reason: input.reason }) + return polling({ + idlePollingIntervalMilliseconds: input.options.idlePollingIntervalMilliseconds, + reason: input.reason, + }) +} + function polling(input: { idlePollingIntervalMilliseconds: number reason: string @@ -159,10 +197,20 @@ function polling(input: { } } -async function redisSelection(input: { url: string; reason: string }): Promise { - const { RedisWakeUpAdapter } = await importRedis() +async function redisSelection(input: { + options: WakeUpSelectionOptions + url: string + reason: string +}): Promise { + const redis = await importRedis() + if (!redis) { + return unavailable({ + options: input.options, + reason: `${REDIS_URL_VARIABLE} is set, and the redis package is not installed`, + }) + } return { - adapter: new RedisWakeUpAdapter({ url: input.url }), + adapter: new redis.RedisWakeUpAdapter({ url: input.url }), capability: { adapter: "redis", crossesProcesses: true, @@ -172,24 +220,18 @@ async function redisSelection(input: { url: string; reason: string }): Promise { +async function importRedis(): Promise { try { return await import("./wake-up/redis.js") - } catch (error) { - throw new TypeError("the redis package is required for the Redis wake-up adapter", { - cause: error, - }) + } catch { + return undefined } } -function notificationAdapter( - database: Database, - options: { channelPrefix?: string } = {}, -): NotificationWakeUpAdapter { - if (!database.wakeUp) { - throw new TypeError(`the ${database.family} database does not provide a notification channel`) - } - return database.wakeUp(options) +function notificationFactory(database: Database): NotificationAdapterFactory | undefined { + const wakeUp = database.wakeUp + if (!wakeUp) return undefined + return (options) => wakeUp.call(database, options) } function redisUrl(options: WakeUpSelectionOptions): string | undefined { @@ -202,14 +244,6 @@ function environmentRedisUrl(): string | undefined { return globalThis.process?.env?.[REDIS_URL_VARIABLE] } -function requiredRedisUrl(options: WakeUpSelectionOptions): string { - const url = redisUrl(options) - if (url === undefined) { - throw new TypeError(`wakeUp "redis" requires ${REDIS_URL_VARIABLE} or an explicit adapter`) - } - return url -} - async function closeQuietly(adapter: WakeUpAdapter): Promise { try { await adapter.close() diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 70ae590..234b620 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -6,7 +6,6 @@ import type { DatabaseConnection, DatabaseFamily, DatabaseTransactionOptions, - RunResult, } from "../src/database/types.js" import { configure, type SolidObjectsRuntime } from "../src/runtime.js" import { @@ -49,16 +48,25 @@ class ProbeWakeUpAdapter implements NotificationWakeUpAdapter { } class NotifyingDatabase implements Database { - readonly family: DatabaseFamily = "postgresql" + readonly family: DatabaseFamily readonly schemaIdentity: string readonly adapters: ProbeWakeUpAdapter[] = [] readonly notifiedChannels: string[] = [] - constructor(private readonly options: { database: Database; delivers: boolean }) { + constructor( + private readonly options: { + database: Database + delivers: boolean + refuses?: boolean + family?: DatabaseFamily + }, + ) { + this.family = options.family ?? "postgresql" this.schemaIdentity = options.database.schemaIdentity } wakeUp(options: { channelPrefix?: string } = {}): NotificationWakeUpAdapter { + if (this.options.refuses) throw new Error("this database refuses a notification adapter") const adapter = new ProbeWakeUpAdapter({ delivers: this.options.delivers, channelPrefix: options.channelPrefix ?? "solid_objects", @@ -86,7 +94,7 @@ class NotifyingDatabase implements Database { private intercept(connection: DatabaseConnection): DatabaseConnection { return { - run: (sql: string, parameters: readonly unknown[] = []): Promise => { + run: (sql, parameters = []) => { if (!sql.includes("pg_notify")) return connection.run(sql, parameters) this.notifiedChannels.push(String(parameters[0])) return Promise.resolve({ changes: 0 }) @@ -183,6 +191,38 @@ describe("wake-up selection", () => { } }) + it("polls and warns when a requested adapter has no notification channel", async () => { + const database = sqlite({ path: ":memory:" }) + const warnings: { event?: string }[] = [] + const logger = { ...silentLogger, warn: (entry: { event?: string }) => warnings.push(entry) } + + const selected = await selectWakeUp({ + ...selectionOptions({ database, setting: "postgresql" }), + logger, + }) + + expect(selected.capability.adapter).toBe("polling") + expect(selected.capability.reason).toMatch(/notification channel/i) + expect(warnings.map(({ event }) => event)).toEqual(["solid_objects.wake_up.unavailable"]) + await database.close() + }) + + it("polls and warns when redis is requested without a url", async () => { + const database = sqlite({ path: ":memory:" }) + const warnings: { event?: string }[] = [] + const logger = { ...silentLogger, warn: (entry: { event?: string }) => warnings.push(entry) } + + const selected = await selectWakeUp({ + ...selectionOptions({ database, setting: "redis" }), + logger, + }) + + expect(selected.capability.adapter).toBe("polling") + expect(selected.capability.reason).toMatch(/SOLID_OBJECTS_REDIS_URL/) + expect(warnings.map(({ event }) => event)).toEqual(["solid_objects.wake_up.unavailable"]) + await database.close() + }) + it("refuses an unknown name rather than polling quietly", async () => { const database = sqlite({ path: ":memory:" }) @@ -247,8 +287,8 @@ describe("wake-up selection", () => { it("warns once about a pooled PostgreSQL session", async () => { const inner = sqlite({ path: ":memory:" }) const database = new NotifyingDatabase({ database: inner, delivers: false }) - const warnings: unknown[] = [] - const logger = { ...silentLogger, warn: (entry: unknown) => warnings.push(entry) } + const warnings: { event?: string }[] = [] + const logger = { ...silentLogger, warn: (entry: { event?: string }) => warnings.push(entry) } await selectWakeUp({ ...selectionOptions({ database }), logger }) @@ -283,8 +323,14 @@ describe("runtime wake-up selection", () => { it("reports a selection that cannot run once rather than on every commit", async () => { const errors: { event?: string }[] = [] const logger = { ...silentLogger, error: (entry: { event?: string }) => errors.push(entry) } + const inner = sqlite({ path: ":memory:" }) runtime = configure({ - database: sqlite({ path: ":memory:" }), + database: new NotifyingDatabase({ + database: inner, + delivers: true, + refuses: true, + family: "sqlite", + }), wakeUp: "postgresql", logger, authorizeMessage: () => true, From 8dfb84a303fcfefec3d9441a76950ad12df1fe28 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:46:14 -0700 Subject: [PATCH 09/12] test: select a requested name without probing The Ruby suite covers this and the TypeScript suite did not. Co-Authored-By: Claude Opus 5 (1M context) --- test/wake-up-selection.test.ts | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 234b620..3f7cefc 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -207,6 +207,19 @@ describe("wake-up selection", () => { await database.close() }) + it("selects a requested name without probing", async () => { + const database = sqlite({ path: ":memory:" }) + process.env["SOLID_OBJECTS_REDIS_URL"] = "redis://127.0.0.1:6379/15" + + const selected = await selectWakeUp(selectionOptions({ database, setting: "redis" })) + + expect(selected.capability.adapter).toBe("redis") + expect(selected.capability.crossesProcesses).toBe(true) + expect(selected.capability.reason).toMatch(/requested/i) + await selected.adapter.close() + await database.close() + }) + it("polls and warns when redis is requested without a url", async () => { const database = sqlite({ path: ":memory:" }) const warnings: { event?: string }[] = [] From 876d0b6db27bddf16ec22b7aacf908f2899bd2b3 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 10:49:30 -0700 Subject: [PATCH 10/12] test: mirror the Ruby selection cases The Ruby suite covers in_process opting out, a Redis URL winning on any database, a database that cannot answer, and a name refused when the configuration is built. The TypeScript suite did not. Co-Authored-By: Claude Opus 5 (1M context) --- test/wake-up-selection.test.ts | 47 ++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 3f7cefc..2e988c9 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -58,6 +58,7 @@ class NotifyingDatabase implements Database { database: Database delivers: boolean refuses?: boolean + unreachable?: boolean family?: DatabaseFamily }, ) { @@ -78,6 +79,7 @@ class NotifyingDatabase implements Database { connection( callback: (connection: DatabaseConnection) => Promise, ): Promise { + if (this.options.unreachable) return Promise.reject(new Error("the database is unreachable")) return this.options.database.connection((connection) => callback(this.intercept(connection))) } @@ -191,6 +193,42 @@ describe("wake-up selection", () => { } }) + it("opts out of selection when in_process is requested", async () => { + const database = sqlite({ path: ":memory:" }) + + const selected = await selectWakeUp(selectionOptions({ database, setting: "in_process" })) + + expect(selected.capability.adapter).toBe("in_process") + expect(selected.capability.crossesProcesses).toBe(false) + expect(selected.capability.reason).toMatch(/requested/i) + await database.close() + }) + + it("selects redis on any database when a url is set", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true }) + process.env["SOLID_OBJECTS_REDIS_URL"] = "redis://127.0.0.1:6379/15" + + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("redis") + expect(selected.capability.crossesProcesses).toBe(true) + expect(database.adapters).toHaveLength(0) + await selected.adapter.close() + await database.close() + }) + + it("polls when the database cannot answer the probe", async () => { + const inner = sqlite({ path: ":memory:" }) + const database = new NotifyingDatabase({ database: inner, delivers: true, unreachable: true }) + + const selected = await selectWakeUp(selectionOptions({ database })) + + expect(selected.capability.adapter).toBe("polling") + expect(selected.capability.crossesProcesses).toBe(false) + await database.close() + }) + it("polls and warns when a requested adapter has no notification channel", async () => { const database = sqlite({ path: ":memory:" }) const warnings: { event?: string }[] = [] @@ -236,6 +274,15 @@ describe("wake-up selection", () => { await database.close() }) + it("refuses an unknown name when the configuration is built", () => { + expect(() => + configure({ + database: sqlite({ path: ":memory:" }), + wakeUp: "carrier_pigeon" as WakeUpSetting, + }), + ).toThrow(/carrier_pigeon/) + }) + it("refuses an unknown name rather than polling quietly", async () => { const database = sqlite({ path: ":memory:" }) From 9716f62be42621511b2e4d6f792e1657039ca199 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 11:02:39 -0700 Subject: [PATCH 11/12] test: refuse a configured adapter that cannot notify The Ruby suite covers the matching case and the TypeScript suite did not. Co-Authored-By: Claude Opus 5 (1M context) --- test/wake-up-selection.test.ts | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/test/wake-up-selection.test.ts b/test/wake-up-selection.test.ts index 2e988c9..7abee14 100644 --- a/test/wake-up-selection.test.ts +++ b/test/wake-up-selection.test.ts @@ -274,6 +274,19 @@ describe("wake-up selection", () => { await database.close() }) + it("refuses a configured adapter that cannot notify", () => { + const adapter: WakeUpAdapter = { + watch: () => ({ wait: () => Promise.resolve(false) }), + notify: () => undefined, + close: () => undefined, + } + Reflect.deleteProperty(adapter, "notify") + + expect(() => configure({ database: sqlite({ path: ":memory:" }), wakeUp: adapter })).toThrow( + /must implement notify/, + ) + }) + it("refuses an unknown name when the configuration is built", () => { expect(() => configure({ From f7bcc71e11e9df57680123a63a8c043507cd79d7 Mon Sep 17 00:00:00 2001 From: Lucas Carlson Date: Tue, 22 Sep 2026 11:02:51 -0700 Subject: [PATCH 12/12] docs: record the three wake-up parity differences Co-Authored-By: Claude Opus 5 (1M context) --- docs/parity.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/parity.md b/docs/parity.md index 46dc318..ea4d443 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -119,9 +119,15 @@ A requested adapter that the environment cannot provide polls instead and says so in both runtimes, rather than claim a cross-process wake-up that cannot happen. Only a name that does not exist is refused. -One detail differs. The pooled-session warning is emitted once per runtime here -and once per process in Ruby, because this runtime supports several runtimes in -one process. +Three details differ, and each follows from the language rather than from the +feature. The pooled-session warning is emitted once per runtime here and once +per process in Ruby, because this runtime supports several runtimes in one +process. A configured adapter must implement `watch`, `notify`, and `close` +here, while Ruby requires `signal` and `wait` and treats `watch` and `stop` as +optional, which is each runtime's own adapter contract. A selection that a later +edit of the settings makes impossible is reported once here and raised in Ruby; +both runtimes refuse an unknown name when the configuration is built, so this +only reaches code that changes the setting afterwards. ## Realtime and browser behavior